ROA for 103.138.48.0/23-24 (ASN 0 vs Origin-AS 136119)

ruizzito commented 3 years ago

Hi guys, I am observing a strange behavior in your network.

Let me try to explain:

We used version: 3.2-2020.10.28.23.06 on our local_cache (GVPALHTA1 - 172.16.177.241)
On our local_validator, we have this:

[@GVPALHTA1 ~]$ curl http://localhost:8080/api/objects/validated | grep -A 3 -B 3 103.138.48.0 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 "maxLength" : 24 }, { "asn" : "0", <<<<<<<< "prefix" : "103.138.48.0/23", "maxLength" : 24 }, { [...]

I mean, the ROA prefix "103.138.48.0/23-24" appears to be "registered by ASN-0".
If we check on our Juniper box, we found this:

`@GRAALHTA3> show validation database record 103.138.48.0 RV database for instance master

Prefix Origin-AS Session State Mismatch 103.138.48.0/23-24 136119 172.16.177.241 valid`

If we look deeper on the trace-options, we see the following:

` @GRAALHTA3> show log rpki.6.gz | match 103.138.48.0 Dec 9 04:36:44.513051 rv_change_db_entry_state: 103.138.48.0/23-24, Origin-AS 0, session 172.16.177.241, unknown -> valid

@GRAALHTA3> show log rpki.0.gz | match 103.138.48.0 Dec 10 08:31:45.390396 rv_change_db_entry_state: 103.138.48.0/23-24, Origin-AS 136119, session 172.16.177.241, unknown -> valid Dec 10 08:31:45.393136 rv_change_db_entry_state: 103.138.48.0/23-24, Origin-AS 0, session 172.16.177.241, valid -> invalid `

It appears that that on "Dec 9 04:36", the ROA state changes from "unknown" to "valid" (by ASN-0). Today (Dec 10 08:31), that ROA changes from ASN-0 to ASN-136119.

Can you please help me to understand this behavior?
Why does local_cache see ASN-0 and Juniper see ASN-136119?
Do you observe the same behaviour on your rtr-rpki clients?

PD-1: We also have checked your public rpki_validator_service (https://rpki-validator.ripe.net/api/export-extended.json) and see the same behavior:

[wuirfb01@gplcomadpe11 ~]$ curl https://rpki-validator.ripe.net/api/export-extended.json | grep -A 6 -B 3 103.138.48.0 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 "serialNumber" : "704250436105586234317453941062193901537241673306" }, { "asn" : "0", "prefix" : "103.138.48.0/23", "maxLength" : 24, "ta" : "APNIC RPKI Root", "notBefore" : "2020-12-10T10:36:02Z", "notAfter" : "2021-12-10T10:41:02Z", "serialNumber" : "257254455200754995803974490035894331837184297494" }, { 100 46.9M 0 46.9M 0 0 31.7M 0 --:--:-- 0:00:01 --:--:-- 31.7M [wuirfb01@gplcomadpe11 ~]$

PD-2: This is our local_cache (just in case)

[@gplcomadpe11 ~]$ curl http://gvpalhta1:8080/api/export-extended.json | grep -A 6 -B 3 103.138.48.0 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 "serialNumber" : "704250436105586234317453941062193901537241673306" }, { "asn" : "0", "prefix" : "103.138.48.0/23", "maxLength" : 24, "ta" : "APNIC RPKI Root", "notBefore" : "2020-12-10T10:36:02Z", "notAfter" : "2021-12-10T10:41:02Z", "serialNumber" : "257254455200754995803974490035894331837184297494" }, { 100 46.9M 0 46.9M 0 0 34.8M 0 --:--:-- 0:00:01 --:--:-- 34.8M

Thanks Rui (ASN12956)

ties commented 3 years ago

Hi,

We started an investigation about this but did not finish this. I currently see this ROA in multiple validator 3 and other (routinator, JDR) instances. Unfortunately it is rather hard to debug in historic data (investigations like this give me ideas for tools that are needed).

Is the issue still occurring?

job commented 3 years ago

@ruizzito

The resource holder of that prefix switched back and forth between AS0 and AS136119 a few times. Here is my historic data:

20201209T034210Z created AS0,103.138.48.0/23-24
20201210T074213Z deleted AS0,103.138.48.0/23-24 and created AS136119,103.138.48.0/23-24
20201210T114212Z deleted AS136119,103.138.48.0/23-24 and created AS0,103.138.48.0/23-24
20201211T070212Z deleted AS0,103.138.48.0/23-24 and created AS136119,103.138.48.0/23-24

Current ROA here: http://console.rpki-client.org/repo-rpki.idnic.net/repo/IDNIC-ID/2/3130332e3133382e34382e302f32332d3234203d3e20313336313139.roa.html

email me at job@sobornost.net if you want more details

ruizzito commented 3 years ago

Hi @ties / @job:

We have a "new" issue. Is not exactly the same case, but is similar. We have a de-synchronization on local-cache vs Juniper Box.

Look:

Prefix: 45.151.115.0/24
Junos: ROA on DDBB

@GRTPAREQ3> show validation database record 45.151.115.0 RV database for instance master

Prefix Origin-AS Session State Mismatch 45.151.115.0/24-24 208046 172.16.177.249 valid

Ripe Local cache: ROA is NOT on DDBB

[wuirfb01@GVPALHTA1 ~]$ curl http://localhost:8080/api/objects/validated | grep -A 3 -B 3 45.151.115.0 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 18.4M 0 18.4M 0 0 24.5M 0 --:--:-- --:--:-- --:--:-- 24.5M [wuirfb01@GVPALHTA1 ~]$

Can you please take a look on this?

Thanks/Cheers, Rui

ties commented 3 years ago

Hi,

I don't see a ROA for 45.151.115.0/24 on multiple validators (checked multiple software implementations). The history seems to have been (at least):

2020-08-25 10:01: ROA created for AS7489 45.151.11.0/24
2020-08-27 20:13: ROA created for AS208046 45.151.11.0/24, ROA deleted for AS208046 45.151.11.0/24
2020-11-16 15:47: ROA created for AS208046 45.151.115.0/24

What software is the Juniper connected to? It would be great to validate what data is being provided over the RTR protocol. Could you dump this with a client (such as rpki-rtr-client or rtrdump (go))?

ruizzito commented 3 years ago

Hi again, 1st of all, thanks for your support/help. I will try to summary my troubleshooting steps:

Our local-cache-server has ripe-sw version "2020.10.28.23.06"

@GVPALHTA1 ~]# yum info rpki-validator ... Installed Packages Name : rpki-validator Arch : noarch Version : 3.2 Release : 2020.10.28.23.06 Size : 50 M Repo : installed From repo : ripencc-rpki-prod ... @GVPALHTA1 ~]$ ip addr show | grep 172 inet 172.16.177.241/29 brd 172.16.177.247 scope global ens34.268

One of our production juniper router has a connection established with that (local-cache-server) for several weeks

@GRAALHTA3> show validation session Session State Flaps Uptime #IPv4/IPv6 records 172.16.177.241 Up 7 5w0d 00:17:34 181688/30570

That router has a ROA for the prefix that I told you yesterday:

@GRAALHTA3> show validation database record 45.151.115.0 RV database for instance master

Prefix Origin-AS Session State Mismatch 45.151.115.0/24-24 208046 172.16.177.249 valid ...

Now I will start a dump, wait several minutes and force a tcp/rpki-rtr flap.

@GRAALHTA3> show system uptime Current time: 2021-01-14 13:23:39 UTC

@GRAALHTA3> monitor traffic interface xe-1/2/1.268 no-resolve matching "port 8323" size 1500 write-file /var/tmp/20210114.GRAALHTA3.rpki.pcap

@GRAALHTA3> show validation session 172.16.177.241 detail | match Serial Serial (Full Update): 1839 Serial (Incremental Update): 1840

@GRAALHTA3> clear validation session 172.16.177.241 Cleared 1 sessions

@GRAALHTA3> show validation session 172.16.177.241 Session State Flaps Uptime #IPv4/IPv6 records 172.16.177.241 Up 8 00:00:16 181707/30590

@GRAALHTA3> show validation session 172.16.177.241 detail | match Serial Serial (Full Update): 1840 Serial (Incremental Update): 1840

Now let's check local roa db again (juniper)

@GRAALHTA3> show validation database record 45.151.115.0 RV database for instance master

Prefix Origin-AS Session State Mismatch 45.151.115.0/24-24 208046 172.16.177.249 valid

As a final step, I stopped the tcpdump.

@GRAALHTA3> file list detail /var/tmp/20210114.GRAALHTA3.rpki.pcap -rw-r--r-- 1 rw wheel 5303338 Jan 14 13:40 /var/tmp/20210114.GRAALHTA3.rpki.pcap total files: 1

The file can be found here: https://www.dropbox.com/sh/n7tu8kmfrvwncyt/AABGmyrE6inK7HnjXZEXgVhfa?dl=0 Can you please take a look?

Thanks, Rui

RIPE-NCC / rpki-validator-3

ROA for 103.138.48.0/23-24 (ASN 0 vs Origin-AS 136119) #304