Birdwatcher returns empty array of routes

[stkonst]

Hi guys,

When running the URL to see the routes for a specific peer I don't receive anything from birdwatcher:

root@rs2.hk:/root# curl http://103.247.138.73:29184//routes/protocol/103.247.139.24 Moved Permanently.

When checking the raw output in my browser I see nothing in the array: {"api":{"Version":"2.0.0","result_from_cache":false,"cache_status":{"cached_at":{"date":"2020-12-25T15:43:33.356887066Z","timezone_type":"UTC","timezone":"UTC"}}},"cached_at":"2020-12-25T15:43:33.356887066Z","routes":[],"ttl":"2020-12-25T15:53:33.356887066Z"}

Do you know why this happens? We run Bird 1.6.8 and the latest version of birdwatcher (but same issue was happening in the previous version).

Thank's Stavros

[johannesmoos]

Hi Stavros,

I guess you already figured it out, but it's most likely related to the additional / when querying the endpoint. With that I can reproduce the error:

jmoos @ alice-lg ~ $ curl https://rs1.pmo.de-cix.net:29184//routes/protocol/R46_3
<a href="/routes/protocol/R46_3">Moved Permanently</a>.

jmoos @ alice-lg ~ $ curl https://rs1.pmo.de-cix.net:29184/routes/protocol/R46_3
{"api":{"Version":"1.13.1","result_from_cache":true,"cache_status":{"cached_at":{"date":"2021-05-01T18:35:42.259799485Z","timezone_type":"UTC","timezone":"UTC"}}},"cached_at":"2021-05-01T18:35:42.259799485Z","routes":[{"age":"2021-02-16 05:21:49","bgp":{"as_path":["32261"],"communities":[[32261,12451],[65101,7052],[65102,7000],[65103,380],[65104,150]],"local_pref":"100","next_hop":"185.1.46.3","origin":"Incomplete"},"from_protocol":"R46_3","gateway":"185.1.46.3","interface":"eth1","learnt_from":"","metric":100,"network":"143.131.172.0/24","primary":true,"type":["BGP","unicast","univ"]}],"ttl":"2021-05-01T18:50:42.259799485Z"}

[stkonst]

Hi Johannes,

Yes the issue is still present. I don't think the problem is the double "/" because when I remove it I still get an empty array of routes:

With double "/"

[skon321@lg-glo-01 ~]$ curl http://103.247.138.73:29184//routes/protocol/103.247.139.24 Moved Permanently.

Without double "/"

[skon321@lg-glo-01 ~]$ curl http://103.247.138.73:29184/routes/protocol/103.247.139.24 {"api":{"Version":"2.0.0","result_from_cache":false,"cache_status":{"cached_at":{"date":"2021-05-03T10:09:52.390322295Z","timezone_type":"UTC","timezone":"UTC"}}},"cached_at":"2021-05-03T10:09:52.390322295Z","routes":[],"ttl":"2021-05-03T10:19:52.390322295Z"}

But when checking the Route Server I can see that the 'protocol 103.247.139.24 has routes: skon321@rs1.hk:~$ sudo birdc "show route protocol '103.247.139.24' count" 'BIRD 1.6.8 ready. 12 of 106786 routes for 95438 networks

Any idea why this happens on Bird 1.6.8 while works fine on Bird 2.0.7 ? Thank's for coming back to the issue, I am still struggling to find a solution.

[johannesmoos]

Hi Stavros,

could you please share the log output of birdwatcher for the query and also sniff the BIRD socket with socat? With that we will be able to see the actual birdc show commands that are executed by birdwatcher and the reply from BIRD.

Johannes

[johannesmoos]

Hi, it's most likely resulting from your use of dots in the protocol name. I did some testing and sniffed the socket: Query:

labadm@rs1:~$ curl -sk localhost:29184/routes/protocol/103.247.139.24 | jq .
{
  "api": {
    "Version": "2.0.0",
    "result_from_cache": false,
    "cache_status": {
      "cached_at": {
        "date": "2021-05-09T12:54:54.297448221Z",
        "timezone_type": "UTC",
        "timezone": "UTC"
      }
    }
  },
  "cached_at": "2021-05-09T12:54:54.297448221Z",
  "routes": [],
  "ttl": "2021-05-09T12:59:54.297448221Z"
}

Output from socat:

show route all protocol 103.247.139.24
< 2021/05/09 14:54:54.296864  length=18 from=46 to=63
9001 syntax error

Basically what happens is that birdwatcher performs an invalid query because the protocol name is not quoted/the dot is not escaped. This is not visible in the result as birdwatcher will return data (and logs the query with 200 OK) even though the query is incorrect (or the queried protocol does not exist).

Comparison on CLI:

labadm@rs1:~$ birdc show route all protocol 103.247.139.24                                                                                                               
BIRD 1.6.8 ready.
syntax error

labadm@rs1:~$ birdc show route all protocol "'103.247.139.24'"
BIRD 1.6.8 ready.
103.247.139.24 is not a protocol #this is expected with my config

You could try the same with BIRD2 and see why it behaves differently.

[stkonst]

Hi Johannes,

Indeed, your last comment was to the point. Birdwatcher has issue with special characters when is operating with Bird 1.6.8.

After renaming one of my BGP peers from "200.0.20.1" to "amsixcw" I could successfully retrieve the prefixes. The issue does not exist when birdwatcher is combined with Bird 2.0+ (I can confirm this as well).

Any workaround to solve this in the code? Maybe a quick patch or dirty hack? Otherwise we need to upgrade to Bird 2.0+ and this will take some time.

Thank's again for the help.

[johannesmoos]

@annikahannig, we should quote all protocol names regardless of if they need quoting (e.g. when including special character like a dot) or not. This will prevent issues like this one.

[johannesmoos]

Good news, Stavros:

labadm@rs1:~$ curl -sk localhost:29184/routes/protocol/80.81.194.116 | jq '.routes[] | [.network] | .[]'
"46.31.120.0/21"
"81.89.92.0/22"
"81.89.89.0/24"
"45.140.112.0/22"
"81.89.90.0/23"
"45.129.80.0/22"
"45.159.240.0/22"
"185.70.20.0/22"
"195.211.56.0/22"
"185.176.192.0/22"

I need to test all affected endpoints and will create a PR soon.

[stkonst]

Hi @johannesmoos

I did the fix locally on my laptop (per your advice) and rolled out in production a custom build of birdwatcher, it works !!! Now I can see the protocol names as IPs and their corresponding routes:

https://lg.ams-ix.net/routeservers/mum-rs1-v4

Will wait for the official fix and close the issue.

[johannesmoos]

Hi @stkonst, the fix is merged, could you please verify that the current version also works for you?