Problem parsing updates from routeviews Amsix

[ctestart]

I've been fetching updates from amsix collector for 2020, and starting mid May 2020 and all through June at least, I only get a few thousand updates (instead of the close to hundred millions I get in early May 2020). I get the error: bgpstream_parsebgp_common.c:594: ERROR: Failed to parse message from 'http://archive.routeviews.org/route-views.amsix/bgpdata/2020.06/UPDATES/updates.20200619.1630.bz2' (-2:Invalid Message) Is there a work around this issue to get all updates?

[cmosig]

Not directly a fix, but a temporary "workaround": Update dumps are generated every 15min, which means you can generate the URLs for the update dumps and then pass them to bgpreader. E.g. for the file above the following works: bgpreader -d singlefile -o upd-file=http://archive.routeviews.org/route-views.amsix/bgpdata/2020.06/UPDATES/updates.20200619.1630.bz2

[digizeph]

For more information, I ran the bgpreader command as suggested by @cmosig and collected the error messages:

mingwei@bgpstream:~$ bgpreader -d singlefile -o upd-file=http://archive.routeviews.org/route-views.amsix/bgpdata/2020.06/UPDATES/updates.20200619.1630.bz2 >/dev/null 2>err
mingwei@bgpstream:~$ cat err|sort|uniq -c
   1582 WARN: INVALID_MSG: Invalid prefix in MP_(UN)REACH_NLRI (parsebgp_bgp_update_mp_reach.c:104)
   3600 WARN: INVALID_MSG: Invalid prefix in announced NLRI (parsebgp_bgp_update.c:1114)
    620 WARN: INVALID_MSG: Invalid prefix in withdrawn NLRI (parsebgp_bgp_update.c:1088)
      1 WARN: No time window specified, defaulting to all available data

And counting the correctly parsed messages:

mingwei@bgpstream:~$ bgpreader -d singlefile -o upd-file=http://archive.routeviews.org/route-views.amsix/bgpdata/2020.06/UPDATES/updates.20200619.1630.bz2 2>/dev/null |wc -l
1267958

@ctestart could you provide more information about the libbgpstream version (or commit hash) you're running on and whether you're using python binding?

[albertodainotti]

Hi all, this is a known behavior (triggered by some corrupted messages in the MRTs) and was fixed in the master branch: https://github.com/CAIDA/libparsebgp/pull/84

We'll soon release a new version (which will include this fix).

In the meantime you can compile from the master branch or use the "unstable" packages:

sudo apt remove bgpstream
sudo apt remove libbgpstream2
curl https://pkg.caida.org/os/debian/bootstrap-unstable.sh  |  bash 
sudo apt install bgpstream

To remove these packages and reinstall the stable ones:

sudo apt remove bgpstream
sudo apt remove libbgpstream2
sudo rm /etc/apt/sources.list.d/caida-unstable.list
sudo apt install bgpstream