RIPE indexing error

[ssinyagin]

I'm running a mirror of RIPE database on IRRd from git master. Queries on some RIPE objects produce garbage output (I found so far only one such object).

Configuration:

irr_mirror_interval 600
irr_database ripe mirror_host nrtm.db.ripe.net 4444
irr_database ripe mirror_protocol 3
irr_database ripe mirror-access 27
irr_database ripe max_journal_bytes 134217728

Query and result:

$ whois -h whois.mydomain:5043 -- '!mas-set,AS-DYNANET'
A736
s:        * To view the original object, please query the RIPE Database at:
remarks:        * http://www.ripe.net/whois
remarks:        ****************************

% Tags relating to '87.237.226.32 - 87.237.226.63'
% RIPE-USER-RESOURCE

inetnum:        87.237.226.64 - 87.237.226.95
admin-c:        DUMY-RIPE
netname:        ICCSAT871
descr:          DVB Customer
country:        SA
tech-c:         DUMY-RIPE
status:         ASSIGNED PA
notify:         hmekky@nstt.net
mnt-by:         NSTT-MNT
mnt-lower:      NSTT-MNT
mnt-lower:      mohnstt-mnt
mnt-routes:     mohnstt-mnt
mnt-routes:     NSTT-MNT
mnt-routes:     MNT-C2N
created:        2006-02-07T12:15:22Z
last-modified:  2007-08-13T10:35:33Z
source:         RIPE
remarks:       C

[ssinyagin]

also garbage here, I just took the latest as-set object from ripe.db:

whois -h whois.mydomain:5043 AS21011:AS-CLIENTS

[ssinyagin]

I re-downloaded fresh databases from radb.net and ripe.net, but the issue is still the same (the garbage output is different now)

[job]

what does the original object look like

[ssinyagin]

Also garbage output:

$ whois -h whois.mydomain:5043 220.108.77.in-addr.arpa

[ssinyagin]

The original is a normal as-set object, you can query them from RIPE or RADB

[job]

Do you also have the problem if you add to your configuration

irr_database ripe filter routing-registry-objects|route6

Do you need inetnum information at all?

[ssinyagin]

Will give it a try. I need only AS sets and numbers, and routes.

[job]

I think most IRRd users run with that filter I pasted because they only need a subset of data, like you and me. It may be that the bug is not exposed with that filter enabled

[ssinyagin]

with this filter, the memory footprint is much less than before, but the search for AS-DYNANET still produces garbage:

irr_database ripe mirror_host nrtm.db.ripe.net 4444
irr_database ripe mirror_protocol 3
irr_database ripe mirror-access 27
irr_database ripe max_journal_bytes 134217728
irr_database ripe filter routing-registry-objects|route6

irr_database radb mirror whois.radb.net 43
irr_database altdb mirror whois.radb.net 43
irr_database aoltw mirror whois.radb.net 43
irr_database arin mirror whois.radb.net 43
irr_database bboi mirror whois.radb.net 43
irr_database bell mirror whois.radb.net 43
irr_database canarie mirror whois.radb.net 43
irr_database easynet mirror whois.radb.net 43
irr_database host mirror whois.radb.net 43
irr_database jpirr mirror whois.radb.net 43
irr_database level3 mirror whois.radb.net 43
irr_database nestegg mirror whois.radb.net 43
irr_database nttcom mirror whois.radb.net 43
irr_database openface mirror whois.radb.net 43
irr_database ottix mirror whois.radb.net 43
irr_database panix mirror whois.radb.net 43
irr_database radb mirror whois.radb.net 43
irr_database reach mirror whois.radb.net 43
irr_database rgnet mirror whois.radb.net 43
irr_database risq mirror whois.radb.net 43
irr_database rogers mirror whois.radb.net 43
irr_database tc mirror whois.radb.net 43

$ whois -h whois.mydomain:5043 -- '!mas-set,AS-DYNANET' 
A736
ed:  2006-02-06T09:14:35Z
source:         RIPE
remarks:        ****************************
remarks:        * THIS OBJECT IS MODIFIED
remarks:        * Please note that all data that is generally regarded as personal
remarks:        * data has been removed from this object.
remarks:        * To view the original object, please query the RIPE Database at:
remarks:        * http://www.ripe.net/whois
remarks:        ****************************

% Tags relating to '81.166.124.0 - 81.166.124.255'
% RIPE-USER-RESOURCE

inetnum:        194.122.142.248 - 194.122.142.255
netname:        OTS60896
descr:          Prevent Vochtwering bv
descr:          VIANEN UT
country:        NL
admin-c:        DUMY-RIPE
tech-c:         DUMY-RIPE
statuC

[job]

Can you check which version or commit of IRRd introduces this problem? I cannot reproduce on IRRd 2.3.10

[ssinyagin]

ok, I'll try. The problem is reproduced with v3.0.8. Downgrading further now.

[ssinyagin]

I reproduced it with v2.3.10, so it must be something in fresh dump from ftp.ripe.net

[job]

maybe there is some strange character somewhere

[ssinyagin]

in

grep as-set: local/var/spool/irr_database/ripe.db

the failure is somewhere between lines 3000 and 8000. I'll make a script that finds the place where it starts failing.

[ssinyagin]

Here, after a non-ASCII symbol, it went crazy:

    inet6num:        2a01:4f8:c17:3231::/64
    netname:         STREAMABC-UG-HAFTUNGSBESCHRAENKT
    descr:           streamABC UG (haftungsbeschränkt)
    country:         DE
    admin-c:         TM6822-RIPE
    tech-c:          TM6822-RIPE
    status:          ASSIGNED
    notify:          ripe-mntner@hetzner.de
    mnt-by:          HOS-GUN
    created:         2016-03-03T02:05:35Z
    last-modified:   2016-03-03T02:05:35Z
    source:          RIPE

I installed a fresh dump from ripe.net, and disabled mirroring, and then all objects before this one are fine, and the objects after that one are generating garbage.

Please adapt the *.db reading procedure to accept non-ASCII symbols. This one is a Latin1 symbol, and I guess there might also be Unicode symbols eventually.

[ssinyagin]

I described the problem to RIPE DBM, and they say it's up to IRRd developers to fix this issue.

[job]

I'd welcome a patch :)

[ssinyagin]

I'm afraid it would take too much time for me to explore the depths of the code

[job]

yes, this is a challenge :(

[lytboris]

It turned out that this bug is not Latin1 related in any way but is all about DB size: RIPEdb has grown over 4Gb.

See #48 for a patch.

[job]

woah.. great find! :)