search

irrdnet / irrd

IRRd version 4
http://irrd.readthedocs.io/en/stable/
Other
154 stars 51 forks source link

gzip dump issue? #381

Closed job closed 4 years ago

job commented 4 years ago

it appears the database dumps are incomplete on my 4.1.0rc2 instance

bench$ cat irrd.yaml  | grep export | sort -u | head -2
            export_destination: /home/irrd/dumps/
            export_timer: 3600
bench$ gunzip -f *.gz
gunzip: afrinic.db.gz: Input/output error
gunzip: altdb.db.gz: crc error
gunzip: apnic.db.gz: Input/output error
gunzip: arin-nonauth.db.gz: crc error
gunzip: arin-whois.db.gz: unrecognized file format
gunzip: arin.db.gz: Input/output error
gunzip: bboi.db.gz: Input/output error
gunzip: bell.db.gz: crc error
gunzip: jpirr.db.gz: Input/output error
gunzip: lacnic.db.gz: crc error
gunzip: level3.db.gz: Input/output error
gunzip: nttcom.db.gz: Input/output error
gunzip: radb.db.gz: crc error
gunzip: registrobr.db.gz: unrecognized file format
gunzip: rgnet.db.gz: unrecognized file format
gunzip: ripe-nonauth.db.gz: Input/output error
gunzip: ripe.db.gz: Input/output error
gunzip: tc.db.gz: Input/output error

bench$ ls -alhtr
total 2633216
drwxr-xr-x  11 irrd  irrd   512B Aug 26 23:30 ..
-rw-r--r--   1 irrd  irrd   593K Aug 27 09:33 altdb.db.gz
-rw-r--r--   1 irrd  irrd     5B Aug 27 09:33 ALTDB.CURRENTSERIAL
-rw-r--r--   1 irrd  irrd   276K Aug 27 09:33 bell.db.gz
-rw-r--r--   1 irrd  irrd     6B Aug 27 09:33 BELL.CURRENTSERIAL
-rw-r--r--   1 irrd  irrd   1.1M Aug 27 09:33 ripe-nonauth.db.gz
-rw-r--r--   1 irrd  irrd     8B Aug 27 09:33 RIPE-NONAUTH.CURRENTSERIAL
-rw-r--r--   1 irrd  irrd   5.1M Aug 27 09:33 level3.db.gz
-rw-r--r--   1 irrd  irrd     6B Aug 27 09:33 LEVEL3.CURRENTSERIAL
-rw-r--r--   1 irrd  irrd     0B Aug 27 09:33 rgnet.db.gz
-rw-r--r--   1 irrd  irrd     3B Aug 27 09:33 RGNET.CURRENTSERIAL
-rw-r--r--   1 irrd  irrd   300K Aug 27 09:33 jpirr.db.gz
-rw-r--r--   1 irrd  irrd     6B Aug 27 09:33 JPIRR.CURRENTSERIAL
-rw-r--r--   1 irrd  irrd   3.1M Aug 27 09:34 nttcom.db.gz
-rw-r--r--   1 irrd  irrd     7B Aug 27 09:34 NTTCOM.CURRENTSERIAL
-rw-r--r--   1 irrd  irrd   7.0M Aug 27 09:34 apnic.db.gz
-rw-r--r--   1 irrd  irrd     7B Aug 27 09:34 APNIC.CURRENTSERIAL
-rw-r--r--   1 irrd  irrd  37.2M Aug 27 09:34 ripe.db.gz
-rw-r--r--   1 irrd  irrd     8B Aug 27 09:34 RIPE.CURRENTSERIAL
-rw-r--r--   1 irrd  irrd  16.0M Aug 27 09:35 radb.db.gz
-rw-r--r--   1 irrd  irrd     7B Aug 27 09:35 RADB.CURRENTSERIAL
-rw-r--r--   1 irrd  irrd  48.0K Aug 27 09:37 bboi.db.gz
-rw-r--r--   1 irrd  irrd     5B Aug 27 09:37 BBOI.CURRENTSERIAL
-rw-r--r--   1 irrd  irrd   1.0M Aug 27 09:37 arin.db.gz
-rw-r--r--   1 irrd  irrd     5B Aug 27 09:37 ARIN.CURRENTSERIAL
-rw-r--r--   1 irrd  irrd   965K Aug 27 09:37 arin-nonauth.db.gz
-rw-r--r--   1 irrd  irrd     5B Aug 27 09:37 ARIN-NONAUTH.CURRENTSERIAL
-rw-r--r--   1 irrd  irrd     0B Aug 27 09:38 registrobr.db.gz
-rw-r--r--   1 irrd  irrd   753K Aug 27 09:38 tc.db.gz
-rw-r--r--   1 irrd  irrd     5B Aug 27 09:38 TC.CURRENTSERIAL
-rw-r--r--   1 irrd  irrd   1.8M Aug 27 09:38 afrinic.db.gz
-rw-r--r--   1 irrd  irrd     6B Aug 27 09:38 AFRINIC.CURRENTSERIAL
-rw-r--r--   1 irrd  irrd     0B Aug 27 09:38 arin-whois.db.gz
-rw-r--r--   1 irrd  irrd  66.2K Aug 27 09:38 lacnic.db.gz
-rw-r--r--   1 irrd  irrd     5B Aug 27 09:38 LACNIC.CURRENTSERIAL
--w-------   1 irrd  irrd   170M Aug 27 10:11 apnic.db
--w-------   1 irrd  irrd  17.8M Aug 27 10:11 arin-nonauth.db
--w-------   1 irrd  irrd  12.9M Aug 27 10:11 arin.db
--w-------   1 irrd  irrd   592K Aug 27 10:11 bboi.db
--w-------   1 irrd  irrd   5.1M Aug 27 10:11 bell.db
--w-------   1 irrd  irrd   3.3M Aug 27 10:11 jpirr.db
--w-------   1 irrd  irrd   768K Aug 27 10:11 lacnic.db
--w-------   1 irrd  irrd  59.4M Aug 27 10:11 level3.db
--w-------   1 irrd  irrd   193M Aug 27 10:11 nttcom.db
--w-------   1 irrd  irrd   310M Aug 27 10:11 radb.db
--w-------   1 irrd  irrd  40.2M Aug 27 10:11 ripe-nonauth.db
--w-------   1 irrd  irrd   387M Aug 27 10:11 ripe.db
--w-------   1 irrd  irrd   8.9M Aug 27 10:11 tc.db
drwxr-xr-x   2 irrd  irrd   1.5K Aug 27 10:11 .
job commented 4 years ago

Some databases are empty because this instance is not on all accesslists, but for example RIPE is expected to be complete

mxsasha commented 4 years ago

I've tried this on irrd02, and it works fine. Valid gzip files, with content that looks reasonable. Sizes seem similar to yours:

-rw-r--r-- 1 irrd irrd    6 Aug 28 18:05 AFRINIC.CURRENTSERIAL
-rw-r--r-- 1 irrd irrd 1.9M Aug 28 18:05 afrinic.db.gz
-rw-r--r-- 1 irrd irrd    5 Aug 28 18:05 ALTDB.CURRENTSERIAL
-rw-r--r-- 1 irrd irrd 617K Aug 28 18:05 altdb.db.gz
-rw-r--r-- 1 irrd irrd    7 Aug 28 18:13 APNIC.CURRENTSERIAL
-rw-r--r-- 1 irrd irrd 7.7M Aug 28 18:13 apnic.db.gz
-rw-r--r-- 1 irrd irrd    5 Aug 28 18:05 ARIN.CURRENTSERIAL
-rw-r--r-- 1 irrd irrd 1.1M Aug 28 18:05 arin.db.gz
-rw-r--r-- 1 irrd irrd    5 Aug 28 18:12 ARIN-NONAUTH.CURRENTSERIAL
-rw-r--r-- 1 irrd irrd 965K Aug 28 18:12 arin-nonauth.db.gz
-rw-r--r-- 1 irrd irrd    5 Aug 28 18:12 BBOI.CURRENTSERIAL
-rw-r--r-- 1 irrd irrd  56K Aug 28 18:12 bboi.db.gz
-rw-r--r-- 1 irrd irrd    6 Aug 28 18:12 BELL.CURRENTSERIAL
-rw-r--r-- 1 irrd irrd 277K Aug 28 18:12 bell.db.gz
-rw-r--r-- 1 irrd irrd    6 Aug 28 18:12 JPIRR.CURRENTSERIAL
-rw-r--r-- 1 irrd irrd 303K Aug 28 18:12 jpirr.db.gz
-rw-r--r-- 1 irrd irrd    6 Aug 28 18:12 LEVEL3.CURRENTSERIAL
-rw-r--r-- 1 irrd irrd 5.3M Aug 28 18:12 level3.db.gz
-rw-r--r-- 1 irrd irrd    7 Aug 28 18:07 NTTCOM.CURRENTSERIAL
-rw-r--r-- 1 irrd irrd 3.2M Aug 28 18:07 nttcom.db.gz
-rw-r--r-- 1 irrd irrd    7 Aug 28 18:14 RADB.CURRENTSERIAL
-rw-r--r-- 1 irrd irrd  17M Aug 28 18:14 radb.db.gz
-rw-r--r-- 1 irrd irrd    4 Aug 28 18:12 RGNET.CURRENTSERIAL
-rw-r--r-- 1 irrd irrd 4.9K Aug 28 18:12 rgnet.db.gz
-rw-r--r-- 1 irrd irrd    8 Aug 28 18:14 RIPE.CURRENTSERIAL
-rw-r--r-- 1 irrd irrd  38M Aug 28 18:14 ripe.db.gz
-rw-r--r-- 1 irrd irrd    8 Aug 28 18:12 RIPE-NONAUTH.CURRENTSERIAL
-rw-r--r-- 1 irrd irrd 1.2M Aug 28 18:12 ripe-nonauth.db.gz
-rw-r--r-- 1 irrd irrd    5 Aug 28 18:13 TC.CURRENTSERIAL
-rw-r--r-- 1 irrd irrd 847K Aug 28 18:13 tc.db.gz

I'm guessing you're seeing this on OpenBSD? We use the Python standard library gzip, perhaps it is broken somehow. You could try:

import gzip
content = b"Lots of content here" * 1000
with gzip.open('gziptest.txt.gz', 'wb') as f:
    f.write(content)

And see if that produces a valid gzip file with your Python.

job commented 4 years ago

yes, that works 

bench$ . irrd-venv/bin/activate
(irrd-venv) bench$ python
Python 3.7.8 (default, Aug 20 2020, 20:55:46)
[Clang 10.0.1 ] on openbsd6
Type "help", "copyright", "credits" or "license" for more information.
>>> import gzip
>>> content = b"Lots of content here" * 1000
>>> with gzip.open('gziptest.txt.gz', 'wb') as f:
...  f.write(content)
...
20000
>>>
(irrd-venv) bench$ gunzip gziptest.txt.gz
(irrd-venv) bench$