search

hadiasghari / pyasn

Python IP address to Autonomous System Number lookup module. (Supports fast local lookups, and historical lookups using archived BGP dumps.)
Other
292 stars 72 forks source link

pyasn_util_convert: Support for gzip #27

Closed sebix closed 8 years ago

sebix commented 8 years ago

RIPE data is gzip-compressed, thus direct support for it would be nice.

Usage example

hadiasghari commented 8 years ago

This is a great idea. The main challenge is finding a cross-platform library in C for the GZip part.

sebix commented 8 years ago

Why not use the gzip module?

hadiasghari commented 8 years ago

It's rather slow to read and load the database in Python, and the loading happens in the C code.

sebix commented 8 years ago

I thought of adding the functionality to the conversion-tool, is there more needed?

hadiasghari commented 8 years ago

@sebix I thought you meant for the ipasn.dat files. Do you mean that?

sebix commented 8 years ago

See the already provided link: https://intelmq.org/2016/04/27/asn-lookup-made-easy.html

To read data from ripe.net, it is needed to convert the data from gzip to bunzip before converting, because the convert utility does not understand gzip.

hadiasghari commented 8 years ago

Ah I see. That should be pretty easy then: pyasn-util-convert needs to do gzip.open() instead of bz2.BZ2File() in a number of places based on the file extension.

jkrauska commented 8 years ago

The mrtparse code does this pretty elegantly.

https://github.com/YoshiyukiYamauchi/mrtparse/blob/master/mrtparse/__init__.py

Search for MAGIC.

hadiasghari commented 8 years ago

@jkrauska thanks, that is pretty neat indeed