search

mjschultz / py-radix

Python radix tree implementation for IPv4 and IPv6 prefix matching.
Other
119 stars 37 forks source link

slow and large memory taken for over 1M records #39

Open whille opened 6 years ago

whille commented 6 years ago

For over 1M records, like: "1.1.1.0/24 APNIC|APNIC||" , it takes minutes to initialize and > 1G memory. Any speed factors I neglected? Here's my test codes: https://github.com/whille/mylab/blob/master/qqzeng-ip-python/RadixSearch.py

mjschultz commented 6 years ago

Do you know if you are using the python-only implementation or the c-based extension? The python-only implementation will be slower and much more memory intensive than the C version.

An alternative implementation is the network-finder utility that might work better for you is a binary search which should consume less memory but might be a bit slower (depending on how much less memory, it might be faster if page swapping is in play). It is designed to be API-compatible with this library, so it should be a simple swap in.