ipvoid_scan "list out of range" and maxmind "not in database"

[christinabejjani]

Hello, I have a .csv file of 993 IP addresses. I've adjusted the config file and columns texts, signed up for all the databases (downloaded the Maxmind databases in both csv and the .mmdb forms), and tried debugging this myself.

1) No matter what, I'm constantly getting the error: "WARNING:root:ipvoid_scan: list index out of range" for ipvoid_scan. As far as I could tell, there was nothing to adjust for ipvoid in the config file, and the python code has a direct link to the IP void site where you input the blacklisted IPs. When I input some of the IPs from the input.csv file, ipvoid the site works just fine. This seems to be a data communication issue with the API (I am registered and changed the api key for apivoid)? But I'm not certain.

2) At various points, Maxmind has failed (e.g., entry #220, #237, #364, #391 in my csv file). It seems to be running, if it's getting into the hundreds and not throwing errors for the other IPs (I deleted the IP at 220, then at 237, etc. in the hopes they'd be outliers), but at those select IP addresses, it throws the error: "ERROR:root:The address XX.XXX.XXX.XX is not in the database." Thus when the code tries to do maxmind_geocode_ip(args,ip), it can't; if it can't find the IP in the database, it fails to read the IP. When that error is thrown, there is a second error: "odf = df[0].apply(lambda c: pd.Series(query_ip(args, c)))" (I'm using a Jupyter notebook interface). It says: "AttributeError: 'Namespace' object has no attribute 'verbose'". It gets caught in the "if args.verbose" part of the try/except loop of the python code. When I was looking through the python code to see if I could add in my own try/except code to catch the maxmind error, it seemed like every database had a mechanism to catch the error except the maxmind_geocode_ip function. Is there a reason for this? I'm not familiar with the IP databases, and when I was signing up for Maxmind, they implied they'd changed the databases in 2019, so I wasn't sure if this was also something that occurred as a result of that change.

[suriyan]

re. 1) We recommend you to using apivoid_api instead of ipvoid_scan. Actually ipvoid_scan using IP Reputation API in background. So we recommend you to disable ipvoid and enable apivoid in the config file like the following:-

[ipvoid]
enable = 0

[apivoid]
enable = 1
api_key = PUT YOU APIVOID API KEY HERE

re. 2) I think not all IP addresses available in MaxMind databases. So please ignore the error message that tell the specific IP is not in the database. That's right if you install our package from PyPI you will get MaxMind databases in 2018. So please feel free to download the latest City database from MaxMind and extract mmdb file to your machine then please update the config file point to the location of MaxMind database file in your machine.

[maxmind]
enable = 1
dbpath = ./db

Let me know if any further issue. Thanks!

[christinabejjani]

My apologies, I didn't get back to you on this. I reinstalled the package when I saw that you had added in a try/catch for Maxmind, but it still doesn't work. It first puts out: "ERROR:root:The address [insert IP here] is not in the database."

That's what the rest of the error looks like. I just kept deleting the IPs in a new file to see whether I could get the maxmind database (everything else is disabled) to run all the way through. I could not, because the next error was: "ERROR:root:'maxmind.location.latitude'"

I commented out the longitude and latitude parts in the columns.txt, and that doesn't make a difference since this is an error within the code. I also shortened the IP .csv file and just ran through 5 IP addresses, disabling all but 1 of the databases as I went along. When I did that, ping, traceroute, and tzwhere also outputted errors, but the rest worked fine. It may be that I am missing some kind of dependency that others have.