search

ratibor78 / geostat

GeoStat, Python script for parsing Nginx and Apache logs files and getting GEO data from incoming IP's.
MIT License
69 stars 20 forks source link

don't exit after encountering non routable ip address #4

Closed umataro closed 5 years ago

umataro commented 5 years ago

At the moment, if I query my nginx server from a private range, geoparser.py script throws an exception and exits. The whole systemd unit has to restart.

i.e.:

-- The start-up result is done.
Jun 06 13:48:23 image python[23573]: Traceback (most recent call last):
Jun 06 13:48:23 image python[23573]:   File "geoparser.py", line 101, in <module>
Jun 06 13:48:23 image python[23573]:     main()
Jun 06 13:48:23 image python[23573]:   File "geoparser.py", line 94, in main
Jun 06 13:48:23 image python[23573]:     logparse(LOGPATH, INFLUXHOST, INFLUXPORT, INFLUXDBDB, INFLUXUSER, INFLUXUSERPASS, MEASUREMENT, GEOIPDB, INODE) # NOQA
Jun 06 13:48:23 image python[23573]:   File "geoparser.py", line 56, in logparse
Jun 06 13:48:23 image python[23573]:     INFO = GI.city(IP)
Jun 06 13:48:23 image python[23573]:   File "/opt/geostat/venv/local/lib/python2.7/site-packages/geoip2/database.py", line 114, in city
Jun 06 13:48:23 image python[23573]:     return self._model_for(geoip2.models.City, 'City', ip_address)
Jun 06 13:48:23 image python[23573]:   File "/opt/geostat/venv/local/lib/python2.7/site-packages/geoip2/database.py", line 195, in _model_for
Jun 06 13:48:23 image python[23573]:     record = self._get(types, ip_address)
Jun 06 13:48:23 image python[23573]:   File "/opt/geostat/venv/local/lib/python2.7/site-packages/geoip2/database.py", line 191, in _get
Jun 06 13:48:23 image python[23573]:     "The address %s is not in the database." % ip_address)
Jun 06 13:48:23 image python[23573]: geoip2.errors.AddressNotFoundError: The address 10.1.18.197 is not in the database.
Jun 06 13:48:23 image systemd[1]: geostat.service: Main process exited, code=exited, status=1/FAILURE
Jun 06 13:48:23 image systemd[1]: geostat.service: Unit entered failed state.
Jun 06 13:48:23 image systemd[1]: geostat.service: Failed with result 'exit-code'.
Jun 06 13:48:25 image systemd[1]: geostat.service: Service hold-off time over, scheduling restart.
Jun 06 13:48:25 image systemd[1]: Stopped GeoStat.
-- Subject: Unit geostat.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit geostat.service has finished shutting down.
Jun 06 13:48:25 image systemd[1]: Started GeoStat.
-- Subject: Unit geostat.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit geostat.service has finished starting up.
-- 
-- The start-up result is done.
Jun 06 13:48:27 image python[23577]: Traceback (most recent call last):
Jun 06 13:48:27 image python[23577]:   File "geoparser.py", line 101, in <module>
Jun 06 13:48:27 image python[23577]:     main()
Jun 06 13:48:27 image python[23577]:   File "geoparser.py", line 94, in main
Jun 06 13:48:27 image python[23577]:     logparse(LOGPATH, INFLUXHOST, INFLUXPORT, INFLUXDBDB, INFLUXUSER, INFLUXUSERPASS, MEASUREMENT, GEOIPDB, INODE) # NOQA
Jun 06 13:48:27 image python[23577]:   File "geoparser.py", line 56, in logparse
Jun 06 13:48:27 image python[23577]:     INFO = GI.city(IP)
Jun 06 13:48:27 image python[23577]:   File "/opt/geostat/venv/local/lib/python2.7/site-packages/geoip2/database.py", line 114, in city
Jun 06 13:48:27 image python[23577]:     return self._model_for(geoip2.models.City, 'City', ip_address)
Jun 06 13:48:27 image python[23577]:   File "/opt/geostat/venv/local/lib/python2.7/site-packages/geoip2/database.py", line 195, in _model_for
Jun 06 13:48:27 image python[23577]:     record = self._get(types, ip_address)
Jun 06 13:48:27 image python[23577]:   File "/opt/geostat/venv/local/lib/python2.7/site-packages/geoip2/database.py", line 191, in _get
Jun 06 13:48:27 image python[23577]:     "The address %s is not in the database." % ip_address)
Jun 06 13:48:27 image python[23577]: geoip2.errors.AddressNotFoundError: The address 10.1.18.197 is not in the database.
Jun 06 13:48:27 image systemd[1]: geostat.service: Main process exited, code=exited, status=1/FAILURE
Jun 06 13:48:27 image systemd[1]: geostat.service: Unit entered failed state.
Jun 06 13:48:27 image systemd[1]: geostat.service: Failed with result 'exit-code'.
Jun 06 13:48:29 image systemd[1]: geostat.service: Service hold-off time over, scheduling restart.
Jun 06 13:48:29 image systemd[1]: Stopped GeoStat.
-- Subject: Unit geostat.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit geostat.service has finished shutting down.

It'd be much better to catch the error, display it and continue. Alternatively, assign some default country to these ip addresses.

Anyway, thank you for the time you've spent writing this.

umataro commented 5 years ago

Actually, I've just noticed the pull request from @matstace solves this.

ratibor78 commented 5 years ago

Yep, thanks, @umataro for the information, I merged the @MatStace pull request, now all must be ok with private IPS too.

ratibor78 commented 5 years ago

The issue closed after @MatStace pull request was accepted.