search

maxmind / MaxMind-DB-Reader-python

Python MaxMind DB reader extension
https://maxminddb.readthedocs.org/
Apache License 2.0
178 stars 37 forks source link

"MODE_MMAP_EXT requires the maxminddb.extension module to be available" though libmaxminddb is installed #62

Closed paul-nameless closed 4 years ago

paul-nameless commented 4 years ago

I've been trying to use MODE_MMAP_EXT mode but couldn't make it run neither in alpine docker container neither on macOS

bash-5.0# pip list | grep maxminddb
maxminddb  1.5.4
bash-5.0# pip list | grep geoip2
geoip2     3.0.0
bash-5.0# apk list | grep libmaxmind
libmaxminddb-doc-1.4.2-r0 x86_64 {libmaxminddb} (Apache-2.0)
libmaxminddb-dev-1.4.2-r0 x86_64 {libmaxminddb} (Apache-2.0)
libmaxminddb-1.4.2-r0 x86_64 {libmaxminddb} (Apache-2.0) [installed]

Test file:

import sys

from geoip2.database import Reader
from maxminddb import MODE_MMAP_EXT

r = Reader('GeoLite2-Country.mmdb', mode=MODE_MMAP_EXT)
print(r.country(sys.argv[1]).country.iso_code)
ash-5.0# python test.py 1.1.1.1
Traceback (most recent call last):
  File "test.py", line 6, in <module>
    r = Reader('GeoLite2-Country.mmdb.1', mode=MODE_MMAP_EXT)
  File "/usr/local/lib/python3.8/site-packages/geoip2/database.py", line 85, in __init__
    self._db_reader = maxminddb.open_database(fileish, mode)
  File "/usr/local/lib/python3.8/site-packages/maxminddb/__init__.py", line 41, in open_database
    raise ValueError(
ValueError: MODE_MMAP_EXT requires the maxminddb.extension module to be available

P.S. I've installed libmaxminddb before the pip packages as mentioned in documentation.

oschwald commented 4 years ago

You will need to install libmaxminddb-dev and you will need to reinstall the maxminddb package.

paul-nameless commented 4 years ago

Hello, thanks for the reply. I've tried installing dev library as well before, but no luck. Here is a simple command to reproduce the issue:

docker run -it --rm python:3.8.2-alpine3.11 sh -c "apk add libmaxminddb-dev libmaxminddb && pip install geoip2 maxminddb && python -c 'from geoip2.database import Reader; from maxminddb import MODE_MMAP_EXT; r = Reader(\"GeoLite2-Country.mmdb\", mode=MODE_MMAP_EXT)'"
oschwald commented 4 years ago

You will also need a C build environment, e.g.:

docker run -it --rm python:3.8.2-alpine3.11 sh -c "apk add libmaxminddb-dev libmaxminddb gcc musl-dev && pip install  geoip2 maxminddb && python -c 'from geoip2.database import Reader; from maxminddb import MODE_MMAP_EXT; r = Reader(\"GeoLite2-Country.mmdb\", mode=MODE_MMAP_EXT)'"
paul-nameless commented 4 years ago

Thanks for the help!

Autopilot9369 commented 4 years ago

Hey - I am facing the same issue on ubuntu 18.04. I am using apt-repository ppa:maxmind/ppa. Using python 3.6.9.

root@admin:/home/subham# apt install libmaxminddb-dev libmaxminddb0 mmdb-bin && pip3 install  geoip2 maxminddb && python3 -c 'from geoip2.database import Reader; from maxminddb import MODE_MMAP_EXT; r = Reader("/etc/app/geoip/database/GeoLite2-Country.mmdb", mode=MODE_MMAP_EXT)'

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  libmaxminddb-dev libmaxminddb0 mmdb-bin
0 upgraded, 3 newly installed, 0 to remove and 69 not upgraded.
Need to get 0 B/71.4 kB of archives.
After this operation, 191 kB of additional disk space will be used.
Selecting previously unselected package libmaxminddb0:amd64.
(Reading database ... 26087 files and directories currently installed.)
Preparing to unpack .../libmaxminddb0_1.4.3-0+maxmind1~bionic_amd64.deb ...
Unpacking libmaxminddb0:amd64 (1.4.3-0+maxmind1~bionic) ...
Selecting previously unselected package libmaxminddb-dev:amd64.
Preparing to unpack .../libmaxminddb-dev_1.4.3-0+maxmind1~bionic_amd64.deb ...
Unpacking libmaxminddb-dev:amd64 (1.4.3-0+maxmind1~bionic) ...
Selecting previously unselected package mmdb-bin.
Preparing to unpack .../mmdb-bin_1.4.3-0+maxmind1~bionic_amd64.deb ...
Unpacking mmdb-bin (1.4.3-0+maxmind1~bionic) ...
Setting up libmaxminddb0:amd64 (1.4.3-0+maxmind1~bionic) ...
Setting up libmaxminddb-dev:amd64 (1.4.3-0+maxmind1~bionic) ...
Setting up mmdb-bin (1.4.3-0+maxmind1~bionic) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1.2) ...

Collecting geoip2
  Using cached https://files.pythonhosted.org/packages/15/85/c170af85a6a096fbe7a8ee3f6abcea7497113a70798398267d3cad8fabf1/geoip2-4.1.0-py2.py3-none-any.whl
Collecting maxminddb
Requirement already satisfied: urllib3<2.0.0,>=1.25.2 in /usr/local/lib/python3.6/dist-packages/urllib3-1.25.9-py3.6.egg (from geoip2)
Requirement already satisfied: aiohttp<4.0.0,>=3.6.2 in /usr/local/lib/python3.6/dist-packages (from geoip2)
Requirement already satisfied: requests<3.0.0,>=2.24.0 in /usr/local/lib/python3.6/dist-packages (from geoip2)
Requirement already satisfied: typing-extensions>=3.6.5 in /usr/local/lib/python3.6/dist-packages (from aiohttp<4.0.0,>=3.6.2->geoip2)
Requirement already satisfied: idna-ssl>=1.0; python_version < "3.7" in /usr/local/lib/python3.6/dist-packages (from aiohttp<4.0.0,>=3.6.2->geoip2)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.6/dist-packages (from aiohttp<4.0.0,>=3.6.2->geoip2)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.6/dist-packages (from aiohttp<4.0.0,>=3.6.2->geoip2)
Requirement already satisfied: chardet<4.0,>=2.0 in /usr/local/lib/python3.6/dist-packages/chardet-3.0.4-py3.6.egg (from aiohttp<4.0.0,>=3.6.2->geoip2)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.6/dist-packages (from aiohttp<4.0.0,>=3.6.2->geoip2)
Requirement already satisfied: async-timeout<4.0,>=3.0 in /usr/local/lib/python3.6/dist-packages (from aiohttp<4.0.0,>=3.6.2->geoip2)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages/certifi-2020.6.20-py3.6.egg (from requests<3.0.0,>=2.24.0->geoip2)
Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3/dist-packages (from requests<3.0.0,>=2.24.0->geoip2)
Installing collected packages: maxminddb, geoip2
Successfully installed geoip2-4.1.0 maxminddb-2.0.3

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/geoip2/database.py", line 106, in __init__
    self._db_reader = maxminddb.open_database(fileish, mode)
  File "/usr/local/lib/python3.6/dist-packages/maxminddb/__init__.py", line 46, in open_database
    "MODE_MMAP_EXT requires the maxminddb.extension module to be available"
ValueError: MODE_MMAP_EXT requires the maxminddb.extension module to be available

Where am I going wrong?

oschwald commented 4 years ago

If you do a pip3 install maxminddb --force-reinstall --verbose --no-cache-dir, it should give you more information on why the building of the extension failed.

Autopilot9369 commented 4 years ago

Thanks. For some weird reason using --ignore-installed flag with pip3 worked (even though maxminddb wasn't installed).

YunfanZhang42 commented 3 years ago

Thanks. For some weird reason using --ignore-installed flag with pip3 worked (even though maxminddb wasn't installed).

This is my experience too using macOS 10.13. After installing maxminddb through brew I had to run this again in order to make python modules work with MODE_MMAP_EXT.