search

Iotic-Labs / py-ubjson

Universal Binary JSON draft-12 serializer for Python
Apache License 2.0
42 stars 12 forks source link

Understanding why my binary extension was not installed for a derived module (bjdata) #16

Closed fangq closed 2 years ago

fangq commented 2 years ago

this is not a bug report, rather a question.

As I mentioned in a previous PR (#14), I've been working on a binary format derived from UBJSON, called BJData. BJData is designed to be an improved version of UBJSON with several notable changes (summarized here: https://github.com/nlohmann/json/pull/3336#issue-1140817315).

Most of the project files, including setup.py, were derived from your project, and I have been making releases via pypi at https://pypi.org/project/bjdata/.

One thing I noticed recently is that pip install py-ubjson can properly build/install the binary extension on both python3.8 and 3.6, but pip install bjdata only installs the binary extension on python 3.6. I don't know what was missing.

I want to get some suggestions on where to look. My setup.py can be found here. I previously blamed a pyproject.toml file that I added, but removing it does not change anything,

is this related to how the package was uploaded to pypi?

Here is the log for Python 3.8 on Ubuntu 20.04 (where ubjson module has binary but bjdata does not)

fangq@ubuntu20_04:~$ python3 -mpip install --upgrade bjdata
Collecting bjdata
  Using cached bjdata-0.3.3.tar.gz (62 kB)
Requirement already satisfied, skipping upgrade: numpy>=1.8.0 in ./.local/lib/python3.8/site-packages (from bjdata) (1.22.3)
Building wheels for collected packages: bjdata
  Building wheel for bjdata (setup.py) ... done
  Created wheel for bjdata: filename=bjdata-0.3.3-cp38-cp38-linux_x86_64.whl size=21413 sha256=9fd6168445b028c3e206b77556b1ba37696a22de0b8413ae76fe9bc3601c21ad
  Stored in directory: /home/users/fangq/.cache/pip/wheels/3f/08/4c/727407fa742f2c9affb45f384c8301464a003003cdd7709f67
Successfully built bjdata
Installing collected packages: bjdata
  Attempting uninstall: bjdata
    Found existing installation: bjdata 0.3.0
    Uninstalling bjdata-0.3.0:
      Successfully uninstalled bjdata-0.3.0
Successfully installed bjdata-0.3.3
#=======================================================================
fangq@ubuntu20_04:~$ python3 -mpip install --upgrade py-ubjson
Collecting py-ubjson
  Using cached py-ubjson-0.16.1.tar.gz (50 kB)
Building wheels for collected packages: py-ubjson
  Building wheel for py-ubjson (setup.py) ... done
  Created wheel for py-ubjson: filename=py_ubjson-0.16.1-cp38-cp38-linux_x86_64.whl size=152740 sha256=e24e4fff44e47da36179823174115a4861196d15d04cfa8b29b3c655d7989d5c
  Stored in directory: /home/users/fangq/.cache/pip/wheels/33/5a/a5/64272f6d5209c961c880545dc33eb1749dec17214e186e913a
Successfully built py-ubjson
Installing collected packages: py-ubjson
Successfully installed py-ubjson-0.16.1
#=======================================================================
fangq@ubuntu20_04:~/.local/lib/python3.8/site-packages$ lsls
total 484
drwxr-xr-x  2 fangq admin   4096 Apr 27 13:03 py_ubjson-0.16.1.dist-info
drwxr-xr-x  3 fangq admin   4096 Apr 27 13:03 ubjson
-rwxr-xr-x  1 fangq admin 465072 Apr 27 13:03 _ubjson.cpython-38-x86_64-linux-gnu.so  <- _ubjson exists, but not _bjdata
drwxr-xr-x  2 fangq admin   4096 Apr 27 13:02 bjdata-0.3.3.dist-info
drwxr-xr-x  3 fangq admin   4096 Apr 27 13:02 bjdata
...

Here is the log for Python 3.6 on Ubuntu 18.04 (both have binaries)

fangq@ubuntu18_04:~$ python3 -mpip install --upgrade bjdata
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: bjdata in /home/users/fangq/.local/lib/python3.6/site-packages (0.3.0)
Collecting bjdata
  Downloading bjdata-0.3.3.tar.gz (62 kB)
     |████████████████████████████████| 62 kB 952 kB/s             
  Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy>=1.8.0 in /home/users/fangq/.local/lib/python3.6/site-packages (from bjdata) (1.21.5)
Building wheels for collected packages: bjdata
  Building wheel for bjdata (setup.py) ... done
  Created wheel for bjdata: filename=bjdata-0.3.3-cp36-cp36m-linux_x86_64.whl size=132873 sha256=03437cb2568a9c410250e3cb51470be5b60d4dcfb24ea3e2e69806efb6fbf30e
  Stored in directory: /home/users/fangq/.cache/pip/wheels/df/b0/5c/268d37d5090f19678a54481572a83e5b83aa28da1191972d54
Successfully built bjdata
Installing collected packages: bjdata
  Attempting uninstall: bjdata
    Found existing installation: bjdata 0.3.0
    Uninstalling bjdata-0.3.0:
      Successfully uninstalled bjdata-0.3.0
Successfully installed bjdata-0.3.3
#=======================================================================
fangq@ubuntu18_04:~$ python3 -mpip install --upgrade py-ubjson
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: py-ubjson in /usr/lib/python3/dist-packages (0.8.5)
Collecting py-ubjson
  Downloading py-ubjson-0.16.1.tar.gz (50 kB)
     |████████████████████████████████| 50 kB 5.2 MB/s             
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: py-ubjson
  Building wheel for py-ubjson (setup.py) ... done
  Created wheel for py-ubjson: filename=py_ubjson-0.16.1-cp36-cp36m-linux_x86_64.whl size=115755 sha256=3df65b636d74cec86a1c815e13d7ea2ab324c70e7da5ae7b0d9365c56ac09a9c
  Stored in directory: /home/users/fangq/.cache/pip/wheels/45/9e/2b/6284eb7057635660d0f9eb410c25b045b0de0cfc40d5ce13a2
Successfully built py-ubjson
Installing collected packages: py-ubjson
Successfully installed py-ubjson-0.16.1
#=======================================================================
fangq@ubuntu18_04:~/.local/lib/python3.6/site-packages$ ls -lt
total 8232
drwxr-xr-x  2 fangq admin    4096 Apr 27 12:49 py_ubjson-0.16.1.dist-info
drwxr-xr-x  3 fangq admin    4096 Apr 27 12:49 ubjson
-rwxr-xr-x  1 fangq admin  281976 Apr 27 12:49 _ubjson.cpython-36m-x86_64-linux-gnu.so  <- both _ubjson and _bjdata are installed
drwxr-xr-x  2 fangq admin    4096 Apr 27 12:49 bjdata-0.3.3.dist-info
drwxr-xr-x  3 fangq admin    4096 Apr 27 12:49 bjdata
-rwxr-xr-x  1 fangq admin  326864 Apr 27 12:49 _bjdata.cpython-36m-x86_64-linux-gnu.so
vtermanis commented 2 years ago

For issues with pip install, I would suggest to always attempting the install with more verbose output, e.g. pip install -vU --no-cache-dir bjdata, which might show you either why compilation failed or why the pre-compiled (if there is one) binary was not installed in the right location.

If I run this myself in a fresh Python 3.8 venv, I see:

error: invalid command 'bdist_wheel' [..] ERROR: Failed building wheel for bjdata

Once I've installed the wheel package, and I run pip install -vU --no-cache-dir bjdata again, I can see:

  x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/tmp/fish/blah/include -I/usr/include/python3.8 -c src/_bjdata.c -o build/temp.linux-x86_64-3.8/src/_bjdata.o -std=c99 -DUSE__BJDATA
  x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/tmp/fish/blah/include -I/usr/include/python3.8 -c src/decoder.c -o build/temp.linux-x86_64-3.8/src/decoder.o -std=c99 -DUSE__BJDATA
  src/decoder.c:22:10: fatal error: numpy/arrayobject.h: No such file or directory
     22 | #include <numpy/arrayobject.h>
        |          ^~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  command 'x86_64-linux-gnu-gcc' failed with exit status 1
  /tmp/pip-install-z781ok7q/bjdata/setup.py:62: UserWarning: Extension module _bjdata: The output above this warning shows how the compilation failed.
    warnings.warn("Extension module %s: The output above this warning shows how the compilation failed."

Whether the above issue (lack of numpy headers / development package installation) is the same one you're having, I obviously can't tell. Hope this give you enough pointer to investigate your installation issue further.