search

bsolomon1124 / pycld3

Python3 bindings for the Compact Language Detector v3 (CLD3)
Apache License 2.0
149 stars 6 forks source link

Cannot install: Linux, Anaconda most recent #6

Closed ma-ji closed 4 years ago

ma-ji commented 4 years ago

All packages should be up-to-date. Following your instruction on installing dependencies. No problem with pycld2. Logs:

Collecting pycld3
  Using cached https://files.pythonhosted.org/packages/8a/48/41bdbadcc407a8eb23337b9ad2f0ec3aab96871a5b6e1017983ba400dad6/pycld3-0.19.tar.gz
Building wheels for collected packages: pycld3
  Building wheel for pycld3 (setup.py) ... error
  ERROR: Complete output from command /root/anaconda3/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-0sgvk93c/pycld3/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-sofl_g_s --python-tag cp37:
  ERROR: running bdist_wheel
  running build
  Creating dirs at /tmp/pip-install-0sgvk93c/pycld3/src/cld_3/protos/
  Running protoc sentence.proto feature_extractor.proto task_spec.proto --cpp_out=/tmp/pip-install-0sgvk93c/pycld3/src/cld_3/protos/
  running build_ext
  building 'cld3' extension
  creating build
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/src
  creating build/temp.linux-x86_64-3.7/src/cld_3
  creating build/temp.linux-x86_64-3.7/src/cld_3/protos
  creating build/temp.linux-x86_64-3.7/src/script_span
  gcc -pthread -B /root/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/local/include/ -I/tmp/pip-install-0sgvk93c/pycld3/src/ -I/tmp/pip-install-0sgvk93c/pycld3/src/cld_3/protos/ -I/root/anaconda3/include/python3.7m -c pycld3.cpp -o build/temp.linux-x86_64-3.7/pycld3.o -std=c++11
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
  In file included from /tmp/pip-install-0sgvk93c/pycld3/src/feature_extractor.h:45:0,
                   from /tmp/pip-install-0sgvk93c/pycld3/src/embedding_feature_extractor.h:23,
                   from /tmp/pip-install-0sgvk93c/pycld3/src/nnet_language_identifier.h:22,
                   from pycld3.cpp:675:
  /tmp/pip-install-0sgvk93c/pycld3/src/cld_3/protos/feature_extractor.pb.h:10:10: fatal error: google/protobuf/port_def.inc: No such file or directory
   #include <google/protobuf/port_def.inc>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for pycld3
  Running setup.py clean for pycld3
Failed to build pycld3
Installing collected packages: pycld3
  Running setup.py install for pycld3 ... error
    ERROR: Complete output from command /root/anaconda3/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-0sgvk93c/pycld3/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-hzi7skf7/install-record.txt --single-version-externally-managed --compile:
    ERROR: running install
    running build
    Running protoc sentence.proto feature_extractor.proto task_spec.proto --cpp_out=/tmp/pip-install-0sgvk93c/pycld3/src/cld_3/protos/
    running build_ext
    building 'cld3' extension
    creating build
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/src
    creating build/temp.linux-x86_64-3.7/src/cld_3
    creating build/temp.linux-x86_64-3.7/src/cld_3/protos
    creating build/temp.linux-x86_64-3.7/src/script_span
    gcc -pthread -B /root/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/local/include/ -I/tmp/pip-install-0sgvk93c/pycld3/src/ -I/tmp/pip-install-0sgvk93c/pycld3/src/cld_3/protos/ -I/root/anaconda3/include/python3.7m -c pycld3.cpp -o build/temp.linux-x86_64-3.7/pycld3.o -std=c++11
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    In file included from /tmp/pip-install-0sgvk93c/pycld3/src/feature_extractor.h:45:0,
                     from /tmp/pip-install-0sgvk93c/pycld3/src/embedding_feature_extractor.h:23,
                     from /tmp/pip-install-0sgvk93c/pycld3/src/nnet_language_identifier.h:22,
                     from pycld3.cpp:675:
    /tmp/pip-install-0sgvk93c/pycld3/src/cld_3/protos/feature_extractor.pb.h:10:10: fatal error: google/protobuf/port_def.inc: No such file or directory
     #include <google/protobuf/port_def.inc>
              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command "/root/anaconda3/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-0sgvk93c/pycld3/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-hzi7skf7/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-0sgvk93c/pycld3/

Thank you very much!

bsolomon1124 commented 4 years ago

Hi @ma-ji , sorry for the trouble. What flavor/version of Linux? (E.g. Ubuntu 18.04) and version of Python 3?

ma-ji commented 4 years ago 
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Python: Python 3.7.6.

I was running the code in this order:

  1. sudo apt-get update
  2. sudo apt-get install protobuf-compiler libprotobuf-dev
  3. python -m pip install pycld3

No need to be sorry about, appreciate your work!

bsolomon1124 commented 4 years ago

I'm having trouble reproducing this one, but will keep investigating. Granted, the Python image below uses Debian 10, though Ubuntu is Debian-based.

 [~/] $ docker run -it --rm --entrypoint=/bin/bash python:3.7.6
root@xxxxxxxxxxxx:/# apt-get update -y \
> && apt-get install -y --no-install-recommends protobuf-compiler libprotobuf-dev && \
> python -m pip install pycld3
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [49.3 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [177 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [5792 B]
Fetched 8326 kB in 4s (2278 kB/s)                         
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libprotobuf-lite17 libprotobuf17 libprotoc17
The following NEW packages will be installed:
  libprotobuf-dev libprotobuf-lite17 libprotobuf17 libprotoc17 protobuf-compiler
0 upgraded, 5 newly installed, 0 to remove and 29 not upgraded.
Need to get 2840 kB of archives.
After this operation, 16.6 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 libprotobuf17 amd64 3.6.1.3-2 [814 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libprotobuf-lite17 amd64 3.6.1.3-2 [167 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 libprotobuf-dev amd64 3.6.1.3-2 [1130 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 libprotoc17 amd64 3.6.1.3-2 [665 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 protobuf-compiler amd64 3.6.1.3-2 [64.9 kB]
Fetched 2840 kB in 1s (5029 kB/s)            
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libprotobuf17:amd64.
(Reading database ... 24593 files and directories currently installed.)
Preparing to unpack .../libprotobuf17_3.6.1.3-2_amd64.deb ...
Unpacking libprotobuf17:amd64 (3.6.1.3-2) ...
Selecting previously unselected package libprotobuf-lite17:amd64.
Preparing to unpack .../libprotobuf-lite17_3.6.1.3-2_amd64.deb ...
Unpacking libprotobuf-lite17:amd64 (3.6.1.3-2) ...
Selecting previously unselected package libprotobuf-dev:amd64.
Preparing to unpack .../libprotobuf-dev_3.6.1.3-2_amd64.deb ...
Unpacking libprotobuf-dev:amd64 (3.6.1.3-2) ...
Selecting previously unselected package libprotoc17:amd64.
Preparing to unpack .../libprotoc17_3.6.1.3-2_amd64.deb ...
Unpacking libprotoc17:amd64 (3.6.1.3-2) ...
Selecting previously unselected package protobuf-compiler.
Preparing to unpack .../protobuf-compiler_3.6.1.3-2_amd64.deb ...
Unpacking protobuf-compiler (3.6.1.3-2) ...
Setting up libprotobuf-lite17:amd64 (3.6.1.3-2) ...
Setting up libprotobuf17:amd64 (3.6.1.3-2) ...
Setting up libprotoc17:amd64 (3.6.1.3-2) ...
Setting up protobuf-compiler (3.6.1.3-2) ...
Setting up libprotobuf-dev:amd64 (3.6.1.3-2) ...
Processing triggers for libc-bin (2.28-10) ...
Collecting pycld3
  Downloading pycld3-0.19.tar.gz (691 kB)
     |████████████████████████████████| 691 kB 4.8 MB/s 
Building wheels for collected packages: pycld3
  Building wheel for pycld3 (setup.py) ... done
  Created wheel for pycld3: filename=pycld3-0.19-cp37-cp37m-linux_x86_64.whl size=2846587 sha256=53c931783fabca26c6e5e379b8e005520d790bd290e3abe68c8f45c3c715230f
  Stored in directory: /root/.cache/pip/wheels/07/ae/30/bdeddc2dcb3da00119faf2d1467212e65379a82cd30a28a15d
Successfully built pycld3
Installing collected packages: pycld3
Successfully installed pycld3-0.19
root@b304c53cb058:/# python -c 'import cld3; print(cld3.get_language("影響包含對氣候的變化以及自然資源的枯竭程度"))'
LanguagePrediction(language='zh', probability=0.999969482421875, is_reliable=True, proportion=1.0)

Also tried installing Anaconda via wget https://repo.anaconda.com/archive/Anaconda3-2019.10-Linux-x86_64.sh && bash Anaconda3-2019.10-Linux-x86_64.sh and then using conda's pip, and the install still seemed to work.

bsolomon1124 commented 4 years ago

Do you have Cython installed @ma-ji ?

ma-ji commented 4 years ago

Yes, I have Cython installed through conda forge. I'll try another time after rebooting my machine. It is still running jobs. Thanks for investigating this!

Rashwan commented 4 years ago

@ma-ji I have the same problem, did you solve yours?

ma-ji commented 4 years ago

@ma-ji I have the same problem, did you solve yours?

Unfortunately, no... Which is good: gives us more confidence this is a real issue!

bsolomon1124 commented 4 years ago

For anyone still having problems here - the gcc traceback says that <google/protobuf/port_def.inc> isn't found in an include directive, though that file has existed in the protobuf headers for some time. So, one step for anyone still having this problem would be to check where the headers are installed to. You could probably find that with

# https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/map_field.cc
sudo find / -name 'map_field.cc' -type f -print 2>/dev/null

Should find that protobuf header file if it exists.

If it does, you can try setting CPLUS_INCLUDE_PATH to mimic the gcc -I option to help gcc find those files. If not, then that would probably imply that the header files are actually missing.

ma-ji commented 4 years ago

Hi @bsolomon1124 , I didn't find the file you mentioned. What should I do? Sorry I'm just a naive social science researcher knowing copy-paste codes.

bsolomon1124 commented 4 years ago

Sorry, my mistake @ma-ji , can you actually check for:

sudo find / -name 'unknown_field_set.h' -type f -print 2>/dev/null
ma-ji commented 4 years ago

@bsolomon1124 I didn't find the map_field.cc following your code. Sorry for not being clear about my problem: I don't know how to deal with the problem thereafter.

Here is what I've got after executing your unknown_field_set.h after trying to install protobuf:

(base) root@mj-clake:~# sudo find / -name 'unknown_field_set.h' -type f -print 2>/dev/null
/usr/local/include/google/protobuf/unknown_field_set.h
/usr/include/google/protobuf/unknown_field_set.h
/root/anaconda3/include/google/protobuf/unknown_field_set.h
/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/include/google/protobuf/unknown_field_set.h
/root/anaconda3/pkgs/libprotobuf-3.11.4-hd408876_0/include/google/protobuf/unknown_field_set.h
/root/anaconda3/pkgs/tensorflow-base-1.14.0-gpu_py37he45bfe2_0/lib/python3.7/site-packages/tensorflow/include/google/protobuf/unknown_field_set.h
/root/anaconda3/pkgs/tensorflow-base-1.14.0-gpu_py37he45bfe2_0/lib/python3.7/site-packages/tensorflow/include/external/protobuf_archive/src/google/protobuf/unknown_field_set.h
/root/anaconda3/pkgs/libprotobuf-3.8.0-hd408876_0/include/google/protobuf/unknown_field_set.h
/root/anaconda3/pkgs/libprotobuf-3.11.3-hd408876_0/include/google/protobuf/unknown_field_set.h
/root/anaconda3/pkgs/libprotobuf-3.9.1-h8b12597_0/include/google/protobuf/unknown_field_set.h
(base) root@mj-clake:~#

Now while I'm trying to install pycld3, it gives me this error:

    ERROR: Command errored out with exit status 1:
     command: /root/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-iceocpwa/pycld3/setup.py'"'"'; __file__='"'"'/tmp/pip-install-iceocpwa/pycld3/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-_q3do0_6/install-record.txt --single-version-externally-managed --compile --install-headers /root/anaconda3/include/python3.7m/pycld3

and this:

    /tmp/pip-install-iceocpwa/pycld3/src/cld_3/protos/sentence.pb.h:298:55: note:   couldn't deduce template parameter ‘T’
         return _internal_metadata_.mutable_unknown_fields();
                                                           ^
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /root/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-iceocpwa/pycld3/setup.py'"'"'; __file__='"'"'/tmp/pip-install-iceocpwa/pycld3/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-_q3do0_6/install-record.txt --single-version-externally-managed --compile --install-headers /root/anaconda3/include/python3.7m/pycld3 Check the logs for full command output.

Hope this helps and appreciate your time!

bsolomon1124 commented 4 years ago

Hi @ma-ji , thank you for your patience.

I've uploaded some Linux wheels to Test PyPI. They are 'manylinux' wheels so should work on almost any x86-64 Linux variant and should not require protobuf at all. Could you be kind enough to try this on your end? Should be as simple as:

$ pip install --index-url https://test.pypi.org/simple/ 'pycld3==0.19'
$ python -c 'import cld3; print(cld3.get_language("影響包含對氣候的變化以及自然資源的枯竭程度"))'

This should install the built wheel from TestPyPI rather than production PyPI. And in the pip install output, you should see the .whl being downloaded rather than a tar.gz source distribution.

ma-ji commented 4 years ago

Thanks @bsolomon1124 , that's a very effective solution! It works perfectly now. Appreciate your time in this project!