search

Ultimaker / cura-build-environment

CMake project to build dependencies for Cura
GNU Affero General Public License v3.0
24 stars 55 forks source link

Dockerfile is broken #57

Closed bebuch closed 5 years ago

bebuch commented 5 years ago

The Dockerfile in the Cura project doesn't build a container that is able to execute Cura.

Reproduce by:

git clone https://github.com/Ultimaker/Cura.git
cd Cura
docker build -t cura .
docker run cura

The last command fails with:

$ docker run cura
Error in sys.excepthook:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/srv/cura/Cura/cura/CuraApplication.py", line 48, in <module>
    from cura.API import CuraAPI
  File "/srv/cura/Cura/cura/API/__init__.py", line 9, in <module>
    from cura.API.Account import Account
  File "/srv/cura/Cura/cura/API/Account.py", line 11, in <module>
    from cura.OAuth2.AuthorizationService import AuthorizationService
  File "/srv/cura/Cura/cura/OAuth2/AuthorizationService.py", line 12, in <module>
    from cura.OAuth2.AuthorizationHelpers import AuthorizationHelpers
  File "/srv/cura/Cura/cura/OAuth2/AuthorizationHelpers.py", line 9, in <module>
    import requests
  File "/usr/local/lib/python3.5/site-packages/requests/__init__.py", line 43, in <module>
    import urllib3
ImportError: No module named 'urllib3'

Original exception was:
Traceback (most recent call last):
  File "cura_app.py", line 129, in <module>
    from cura.CuraApplication import CuraApplication
  File "/srv/cura/Cura/cura/CuraApplication.py", line 48, in <module>
    from cura.API import CuraAPI
  File "/srv/cura/Cura/cura/API/__init__.py", line 9, in <module>
    from cura.API.Account import Account
  File "/srv/cura/Cura/cura/API/Account.py", line 11, in <module>
    from cura.OAuth2.AuthorizationService import AuthorizationService
  File "/srv/cura/Cura/cura/OAuth2/AuthorizationService.py", line 12, in <module>
    from cura.OAuth2.AuthorizationHelpers import AuthorizationHelpers
  File "/srv/cura/Cura/cura/OAuth2/AuthorizationHelpers.py", line 9, in <module>
    import requests
  File "/usr/local/lib/python3.5/site-packages/requests/__init__.py", line 43, in <module>
    import urllib3
ImportError: No module named 'urllib3'

This was reported in Cura Issue 4950. I think the real Issue is here in the build environment.

I already tried to install urllib3 inside the container by yum install -y python-urllib3, but it didn't work. (Same error message.)

Via Google I found that the urllib3 Centos 7 is very old and the only proper way to install a working version there is using pip. (I didn't verify yet if this is correct.)

LipuFei commented 5 years ago

Hi @bebuch , we install python urllib3 separately via cura-build-environment: https://github.com/Ultimaker/cura-build-environment/blob/3a4a96ae80ea104dea820d5c5c7b3efdc74488cd/projects/python.cmake#L151 The docker image ultimaker/cura-build-environment:1 on Docker Hub could be old. I'm not certain if it works, but you could try building a new cura-build-environment docker image and use that to build and run cura docker image.

LipuFei commented 5 years ago

Just a realized that a quicker way to check is to run your cura container manually by overriding the entrypoint to /bin/bash, then check which python3 is by default used. It should be the one in /usr/loca/bin. If not, make sure that /usr/local/bin/python3 is used. If the problem still persists, you can manually install urllib3 via /usr/local/bin/python3 -m pip install urllib3 and then try running Cura again.

bebuch commented 5 years ago

Thanks for the fast reply! The python3 binary is the correct one in /usr/local/bin. I was able to install some more needed python dependencies via:

pip3 install urllib3 chardet certifi idna

Now shapely is missing, which fails to be installed via pip:

# /usr/local/bin/python3 -m pip install shapely
Collecting shapely
  Downloading https://files.pythonhosted.org/packages/a2/fb/7a7af9ef7a35d16fa23b127abee272cfc483ca89029b73e92e93cdf36e6b/Shapely-1.6.4.post2.tar.gz (225kB)
    100% |████████████████████████████████| 235kB 798kB/s 
    Complete output from command python setup.py egg_info:
    Failed `CDLL(libgeos_c.so.1)`
    Failed `CDLL(libgeos_c.so)`
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-pgzycm_r/shapely/setup.py", line 80, in <module>
        from shapely._buildcfg import geos_version_string, geos_version, \
      File "/tmp/pip-build-pgzycm_r/shapely/shapely/_buildcfg.py", line 167, in <module>
        fallbacks=['libgeos_c.so.1', 'libgeos_c.so'])
      File "/tmp/pip-build-pgzycm_r/shapely/shapely/_buildcfg.py", line 161, in load_dll
        libname, fallbacks or []))
    OSError: Could not find library geos_c or load any of its variants ['libgeos_c.so.1', 'libgeos_c.so']

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-pgzycm_r/shapely/
You are using pip version 8.1.1, however version 19.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

I don't think it can be easily fixed within using the old cura-build-environment version from Dockerhub.

Building the current version of cura-build-environment is already in progress. ;-)

LipuFei commented 5 years ago

Hi @bebuch , I think ultimaker/cura-build-environment:1 is probably too old. It's from one year ago. Hope the current version works.

bebuch commented 5 years ago

It fails to build:

Step 25/28 : RUN make
 ---> Running in 1c004583f891
[…]
Scanning dependencies of target sqlite3
[ 42%] Creating directories for 'sqlite3'
[ 42%] Performing download step (download, verify and extract) for 'sqlite3'
-- Downloading...
   dst='/srv/cura-build-environment/cura-build-environment/build/sqlite3-prefix/src/sqlite-autoconf-3260000.tar.gz'
   timeout='none'
-- Using src='https://sqlite.org/2018/sqlite-autoconf-3260000.tar.gz'
-- [download 0% complete]
[…]
-- [download 100% complete]
-- verifying file...
       file='/srv/cura-build-environment/cura-build-environment/build/sqlite3-prefix/src/sqlite-autoconf-3260000.tar.gz'
-- Downloading... done
-- extracting...
     src='/srv/cura-build-environment/cura-build-environment/build/sqlite3-prefix/src/sqlite-autoconf-3260000.tar.gz'
     dst='/srv/cura-build-environment/cura-build-environment/build/sqlite3-prefix/src/sqlite3'
-- extracting... [tar xfz]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done
[ 43%] Performing patch step for 'sqlite3'
/bin/sh: libtoolize: command not found
make[2]: *** [sqlite3-prefix/src/sqlite3-stamp/sqlite3-patch] Error 127                                                                                                                                                              
make[1]: *** [CMakeFiles/sqlite3.dir/all] Error 2                                                                                                                                                                                    
make: *** [all] Error 2                                                                                                                                                                                                              
The command '/bin/sh -c make' returned a non-zero code: 2

By the way, do you have the possibility to set up a Continuous Integration process that builds it once a week and, if successful, uploads it to Dockerhub? That would help a lot.

LipuFei commented 5 years ago

@bebuch The Dockerfile for master is missing libtool. You can try tag 4.0, which doesn't not have this problem. Sorry about that, the whole build process takes a long time...

For setting up a CI process, we plan to redesign the current build process because it's not efficient, so we may not set up a temporary CI process with the current build env repo. We will discuss this later and let you know.

bebuch commented 5 years ago

Sounds good and thanks for your help so far!

bebuch commented 5 years ago

After some more investigation I found the the current master is missing the packages libtool and gcc-gfortran. After fixing that, the configuration of libSavitar and libArcus fails with:

[ 58%] Performing configure step for 'Savitar'
-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.9.2
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /opt/rh/devtoolset-3/root/usr/bin/c++
-- Check for working CXX compiler: /opt/rh/devtoolset-3/root/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at /usr/share/cmake3/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find PythonInterp: Found unsuitable version "1.4", but required
  is at least "3.4.0" (found /usr/bin/python3.4)
Call Stack (most recent call first):
  /usr/share/cmake3/Modules/FindPackageHandleStandardArgs.cmake:376 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake3/Modules/FindPythonInterp.cmake:159 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  cmake/FindSIP.cmake:32 (find_package)
  CMakeLists.txt:15 (find_package)

-- Configuring incomplete, errors occurred!
See also "/srv/cura-build-environment/cura-build-environment/build/Savitar-prefix/src/Savitar-build/CMakeFiles/CMakeOutput.log".
make[2]: *** [Savitar-prefix/src/Savitar-stamp/Savitar-configure] Error 1
make[1]: *** [CMakeFiles/Savitar.dir/all] Error 2
make: *** [all] Error 2
[ 75%] Performing configure step for 'Arcus'
CMake Error at /usr/share/cmake3/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find PythonInterp: Found unsuitable version "1.4", but required
  is at least "3.4.0" (found /usr/bin/python3.4)
Call Stack (most recent call first):
  /usr/share/cmake3/Modules/FindPackageHandleStandardArgs.cmake:376 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake3/Modules/FindPythonInterp.cmake:159 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  cmake/FindSIP.cmake:27 (find_package)
  CMakeLists.txt:25 (find_package)

-- Configuring incomplete, errors occurred!
See also "/srv/cura-build-environment/cura-build-environment/build/Arcus-prefix/src/Arcus-build/CMakeFiles/CMakeOutput.log".
See also "/srv/cura-build-environment/cura-build-environment/build/Arcus-prefix/src/Arcus-build/CMakeFiles/CMakeError.log".
make[2]: *** [Arcus-prefix/src/Arcus-stamp/Arcus-configure] Error 1
make[1]: *** [CMakeFiles/Arcus.dir/all] Error 2
make: *** [all] Error 2

It seem that find_package(PythonInterp 3.4.0 REQUIRED) from libSavitar's FindSIP.cmake does choose the wrong python executable. I couldn't fix that.

By the way, the installed cmake is 3.13.1 which recommends using FindPython3 instead of PythonInterp.

# cmake3  --version
cmake3 version 3.13.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).
LipuFei commented 5 years ago

Hi @bebuch , we plan to make this Dockerfile work first before we migrate other repositories to GitLab, so we will have a working build environment docker image to work with. Fancy stuff will come later.

For dev, JIRA ticket: CURA-6187

LipuFei commented 5 years ago

CURA-6187 has been finished. There's a new Dockerfile now.