Docker image for simex - Githubissues

cloudbustinguk commented 3 years ago

Working on a few refactors of the simex_install.sh install script:

finishing deferred symbolic links:
  SimEx-develop/Tests/python/functest/SimExTest/Backengines/WPG/paths.py -> ../../../../paths.origin.py
  SimEx-develop/Tests/python/functest/TestFiles -> ../unittest/TestFiles
  SimEx-develop/Tests/python/functest/TestUtilities -> ../unittest/TestUtilities/
  SimEx-develop/Tests/python/functest/paths.py -> ../paths.origin.py
/opt/simex_install.sh: line 13: cd: simex_platform-develop: No such file or directory
Error: error building at STEP "RUN bash /opt/simex_install.sh": error while running runtime: exit status 1

cloudbustinguk commented 3 years ago

There appears to be some issues with the python installation:

We have an older version of Miniconda, which is 2.7. The installation scipts for SimEx call for many modules, some of which there are no wheels available, and must be built. The builds break due to using 3.x syntax:

    File "/tmp/pip-install-eC4mRp/llvmlite/ffi/build.py", line 122                                                                                                                                               
      raise ValueError(msg.format(_ver_check_skip)) from e                                                                                                                                                                                                              ^                                                                                                                                                         
  SyntaxError: invalid syntax                                                                                                                                                                                    
  error: command '/opt/miniconda/bin/python' failed with exit status 1

and

 File "breathe/__init__.py", line 10                                                                                                                                                                        
        def setup(app: Sphinx):                                                                                                                                                                                                       ^                                                                                                                                                                                           
    SyntaxError: invalid syntax

and

on/pip-egg-info                                                                                                                                                                                                
         cwd: /tmp/pip-install-hhb84U/biopython/                                                                                                                                                                 
    Complete output (1 lines):                                                                                                                                                                                   
    Biopython requires Python 3.6 or later. Python 2.7 detected.

etc..

A known problem with Anaconda:

Traceback (most recent call last):                                                                                                                                                                               
  File "/opt/miniconda/bin/conda", line 7, in <module>                                                                                                                                                           
    from conda.cli import main                                                                                                                                                                                   
ImportError: No module named conda.cli

.. to which Stackoverflow answers point to incorrect major versions, pathing issues, etc.

Will continue to work on resolving these.

JunCEEE commented 3 years ago

Does completely switching to python 3.x solve all the above issues?

cloudbustinguk commented 3 years ago

Does completely switching to python 3.x solve all the above issues?

Yep - will form a mini list within this issue, to deal with the new issues which have arisen.

cloudbustinguk commented 3 years ago

Issue list:

[x] HDF5 Libraries missing in base image
[x] flex/bison missing
[x] CMake version bump needed

CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
CMake 3.12 or higher is required.  You are running version 2.8.12.2

[x] Workaround PEP517 module issues

ERROR: Could not build wheels for pyFAI which use PEP 517 and cannot be installed directly

RESOLUTION: Bump requirements.txt's PyFAI to <= 0.17 (wheel is provided, so no build necessary). According to @ejcjason, 0.18+ contains a deprecation warning for various functions, to 0.17 should be a safe bet.

[x] HDF5 Libraries not found in CMake Docker build, but found inside container build Strange issue - something in the Dockerfile is throwing CMake off on the full build, but when spawning the failed container and shelling into it, CMake finds the libraries no problem.

CHECKING Sources/GenesisPhotonSource                                           
-- ***** Including module S2EReconstruction_DM from: Analyzers/S2EReconstruction_DM ****                                                                      
-- Could NOT find MKL (missing: MKL_ROOT) 
-- Found FFTW: /usr/lib64/libfftw3.so                                          
FFTW_INCLUDES=/usr/include                                                     
FFTW_LIBRARY_DIR=/usr/lib64                                                                                                                                   
-- *********************************************************                   
-- ***** Including module S2EReconstruction_EMC from: Analyzers/S2EReconstruction_EMC ****
-- Python module Cython.Distutils is found                                     
-- Python module numpy is found 
-- Python module h5py is found
-- *********************************************************                   
-- ***** Including module CrystFELPhotonDiffractor from: Diffractors/CrystFELPhotonDiffractor ****                                                            
CMake Error at /usr/share/cmake3/Modules/FindPackageHandleStandardArgs.cmake:164 (message):                                                                   
  Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_INCLUDE_DIRS C CXX)        
  (found version "")           
Call Stack (most recent call first):                                           
  /usr/share/cmake3/Modules/FindPackageHandleStandardArgs.cmake:445 (_FPHSA_FAILURE_MESSAGE)                                                                  
  /usr/share/cmake3/Modules/FindHDF5.cmake:929 (find_package_handle_standard_args)                                                                            
  Modules/Diffractors/CrystFELPhotonDiffractor/CMakeLists.txt:6 (FIND_PACKAGE)

Can get additional debug with HDF5_FIND_DEBUG as per here

RESOLUTION Errant HDF5_ROOT variable in install_simex.sh was pointing to conda (assuming it was expected that picking up the shared objects from h5py was enough? CMake did not think so). Note, that as an earlier issue, we installed the RPM of hdf5-devel, and so by removing the HDF5_ROOT var, we allowed FindHDF5 to do its job.

[x] PyQT4 issues with 3.x conda

PackagesNotFoundError: The following packages are not available from current channels:                                                                                                                           

  - pyqt=4

As confirmed my @ejcjason, PyQT5 is fine

cloudbustinguk commented 3 years ago

Ok, dockerhub/github are up:

DockerHub GitHub

JunCEEE commented 3 years ago

Wait, do we have the pyqt=4 restriction? I'm using pyqt=5.9.2 in my testing environment without problems.

cloudbustinguk commented 3 years ago

Wait, do we have the pyqt=4 restriction? I'm using pyqt=5.9.2 in my testing environment without problems.

The code runs fine with PyQT5? This is great news - it's causing trouble to activate a py36 conda just so that we can have PyQT4.

cloudbustinguk commented 3 years ago

@ejcjason You could already try and pull from the DockerHub to see if it works:

docker pull cloudbusting/simex-cloudbusting

Would be happy for your feedback to see if anything needs work.

Beware - the image is a monster at ~6GB (without test data)

shervin86 commented 3 years ago

@cloudbustinguk to which branch did you push your changes?

cloudbustinguk commented 3 years ago

Ah, it's only pushed to my own github - I informed @ejcjason to test it directly from dockerhub. I would like to make a MR once it's confirmed as working.

shervin86 commented 3 years ago

I just tried to pull from the hub, but I can't help in testing... 3.4 GB in my docker/ directory

shervin86 commented 3 years ago

do you mind putting your changes in the branch "shervin" ? This way we can merge it as soon as it is tested

cloudbustinguk commented 3 years ago

I just tried to pull from the hub, but I can't help in testing... 3.4 GB in my docker/ directory

yes it's a big one!

cloudbustinguk commented 3 years ago

do you mind putting your changes in the branch "shervin" ? This way we can merge it as soon as it is tested

Done! Please see PR: https://github.com/PaNOSC-ViNYL/SimEx/pull/226

cloudbustinguk commented 3 years ago

@shervin86 would you mind testing the new image, and I will push changes on success?

docker pull cloudbusting/simex

cloudbustinguk commented 3 years ago

@shervin86

Hi Shervin - I've just pushed a new image to the dockerhub. This should fix the SDF issue (not being installed to the expected path).

Please first purge any old pods running/old images:

podman stop $(podman ps -q) ; podman rm $(podman ps -qa) ; podman rmi $(podman images -q)

Then simply attempt to run the new pod:

podman run -it cloudbusting/simex

Cheers

Dan

PaNOSC-ViNYL / ViNYL-project

Docker image for simex #50