LCOGT / lcogtsnpipe

this is the pipeline that ingest and reduce new data from the lcogt key project
MIT License
2 stars 16 forks source link

Installation via Docker fails #106

Open LydiaMak opened 2 years ago

LydiaMak commented 2 years ago

I am on macOS Monterey 12.4, Apple M1 chip and when I am trying the Docker-compose installation, I am getting this error.

` => ERROR [ 8/20] RUN pip install cryptography==2.4.1 astropy matplotlib==2.2.5 pyraf mysql-python scipy astroquery==v0.4 statsm 4.9s


[ 8/20] RUN pip install cryptography==2.4.1 astropy matplotlib==2.2.5 pyraf mysql-python scipy astroquery==v0.4 statsmodels==0.10 cython reproject:

11 0.233 DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support

11 0.804 Collecting cryptography==2.4.1

11 1.001 Downloading cryptography-2.4.1.tar.gz (468 kB)

11 1.371 Installing build dependencies: started

11 4.817 Installing build dependencies: finished with status 'error'

11 4.818 ERROR: Command errored out with exit status 1:

11 4.818 command: /usr/local/bin/python /usr/local/lib/python2.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-qAzsgo/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=18.5' wheel 'cffi>=1.7,!=1.11.3; python_implementation != '"'"'PyPy'"'"''

11 4.818 cwd: None

11 4.818 Complete output (143 lines):

11 4.818 DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support

11 4.818 Collecting setuptools>=18.5

11 4.818 Downloading setuptools-44.1.1-py2.py3-none-any.whl (583 kB)

11 4.818 Collecting wheel

11 4.818 Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)

11 4.818 Collecting cffi!=1.11.3,>=1.7

11 4.818 Downloading cffi-1.15.1.tar.gz (508 kB)

11 4.818 Collecting pycparser

11 4.818 Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)

11 4.818 Building wheels for collected packages: cffi

11 4.818 Building wheel for cffi (setup.py): started

11 4.818 Building wheel for cffi (setup.py): finished with status 'error'

11 4.818 ERROR: Command errored out with exit status 1:

11 4.818 command: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-CduPAC/cffi/setup.py'"'"'; file='"'"'/tmp/pip-install-CduPAC/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-da2mXg

11 4.818 cwd: /tmp/pip-install-CduPAC/cffi/

11 4.818 Complete output (56 lines):

11 4.818 Package libffi was not found in the pkg-config search path.

11 4.818 Perhaps you should add the directory containing `libffi.pc'

11 4.818 to the PKG_CONFIG_PATH environment variable

11 4.818 No package 'libffi' found

11 4.818 Package libffi was not found in the pkg-config search path.

11 4.818 Perhaps you should add the directory containing `libffi.pc'

11 4.818 to the PKG_CONFIG_PATH environment variable

11 4.818 No package 'libffi' found

11 4.818 Package libffi was not found in the pkg-config search path.

11 4.818 Perhaps you should add the directory containing `libffi.pc'

11 4.818 to the PKG_CONFIG_PATH environment variable

11 4.818 No package 'libffi' found

11 4.818 Package libffi was not found in the pkg-config search path.

11 4.818 Perhaps you should add the directory containing `libffi.pc'

11 4.818 to the PKG_CONFIG_PATH environment variable

11 4.818 No package 'libffi' found

11 4.818 Package libffi was not found in the pkg-config search path.

11 4.818 Perhaps you should add the directory containing `libffi.pc'

11 4.818 to the PKG_CONFIG_PATH environment variable

11 4.818 No package 'libffi' found

11 4.818 running bdist_wheel

11 4.818 running build

11 4.818 running build_py

11 4.818 creating build

11 4.818 creating build/lib.linux-aarch64-2.7

11 4.818 creating build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/cparser.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/commontypes.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/api.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/ffiplatform.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/cffi_opcode.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/pkgconfig.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/vengine_gen.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/backend_ctypes.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/recompiler.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/error.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/setuptools_ext.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/init.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/verifier.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/vengine_cpy.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/model.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/lock.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/_cffi_include.h -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/parse_c_type.h -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/_embedding.h -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/_cffi_errors.h -> build/lib.linux-aarch64-2.7/cffi

11 4.818 running build_ext

11 4.818 building '_cffi_backend' extension

11 4.818 creating build/temp.linux-aarch64-2.7

11 4.818 creating build/temp.linux-aarch64-2.7/c

11 4.818 gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DFFI_BUILDING=1 -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-aarch64-2.7/c/_cffi_backend.o

11 4.818 c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory

11 4.818 #include

11 4.818 ^

11 4.818 compilation terminated.

11 4.818 error: command 'gcc' failed with exit status 1

11 4.818 ----------------------------------------

11 4.818 ERROR: Failed building wheel for cffi

11 4.818 Running setup.py clean for cffi

11 4.818 Failed to build cffi

11 4.818 Installing collected packages: setuptools, wheel, pycparser, cffi

11 4.818 Running setup.py install for cffi: started

11 4.818 Running setup.py install for cffi: finished with status 'error'

11 4.818 ERROR: Command errored out with exit status 1:

11 4.818 command: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-CduPAC/cffi/setup.py'"'"'; file='"'"'/tmp/pip-install-CduPAC/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-pNW9Mb/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-qAzsgo/overlay --compile --install-headers /tmp/pip-build-env-qAzsgo/overlay/include/python2.7/cffi

11 4.818 cwd: /tmp/pip-install-CduPAC/cffi/

11 4.818 Complete output (56 lines):

11 4.818 Package libffi was not found in the pkg-config search path.

11 4.818 Perhaps you should add the directory containing `libffi.pc'

11 4.818 to the PKG_CONFIG_PATH environment variable

11 4.818 No package 'libffi' found

11 4.818 Package libffi was not found in the pkg-config search path.

11 4.818 Perhaps you should add the directory containing `libffi.pc'

11 4.818 to the PKG_CONFIG_PATH environment variable

11 4.818 No package 'libffi' found

11 4.818 Package libffi was not found in the pkg-config search path.

11 4.818 Perhaps you should add the directory containing `libffi.pc'

11 4.818 to the PKG_CONFIG_PATH environment variable

11 4.818 No package 'libffi' found

11 4.818 Package libffi was not found in the pkg-config search path.

11 4.818 Perhaps you should add the directory containing `libffi.pc'

11 4.818 to the PKG_CONFIG_PATH environment variable

11 4.818 No package 'libffi' found

11 4.818 Package libffi was not found in the pkg-config search path.

11 4.818 Perhaps you should add the directory containing `libffi.pc'

11 4.818 to the PKG_CONFIG_PATH environment variable

11 4.818 No package 'libffi' found

11 4.818 running install

11 4.818 running build

11 4.818 running build_py

11 4.818 creating build

11 4.818 creating build/lib.linux-aarch64-2.7

11 4.818 creating build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/cparser.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/commontypes.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/api.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/ffiplatform.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/cffi_opcode.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/pkgconfig.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/vengine_gen.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/backend_ctypes.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/recompiler.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/error.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/setuptools_ext.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/init.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/verifier.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/vengine_cpy.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/model.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/lock.py -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/_cffi_include.h -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/parse_c_type.h -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/_embedding.h -> build/lib.linux-aarch64-2.7/cffi

11 4.818 copying cffi/_cffi_errors.h -> build/lib.linux-aarch64-2.7/cffi

11 4.818 running build_ext

11 4.818 building '_cffi_backend' extension

11 4.818 creating build/temp.linux-aarch64-2.7

11 4.818 creating build/temp.linux-aarch64-2.7/c

11 4.818 gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DFFI_BUILDING=1 -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-aarch64-2.7/c/_cffi_backend.o

11 4.818 c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory

11 4.818 #include

11 4.818 ^

11 4.818 compilation terminated.

11 4.818 error: command 'gcc' failed with exit status 1

11 4.818 ----------------------------------------

11 4.818 ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-CduPAC/cffi/setup.py'"'"'; file='"'"'/tmp/pip-install-CduPAC/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-pNW9Mb/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-qAzsgo/overlay --compile --install-headers /tmp/pip-build-env-qAzsgo/overlay/include/python2.7/cffi Check the logs for full command output.

11 4.818 WARNING: You are using pip version 20.0.2; however, version 20.3.4 is available.

11 4.818 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.

11 4.818 ----------------------------------------

11 4.828 ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python2.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-qAzsgo/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=18.5' wheel 'cffi>=1.7,!=1.11.3; python_implementation != '"'"'PyPy'"'"'' Check the logs for full command output.

11 4.832 WARNING: You are using pip version 20.0.2; however, version 20.3.4 is available.

11 4.832 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.


executor failed running [/bin/sh -c pip install cryptography==2.4.1 astropy matplotlib==2.2.5 pyraf mysql-python scipy astroquery==v0.4 statsmodels==0.10 cython reproject]: exit code: 1`

I have installed libffi via brew.

Thank you,

Lydia

abostroem commented 2 years ago

If I'm reading this error correctly I think we can solve it with either specifying a specific version of pip or using the --use-deprecated flag. Thoughts @cmccully?

LydiaMak commented 2 years ago

I have fixed the issue after I changed the docker file like that:

`# To make this a 32 bit version python:2.7.18-slim-stretch -> i386/python:2.7.18-slim-stretch FROM python:2.7.18-slim-stretch

ENV iraf /iraf/iraf/

To make this a 32 bit version linux64 -> linux

ENV IRAFARCH linux64

RUN apt-get update \ && apt -y install gcc make flex git \ && apt-get install -y libffi-dev libssl-dev \ && apt -y install libcurl4-openssl-dev libexpat-dev libreadline-dev gettext \ && apt -y install libopenblas-dev liblapack-dev \ && apt-get autoclean \ && rm -rf /var/lib/apt/lists/*

RUN mkdir -p $iraf \ && cd /iraf \ && git clone https://github.com/iraf-community/iraf.git \ && cd $iraf \ && git checkout ba22d13 \ && ./install < /dev/null \ && make $IRAFARCH \ && make sysgen

RUN apt-get update \ && apt-get -y install libx11-dev libcfitsio-bin wget x11-apps libtk8.6 sextractor procps g++ \ mysql-client libmariadbclient-dev openssh-client wcstools libxml2 vim libssl1.0.2 zip pkg-config \ libpng-dev libfreetype6-dev libcfitsio-dev \ && apt-get autoclean \ && rm -rf /var/lib/apt/lists/*

RUN apt-get update \ && apt-get -y install autoconf \ && apt-get -y install automake \ && apt-get -y install libtool-bin \ && cd / \ && git clone https://github.com/astromatic/swarp.git \ && cd swarp \ && ./autogen.sh \ && ./configure \ && make \ && ln -s /swarp/src/swarp /usr/bin/

RUN ln -s /usr/bin/sextractor /usr/bin/sex

RUN python -m pip install --upgrade pip

RUN pip install numpy>=1.12 RUN pip install cryptography==2.4.1 astropy matplotlib==2.2.5 pyraf mysql-python scipy astroquery==v0.4 statsmodels==0.10 cython reproject

RUN pip install sep==1.0.3 git+https://github.com/dguevel/PyZOGY.git && rm -rf ~/.cache/pip

RUN wget http://ds9.si.edu/download/debian9/ds9.debian9.8.2.tar.gz \ && tar -xzvf ds9.debian9.8.2.tar.gz -C /usr/local/bin \ && rm -rf ds9.debian9.8.2.tar.gz

RUN wget http://cdsarc.u-strasbg.fr/ftp/pub/sw/cdsclient.tar.gz \ && tar -xzvf cdsclient.tar.gz -C /usr/src && rm cdsclient.tar.gz \ && cd /usr/src/cdsclient-* && ./configure && make && make install

RUN cd / \ && git clone https://github.com/acbecker/hotpants.git \ && cd hotpants \ && make \ && ln -s /hotpants/hotpants /usr/bin/

ENV LCOSNPIPE /lcogtsnpipe

RUN mkdir -p /home/supernova/iraf && /usr/sbin/groupadd -g 20000 "domainusers" \ && /usr/sbin/useradd -g 20000 -d /home/supernova -M -N -u 10197 supernova \ && chown -R supernova:domainusers /home/supernova \ && mkdir -p $LCOSNPIPE

RUN chown -R supernova:domainusers $LCOSNPIPE /usr/local

USER supernova

COPY --chown=supernova:domainusers . $LCOSNPIPE

WORKDIR $LCOSNPIPE/trunk

RUN python setup.py build -f && python setup.py install -f

WORKDIR /home/supernova/iraf

RUN mkiraf --term=xgterm -i

WORKDIR /home/supernova

ENTRYPOINT /bin/bash `

Thank you,

Lydia

abostroem commented 2 years ago

reopening so we can remember to fix this for everyone else

LydiaMak commented 2 years ago

To add on this, I also had to re-build the docker image like this:

docker build --platform=linux/amd64 -t lcogtsnpipe lcogtsnpipe

otherwise I was getting

qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

when I was trying to open ds9.