2 error when install pybufr-ecmwf on ubuntu

[ZhangAllen98]

Executing command: python3 ./run_f2py_tool.py --build-dir f2py_build --f90flags='-fno-second-underscore -O -Dlinux -fPIC' --f77flags='-fno-second-underscore -O -Dlinux -fPIC' --fcompiler=gfortran ./f2py_build/signatures.pyf -L./ -lbufr -c 2>1 >> f2py_build.log ERROR: build of python wrapper failed the compilation or linking stage failed .................................................................. .................................................................. .......................................................... Executing command: python3 ./run_f2py_tool.py --build-dir f2py_build --f90flags='-fno-second-underscore -O -Dlinux -fPIC' --f77flags='-fno-second-underscore -O -Dlinux -fPIC' --fcompiler=gfortran ./f2py_build/signatures.pyf -L./ -lbufr -c 2>1 >> f2py_build.log ERROR: build of python wrapper failed the compilation or linking stage failed Traceback (most recent call last): File "", line 1, in File "/tmp/pip-build-3dy3x23o/pybufr-ecmwf/setup.py", line 410, in provides=["pybufr_ecmwf"]) File "/home/zhangqw/anaconda3/lib/python3.5/distutils/core.py", line 148, in setup dist.run_commands() File "/home/zhangqw/anaconda3/lib/python3.5/distutils/dist.py", line 955, in run_commands self.run_command(cmd) File "/home/zhangqw/anaconda3/lib/python3.5/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/home/zhangqw/anaconda3/lib/python3.5/site-packages/setuptools/command/install.py", line 61, in run return orig.install.run(self) File "/home/zhangqw/anaconda3/lib/python3.5/distutils/command/install.py", line 539, in run self.run_command('build') File "/home/zhangqw/anaconda3/lib/python3.5/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/zhangqw/anaconda3/lib/python3.5/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/tmp/pip-build-3dy3x23o/pybufr-ecmwf/setup.py", line 119, in run _build.run(self) File "/home/zhangqw/anaconda3/lib/python3.5/distutils/command/build.py", line 135, in run self.run_command(cmd_name) File "/home/zhangqw/anaconda3/lib/python3.5/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/zhangqw/anaconda3/lib/python3.5/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/home/zhangqw/anaconda3/lib/python3.5/distutils/command/build_ext.py", line 339, in run self.build_extensions() File "/home/zhangqw/anaconda3/lib/python3.5/distutils/command/build_ext.py", line 448, in build_extensions self._build_extensions_serial() File "/home/zhangqw/anaconda3/lib/python3.5/distutils/command/build_ext.py", line 473, in _build_extensions_serial self.build_extension(ext) File "/tmp/pip-build-3dy3x23o/pybufr-ecmwf/setup.py", line 318, in build_extension ibi.build() File "/tmp/pip-build-3dy3x23o/pybufr-ecmwf/build_interface.py", line 878, in build self.generate_python_wrapper(source_dir) File "/tmp/pip-build-3dy3x23o/pybufr-ecmwf/build_interface.py", line 1856, in generate_python_wrapper raise InterfaceBuildError pybufr_ecmwf.custom_exceptions.InterfaceBuildError

----------------------------------------

Command "/home/zhangqw/anaconda3/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-3dy3x23o/pybufr-ecmwf/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-ucj91k6_-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-3dy3x23o/pybufr-ecmwf/

[jdkloe]

Thanks for your report. Just a few questions to make the problem more clear. It seems you are using python3 from an anaconda install. -what numpy version does this contain? -does this numpy include the f2py commandline tool? -what platform are you on (linux, mac)? -do you have a fortran compiler installed? Which one and which version? -how did you download the pybufr-ecmwf software, and which version? -what command did you use to start the installation?

Best regards, Jos

[ZhangAllen98]

Thanks. Finally, I installed it from conda-forge directly and it works.

[ZhangAllen98]

@jdkloe however, it seems that NCEP bufr file can not be opened since it is designed for ecmwf?

[jdkloe]

Thanks for reporting back. I still would be interested to now what the conditions where in which the install failed. Maybe I can improve something.

As for the NCEP BUFR file, I will have to investigate what is happening here. Could you provide me with a sample file or provide the url where it can be found?

[ZhangAllen98]

@jdkloe the NCEP BUFR file can be downloaded from the following url, https://rda.ucar.edu/datasets/ds351.0/index.html#sfol-wl-/data/ds351.0?g=112 https://rda.ucar.edu/datasets/ds461.0/index.html#sfol-wl-/data/ds461.0?g=112 something like gdassfcobs.20120101.tar.gz. Thanks for your help.

[jdkloe]

After some testing I can report that the bufr files from ucar are currently not compatible with the ECMWF bufr reading software. I tried both the old bufrdc and the new eccodes library, and both fail. This is caused by the fact that ucar has chosen to insert the bufr table definitions into the bufr files them selves. The ECMWF software provides all kinds of BUFR tables in their software installation, but not the ones needed to decode these ucar bufr files. BUFR files can only be decoded if the BUFR table files are available, so they should first be extracted from the first few messages in the ucar bufr files, before the remainder of the file can be decoded. I can add this functionality to the pybufr-ecmwf module, but this will take some time, so I cannot promise you when that will be available. If you need to decode/use these files urgently, the best thing you can do is download the ncar/ucar bufr software from https://rda.ucar.edu/#!BUFR.

[ZhangAllen98]

@jdkloe It is so nice of you to offer me some help. Fortunately, I found another software from the website: https://github.com/jswhit/py-ncepbufr. But , it seems that I can not install it well. Would you mind giving me some advice? my platform is Ubuntu 14.04, (gcc & gfortran 4.8.4) numpy1.13.1, Python 3.5.4 |Anaconda custom (64-bit)| (default, Nov 3 2017, 20:01:27) [GCC 7.2.0] on linux here is the log:

`compile options: '-Ibuild/src.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src -I/home/zhangqw/anaconda3/lib/python3.5/site-packages/numpy/core/include -I/home/zhangqw/anaconda3/include/python3.5m -c'
gcc: build/src.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/fortranobject.c
In file included from /home/zhangqw/anaconda3/lib/python3.5/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0,
                 from /home/zhangqw/anaconda3/lib/python3.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /home/zhangqw/anaconda3/lib/python3.5/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from build/src.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/fortranobject.h:13,
                 from build/src.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/fortranobject.c:2:
/home/zhangqw/anaconda3/lib/python3.5/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^
build/src.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/fortranobject.c: In function ‘format_def’:
build/src.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/fortranobject.c:138:18: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
         if (size < sizeof(notalloc)) {
                  ^
gcc: build/src.linux-x86_64-3.5/src/_bufrlibmodule.c
In file included from /home/zhangqw/anaconda3/lib/python3.5/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0,
                 from /home/zhangqw/anaconda3/lib/python3.5/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /home/zhangqw/anaconda3/lib/python3.5/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from build/src.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/fortranobject.h:13,
                 from build/src.linux-x86_64-3.5/src/_bufrlibmodule.c:19:
/home/zhangqw/anaconda3/lib/python3.5/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^
build/src.linux-x86_64-3.5/src/_bufrlibmodule.c:180:12: warning: ‘f2py_size’ defined but not used [-Wunused-function]
 static int f2py_size(PyArrayObject* var, ...)
            ^
compiling Fortran sources
Fortran f77 compiler: gfortran -Wall -g -ffixed-form -fno-second-underscore -m64 -O3 -funroll-loops
Fortran f90 compiler: /usr/bin/gfortran -Wall -g -fno-second-underscore -m64 -O3 -funroll-loops
Fortran fix compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -m64 -O3 -funroll-loops
compile options: '-Ibuild/src.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src -I/home/zhangqw/anaconda3/lib/python3.5/site-packages/numpy/core/include -I/home/zhangqw/anaconda3/include/python3.5m -c'
gfortran:f77: build/src.linux-x86_64-3.5/src/_bufrlib-f2pywrappers.f
/usr/bin/gfortran -Wall -g -L/home/zhangqw/mywrf/libraries/zlib/lib -L/home/zhangqw/mywrf/libraries/netcdf/lib -L/home/zhangqw/mywrf/libraries/HDF5/lib build/temp.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/_bufrlibmodule.o build/temp.linux-x86_64-3.5/build/src.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/fortranobject.o build/temp.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/_bufrlib-f2pywrappers.o -Lsrc -L/home/zhangqw/anaconda3/lib -lbufr -lpython3.5m -lgfortran -o build/lib.linux-x86_64-3.5/_bufrlib.cpython-35m-x86_64-linux-gnu.so
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
src/libbufr.a(cktaba.o): In function `cktaba_':
cktaba.f:(.text+0xf5): relocation truncated to fit: R_X86_64_PC32 against symbol `sc3bfr_' defined in COMMON section in src/libbufr.a(reads3.o)
cktaba.f:(.text+0x908): relocation truncated to fit: R_X86_64_PC32 against symbol `sc3bfr_' defined in COMMON section in src/libbufr.a(reads3.o)
src/libbufr.a(openbf.o): In function `openbf_':
openbf.f:(.text+0x3d7): relocation truncated to fit: R_X86_64_PC32 against symbol `sc3bfr_' defined in COMMON section in src/libbufr.a(reads3.o)
openbf.f:(.text+0x970): relocation truncated to fit: R_X86_64_PC32 against symbol `sc3bfr_' defined in COMMON section in src/libbufr.a(reads3.o)
src/libbufr.a(rdcmps.o): In function `rdcmps_':
rdcmps.f:(.text+0x41): relocation truncated to fit: R_X86_64_PC32 against symbol `rlccmn_' defined in COMMON section in src/libbufr.a(rdcmps.o)
src/libbufr.a(readerme.o): In function `readerme_':
readerme.f:(.text+0x253): relocation truncated to fit: R_X86_64_PC32 against symbol `sc3bfr_' defined in COMMON section in src/libbufr.a(reads3.o)
src/libbufr.a(readlc.o): In function `readlc_':
readlc.f:(.text+0x55c): relocation truncated to fit: R_X86_64_PC32 against symbol `rlccmn_' defined in COMMON section in src/libbufr.a(rdcmps.o)
readlc.f:(.text+0x569): relocation truncated to fit: R_X86_64_PC32 against symbol `rlccmn_' defined in COMMON section in src/libbufr.a(rdcmps.o)
readlc.f:(.text+0x8ac): relocation truncated to fit: R_X86_64_PC32 against symbol `rlccmn_' defined in COMMON section in src/libbufr.a(rdcmps.o)
readlc.f:(.text+0x94f): relocation truncated to fit: R_X86_64_PC32 against symbol `rlccmn_' defined in COMMON section in src/libbufr.a(rdcmps.o)
src/libbufr.a(readmg.o): In function `readmg_':
readmg.f:(.text+0x141): additional relocation overflows omitted from the output
collect2: error: ld returned 1 exit status
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
src/libbufr.a(cktaba.o): In function `cktaba_':
cktaba.f:(.text+0xf5): relocation truncated to fit: R_X86_64_PC32 against symbol `sc3bfr_' defined in COMMON section in src/libbufr.a(reads3.o)
cktaba.f:(.text+0x908): relocation truncated to fit: R_X86_64_PC32 against symbol `sc3bfr_' defined in COMMON section in src/libbufr.a(reads3.o)
src/libbufr.a(openbf.o): In function `openbf_':
openbf.f:(.text+0x3d7): relocation truncated to fit: R_X86_64_PC32 against symbol `sc3bfr_' defined in COMMON section in src/libbufr.a(reads3.o)
openbf.f:(.text+0x970): relocation truncated to fit: R_X86_64_PC32 against symbol `sc3bfr_' defined in COMMON section in src/libbufr.a(reads3.o)
src/libbufr.a(rdcmps.o): In function `rdcmps_':
rdcmps.f:(.text+0x41): relocation truncated to fit: R_X86_64_PC32 against symbol `rlccmn_' defined in COMMON section in src/libbufr.a(rdcmps.o)
src/libbufr.a(readerme.o): In function `readerme_':
readerme.f:(.text+0x253): relocation truncated to fit: R_X86_64_PC32 against symbol `sc3bfr_' defined in COMMON section in src/libbufr.a(reads3.o)
src/libbufr.a(readlc.o): In function `readlc_':
readlc.f:(.text+0x55c): relocation truncated to fit: R_X86_64_PC32 against symbol `rlccmn_' defined in COMMON section in src/libbufr.a(rdcmps.o)
readlc.f:(.text+0x569): relocation truncated to fit: R_X86_64_PC32 against symbol `rlccmn_' defined in COMMON section in src/libbufr.a(rdcmps.o)
readlc.f:(.text+0x8ac): relocation truncated to fit: R_X86_64_PC32 against symbol `rlccmn_' defined in COMMON section in src/libbufr.a(rdcmps.o)
readlc.f:(.text+0x94f): relocation truncated to fit: R_X86_64_PC32 against symbol `rlccmn_' defined in COMMON section in src/libbufr.a(rdcmps.o)
src/libbufr.a(readmg.o): In function `readmg_':
readmg.f:(.text+0x141): additional relocation overflows omitted from the output
collect2: error: ld returned 1 exit status
analyzevars: charselector={'len': '8'} unhandled.analyzevars: charselector={'len': '8'} unhandled.error: Command "/usr/bin/gfortran -Wall -g -L/home/zhangqw/mywrf/libraries/zlib/lib -L/home/zhangqw/mywrf/libraries/netcdf/lib -L/home/zhangqw/mywrf/libraries/HDF5/lib build/temp.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/_bufrlibmodule.o build/temp.linux-x86_64-3.5/build/src.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/fortranobject.o build/temp.linux-x86_64-3.5/build/src.linux-x86_64-3.5/src/_bufrlib-f2pywrappers.o -Lsrc -L/home/zhangqw/anaconda3/lib -lbufr -lpython3.5m -lgfortran -o build/lib.linux-x86_64-3.5/_bufrlib.cpython-35m-x86_64-linux-gnu.so" failed with exit status 1

`

[jdkloe]

mmm, this is not very clear. My first advice would be to report this to the author of this code, but I see you already did. Then, when I run on my own machine it compiles and installs just fine (from the latest github version downloaded as zip file). One thing I notice in the outputs is that in your output I don't see the -fPIC option in the gfortran list of options. This one is essential, you cannot build a python module (which is a shared object library) without this option. You could try to set the environment setting FFLAGS='-fPIC' and see if this helps?

[ZhangAllen98]

@jdkloe Thanks a lot for your kindness. When I set FFLAGS ="-fPIC" ,CFLAGS="-fPIC",it still didn't work. But , when I casually try to "conda install -c anaconda numpy",which just upgrade the numpy from 1.13.1 to 1.13.3, It unexpectedly works. lol Thanks again for your help. Best regards.

[jdkloe]

Thanks again for sharing your solution. I'll relabel remaining request to support loading ncep bufr files in pybufr-ecmwf as a feature-request (enhamcement).

[acrosby]

I have noticed the same InterfaceBuildError on Ubuntu 16.04. I have tried intel and gnu compilers as well as specifying -fPIC. F2py is installed. The following is how I have attempted to install from pypi via pip.

pip install pybufr-ecmwf
# and
pip install git+https://github.com/jdkloe/pybufr-ecmwf.git

These are the version(s) of numpy I tried: numpy==1.11.0 numpy==1.13.3

[jdkloe]

thanks for your report. Unfortunately, I cannot reproduce your problem so it seems some information is missing. I have no Ubuntu installation at hand, so I tried to rerun the installation you describe in a Docker container, and this seems to run just fine. This is the Dockerfile that I used to create the container:

# build on top of Ubuntu xenial base
FROM docker.io/danieldent/ubuntu-xenial-base

RUN apt-get update & apt-get -y upgrade
RUN apt-get install -y gfortran
RUN apt-get install -y python-pip
RUN apt-get install -y python-numpy

RUN LC_ALL=C pip install pybufr-ecmwf

Very important is that python, numpy and pybufr-ecmwf are all compiled using the same version of the gcc/gfortran compiler, otherwise it will fail. If you download code from different locations or repositories this can easily become a problem. From what location did you download gfortran and numpy?

[acrosby]

Very important is that python, numpy and pybufr-ecmwf are all compiled using the same version of the gcc/gfortran compiler, otherwise it will fail. If you download code from different locations or repositories this can easily become a problem. From what location did you download gfortran and numpy?

This is probably what is going on. I am pip installing numpy rather than using the distribution's numpy, but I am using the distribution's gfortran (or our licensed version of Intel fortran) and the distribution's python/python-dev. I am also using the full ubuntu:16.04 Docker image although I'm not sure I mentioned that before, and I'm not sure how that might be different from danieldent/ubuntu-xenial-base that you are using.

[zengxiaoqing]

I have the same error.
centsos7 64bit python3.6.4
APScheduler==3.5.1 basemap==1.0.7 cycler==0.10.0 Cython==0.27.3 f90nml==1.0.2 h5py==2.7.1 matplotlib==2.1.1 netCDF4==1.3.1 numpy==1.14.0 pandas==0.22.0 patsy==0.5.0 psutil==5.4.3 pygrib==2.0.2 pyparsing==2.2.0 pyproj==1.9.5.1 python-dateutil==2.6.1 pytz==2017.3 scikit-learn==0.19.1 scipy==0.19.1 six==1.11.0 statsmodels==0.8.0 tzlocal==1.5.1 wrapt==1.10.11 wrf-python==1.1.3 xarray==0.10.2

Using built-in specs. COLLECT_GCC=gfortran COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc/x86_64-pc-linux-gnu/7.3.0/lto-wrapper Target: x86_64-pc-linux-gnu Configured with: ./configure -enable-checking=release -enable-languages=c,c++,fortran -disable-multilib Thread model: posix gcc version 7.3.0 (GCC)

pybufr_ecmwf.custom_exceptions.InterfaceBuildError    
----------------------------------------

Command "/opt/python3.6/bin/python3.6 -u -c "import setuptools, tokenize;file='/tmp/pip-req-build-l3ef8mvm/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-fupw7xhy/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-req-build-l3ef8mvm/

[jdkloe]

@acrosby thanks for reporting back. I'll have to see if I can reproduce the problem when installing numpy from pip in stead of from the distribution repository. Could you install pybufr-ecmwf using the method I suggested, or is that not an option for you because you need this specific pip version of numpy?

[jdkloe]

@zengxiaoqing thanks for your report. Apart from reporting the software versions you used, could you also tell me from what source you downloaded them? Especially what locations did you use for python, gfortran and numpy?

[jdkloe]

@zengxiaoqing I tried to replay your software configuration in a centos docker containter, but at the first attempt I already noted that there is no python3 available in the standard centos repository. So you must have downloaded it from another source, which one was it?

[zengxiaoqing]

@jdkloe

python3 It's my own installation，
Python 3.6.4 (default, Jan 17 2018, 15:47:36) [GCC 6.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.

The system comes with python2.7.5 Python 2.7.5 (default, Feb 11 2014, 07:46:25) [GCC 4.8.2 20140120 (Red Hat 4.8.2-13)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

[zengxiaoqing]

I use the following statement to install，

rm -rf $SOURCE_PATH/pybufr-ecmwf-master unzip $SOURCE_PATH/pybufr-ecmwf-master.zip > /dev/null 2>&1 cd $SOURCE_PATH/pybufr-ecmwf-master python3 setup.py build \ --c-compiler /usr/local/bin \ --fortran-compiler /usr/local/bin

In the end I found the cause of the error：

File "setup.py", line 322, in build_extension ibi.build() File "/vdisk2/zxq/Program_packages/python_whl/36/pybufr-ecmwf-master/build_interface.py", line 883, in build self.generate_python_wrapper(source_dir) File "/vdisk2/zxq/Program_packages/python_whl/36/pybufr-ecmwf-master/build_interface.py", line 1883, in generate_python_wrapper self.integer_sizes, self.set_jelem) File "/vdisk2/zxq/Program_packages/python_whl/36/pybufr-ecmwf-master/build_interface.py", line 642, in adapt_f2py_signature_file insert_pb_interface_definition(sfd, integer_sizes) File "/vdisk2/zxq/Program_packages/python_whl/36/pybufr-ecmwf-master/build_interface.py", line 439, in insert_pb_interface_definition bytesizelong = integer_sizes[1] TypeError: 'NoneType' object is not subscriptable

How to solve this error

[jdkloe]

@zengxiaoqing today I could spend some more time on testing the installation on CentOS7. I used docker to test it, and managed to get it installed for both python2 and python3 and run all unittests on it without errors.

Again I must emphasize that the key is to use the same compiler (gcc and gfortran) that was used to compile the python interpreter and numpy. This means you can best use the system gcc version. For python2 no special tricks are needed. For python3 in docker this is easiest when you use anaconda to install python3 and numpy. The conda/miniconda3-centos7 image that I took as starting point already did the difficult work for you, i.e. to install gcc 4.4 and recompile a bunch of tools using that gcc version. Look at the Docker recipes on https://github.com/conda/conda-docker to see what is needed (this is not a simple thing to do). Installing your own downloaded copies of gcc/gfortran and python3 will never work. In general I would advise all users that wish to build pybufr-ecmwf with python3 to go for a linux distribution that comes with python3 as a standard package.

If you have no choice and need to use CentOS (or RedHat) try this:

You best start with the most recent version of pybufr-ecmwf, downloaded as zip file from github. The following docker recipes work fine for me.

This creates and tests the python2 version installed as root:

FROM docker.io/centos
RUN yum update -y
RUN yum install unzip gcc gcc-gfortran make which numpy numpy-f2py -y
# downloaded pybufr-ecmwf-master.zip from https://pypi.org/project/pybufr-ecmwf/
COPY pybufr-ecmwf-master.zip /tmp
RUN cd /tmp; unzip pybufr-ecmwf-master.zip
RUN cd /tmp/pybufr-ecmwf-master; python2 setup.py build
RUN cd /tmp/pybufr-ecmwf-master; python2 setup.py install
RUN cd /tmp/pybufr-ecmwf-master; USER=root python2 unittests.py

This creates and tests the python2 version installed as a normal user:

FROM docker.io/centos
RUN yum update -y
RUN yum install unzip gcc gcc-gfortran make which numpy numpy-f2py -y

# create a local user inside the container
RUN useradd -c 'user' -m -d /home/user -s /bin/bash user

# downloaded pybufr-ecmwf-master.zip from https://pypi.org/project/pybufr-ecmwf/
COPY pybufr-ecmwf-master.zip /home/user/
RUN chown -R user:user /home/user/pybufr-ecmwf-master.zip

# switch to the local user
USER user
ENV HOME /home/user

RUN cd $HOME; unzip pybufr-ecmwf-master.zip
RUN cd $HOME/pybufr-ecmwf-master; python2 setup.py build
RUN cd $HOME/pybufr-ecmwf-master; python2 setup.py install --user
RUN cd $HOME/pybufr-ecmwf-master; python2 unittests.py

This creates and tests the python3 version installed as root based on miniconda:

FROM conda/miniconda3-centos7
RUN yum update -y
RUN yum install unzip gcc gcc-gfortran which make -y

# install python3 numpy using conda
RUN conda install numpy numpy -y

# downloaded pybufr-ecmwf-master.zip
# from https://pypi.org/project/pybufr-ecmwf/
COPY pybufr-ecmwf-master.zip /tmp

RUN cd /tmp; unzip pybufr-ecmwf-master.zip
RUN cd /tmp/pybufr-ecmwf-master; python3 setup.py build
RUN cd /tmp/pybufr-ecmwf-master; python3 setup.py install
RUN cd /tmp/pybufr-ecmwf-master; USER=root python3 unittests.py

This creates and tests the python3 version installed as a normal user based on minconda:

FROM conda/miniconda3-centos7
RUN yum update -y
RUN yum install unzip gcc gcc-gfortran which make -y

# install python3 numpy using conda
RUN conda install numpy numpy -y

# create a local user inside the container
RUN useradd -c 'user' -m -d /home/user -s /bin/bash user

# downloaded pybufr-ecmwf-master.zip
# from https://pypi.org/project/pybufr-ecmwf/
COPY pybufr-ecmwf-master.zip /home/user/
RUN chown -R user:user /home/user/pybufr-ecmwf-master.zip

# switch to the local user
USER user
ENV HOME /home/user

RUN cd $HOME; unzip pybufr-ecmwf-master.zip
RUN cd $HOME/pybufr-ecmwf-master; python3 setup.py build
RUN cd $HOME/pybufr-ecmwf-master; python3 setup.py install --user
RUN cd $HOME/pybufr-ecmwf-master; python3 unittests.py

I hope this will be helpful for you.

[DivinytyToffee]

@jdkloe Hi. When I tried install your library onto docker, how your write right here, I get a error. I use: alembic, python 3.7.2, gcc and gfortran have same version(8.2.0-r2), I installed numpy(numpy.f2py is caused), but with pip, or manual installed I got an error: Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-install-tnrn4s32/pybufr-ecmwf/setup.py", line 410, in <module> provides=["pybufr_ecmwf"]) File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup dist.run_commands() File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/usr/local/lib/python3.7/site-packages/setuptools/command/install.py", line 61, in run return orig.install.run(self) File "/usr/local/lib/python3.7/distutils/command/install.py", line 545, in run self.run_command('build') File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/tmp/pip-install-tnrn4s32/pybufr-ecmwf/setup.py", line 119, in run _build.run(self) File "/usr/local/lib/python3.7/distutils/command/build.py", line 135, in run self.run_command(cmd_name) File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/usr/local/lib/python3.7/distutils/command/build_ext.py", line 339, in run self.build_extensions() File "/usr/local/lib/python3.7/distutils/command/build_ext.py", line 448, in build_extensions self._build_extensions_serial() File "/usr/local/lib/python3.7/distutils/command/build_ext.py", line 473, in _build_extensions_serial self.build_extension(ext) File "/tmp/pip-install-tnrn4s32/pybufr-ecmwf/setup.py", line 318, in build_extension ibi.build() File "/tmp/pip-install-tnrn4s32/pybufr-ecmwf/build_interface.py", line 888, in build self.extract_constants() File "/tmp/pip-install-tnrn4s32/pybufr-ecmwf/build_interface.py", line 1874, in extract_constants import ecmwfbufr ImportError: Error relocating /tmp/pip-install-tnrn4s32/pybufr-ecmwf/build/lib.linux-x86_64-3.7/pybufr_ecmwf/ecmwfbufr.cpython-37m-x86_64-linux-gnu.so: gbytes_: symbol not found Did you know, how it can be fixed?

[jdkloe]

Hi @DivinytyToffee, I am missing some key information here so the situation is not completely clear to me. The most important thing I need to know is how you started this build. So if you took one of the Docker files I provided and modified it, then please share the modified Docker file so I can see what you modified it. If you took the instructions from one of these Dockerfiles and tried to run the commands manually, then what was the exact order of commands that you issued? I need to be able to reproduce the error condition on my side before I can fix it.

For your information, I just reran the build for the last Docker recipe that I provided above, using centos-7 and conda, doing the install as unprivileged user, and it still runs just fine on my side.

[DivinytyToffee]

@jdkloe. This is code for run docker. gcc and gfortran commented, as and pipenv install, because I tried run docker with automatically install pybufr, but the install crashed on this step, and I tried make manually installation, but I get same problem.

# Base image
FROM python:3.7.2-alpine3.9

# Forwarding DJANGO_ENV
ARG DJANGO_ENV

# System envoriments
ENV DJANGO_ENV=${DJANGO_ENV} \
  LANG=C.UTF-8 \
  PYTHONFAULTHANDLER=1 \
  PYTHONUNBUFFERED=1 \
  PYTHONHASHSEED=random \
  PIP_NO_CACHE_DIR=off \
  PIP_DISABLE_PIP_VERSION_CHECK=on \
  PIP_DEFAULT_TIMEOUT=100 \
  PIPENV_HIDE_EMOJIS=true \
  PIPENV_COLORBLIND=true \
  PIPENV_NOSPIN=true \
  PIPENV_DOTENV_LOCATION=config/.env

# System deps
RUN apk --no-cache add \
     bash \
     build-base \
#     "gcc>7.3.0" \
     gettext \
#     "gfortran>7.3.0" \
     libffi-dev \
     openblas-dev \
     postgresql-dev \
     tini

# Set up workdir
WORKDIR /opt/app

# Add main executable script and run it
#ENTRYPOINT ["/sbin/tini", "--", "/entrypoint.sh"]
CMD ["sh"]

[jdkloe]

Hi @DivinytyToffee,

for the docker base image that you tried to use this cannot work. You used python:3.7.2-alpine3.9 and this takes alpine, which provides python3.6, and installs a new python3.7 on top of it.

However, pybufr-ecmwf needs numpy and numpy-f2py to be installed. Alpine provides only the 3.6 packages for these dependencies, so that is not enough. If you really need to use python3.7, then you need to provide numpy and numpy.f2py modules for that python version as well (or find a Docker image that provides them).

As alternative I looked at alpine:3.8 and tried to create a Docker file that installs pybufr-ecmwf in it using pip3. For this image python3.6 and numpy and numpy.f2py are available, so that part is easy.

Unfortunately there are some other issues to be solved. The alpine packages py3-numpy and py3-numpy-f2py seem not complete (that is a packaging bug in my opinion). They do not contain header and c files needed for f2py to correctly build this fortran module. For your convenience I provided them in 2 tar files. You can also download them from: https://github.com/numpy/numpy/tree/master/numpy/f2py/src and https://github.com/numpy/numpy/tree/master/numpy/core/include/numpy

f2py_src.tar.gz numpy_include.tar.gz

A next problem seems that gcc does not like some of the c code contained in pybufr-ecmwf. If I give the setup.py script the option --c-flags "--std=c99" it works, but I cannot get pip to pass the option to setup.py, so I split the install in 3 steps. First I use pip3 to download the module, and then I use 2 calls to setup.py to do the build and install. This seems to work well now.

This is the Docker file I used for testing:

FROM alpine:3.8
RUN apk --no-cache add \
     bash build-base gcc gfortran \
     py3-numpy py3-numpy-f2py
WORKDIR /opt/app

COPY f2py_src.tar.gz numpy_include.tar.gz /opt/app/
RUN mkdir -p /usr/lib/python3.6/site-packages/numpy/f2py/src/
RUN mkdir -p /usr/lib/python3.6/site-packages/numpy/core/include/numpy/
RUN cd /opt/app/; tar xfz f2py_src.tar.gz; \
    cp f2py_src/* /usr/lib/python3.6/site-packages/numpy/f2py/src/
RUN cd /opt/app/; tar xfz numpy_include.tar.gz; \
    cp numpy_include/* /usr/lib/python3.6/site-packages/numpy/core/include/numpy/

RUN pip3 download pybufr-ecmwf
RUN tar zxvf pybufr-ecmwf-0.82.tar.gz
RUN cd pybufr-ecmwf-0.82; \
    python3 setup.py build --c-flags "--std=c99"; \
    python3 setup.py install

# dmonstrate that the module works for decoding a BUFR file
RUN python3 pybufr-ecmwf-0.82/example_programs/bufr_to_ascii.py \
    -i pybufr-ecmwf-0.82/test/testdata/Testfile.BUFR

CMD ["sh"]

[DivinytyToffee]

@jdkloe, I'm very grateful for your help! You saved my ass from writing a BUFR decoder. For ran in python3.7 I done that

FROM python:3.7.2-alpine3.8
RUN apk --no-cache add \
     bash build-base gcc gfortran \
     py3-numpy py3-numpy-f2py
WORKDIR /opt/app
COPY f2py_src.tar.gz numpy_include.tar.gz /opt/app/
RUN mkdir -p /usr/lib/python3.7/site-packages/numpy/f2py/src/
RUN mkdir -p /usr/lib/python3.7/site-packages/numpy/core/include/numpy/
RUN cd /opt/app/; tar xfz f2py_src.tar.gz; \
    cp f2py_src/* /usr/lib/python3.7/site-packages/numpy/f2py/src/
RUN cd /opt/app/; tar xfz numpy_include.tar.gz; \
    cp numpy_include/* /usr/lib/python3.7/site-packages/numpy/core/include/numpy/

RUN pip install numpy
# Install packages
RUN pip download pybufr-ecmwf
RUN tar zxvf pybufr-ecmwf-0.82.tar.gz
RUN cd pybufr-ecmwf-0.82; \
    python setup.py build --c-flags "--std=c99"; \
    python setup.py install