search

xgcm / aerobulk-python

A python wrapper for aerobulk (https://github.com/brodeau/aerobulk)
GNU General Public License v3.0
14 stars 4 forks source link

Package Publishing #7

Closed jbusecke closed 2 years ago

jbusecke commented 2 years ago

My brain is fried for today, but let me summarize what worked and what didnt in my quest to package and move this code to both pypi and conda-forge

First of all, am I correct that I should aim for the following situation?:

Assuming that this is the case I have achieved the following today. The current state of the repo does compile and work on my local machine.

I have tried to upload a tarball to test.pypi.com, which I can download on the pangeo cloud, but have not yet managed to install. I tried to mamba install gfortran and then run pip install -i https://test.pypi.org/simple/ aerobulk-python but got this bad boy of an error message:

...this thing was too long for the terminal history on pangeo cloud 😭 I/srv/conda/envs/notebook/lib/python3.9/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.9/numpy/distutils/include -I/srv/conda/envs/notebook/include/python3.9 -c' extra options: '-Jbuild/temp.linux-x86_64-3.9/aerobulk -Ibuild/temp.linux-x86_64-3.9/aerobulk' INFO: extra f90 options: '-fdefault-real-8 -ffree-line-length-200 -O3 -w' INFO: gfortran:f90: ./src/aerobulk/src/mod_const.f90 INFO: extra f90 options: '-fdefault-real-8 -ffree-line-length-200 -O3 -w' INFO: gfortran:f90: ./src/aerobulk/src/mod_phymbl.f90 INFO: extra f90 options: '-fdefault-real-8 -ffree-line-length-200 -O3 -w' INFO: gfortran:f90: ./src/aerobulk/src/mod_skin_coare.f90 INFO: extra f90 options: '-fdefault-real-8 -ffree-line-length-200 -O3 -w' INFO: gfortran:f90: ./src/aerobulk/src/mod_skin_ecmwf.f90 INFO: extra f90 options: '-fdefault-real-8 -ffree-line-length-200 -O3 -w' INFO: gfortran:f90: ./src/aerobulk/src/mod_blk_andreas.f90 INFO: extra f90 options: '-fdefault-real-8 -ffree-line-length-200 -O3 -w' INFO: gfortran:f90: ./src/aerobulk/src/mod_common_coare.f90 INFO: extra f90 options: '-fdefault-real-8 -ffree-line-length-200 -O3 -w' INFO: gfortran:f90: ./src/aerobulk/src/mod_blk_coare3p0.f90 INFO: extra f90 options: '-fdefault-real-8 -ffree-line-length-200 -O3 -w' INFO: gfortran:f90: ./src/aerobulk/src/mod_blk_coare3p6.f90 INFO: extra f90 options: '-fdefault-real-8 -ffree-line-length-200 -O3 -w' INFO: gfortran:f90: ./src/aerobulk/src/mod_blk_ecmwf.f90 INFO: extra f90 options: '-fdefault-real-8 -ffree-line-length-200 -O3 -w' INFO: gfortran:f90: ./src/aerobulk/src/mod_blk_ncar.f90 INFO: extra f90 options: '-fdefault-real-8 -ffree-line-length-200 -O3 -w' INFO: gfortran:f90: ./src/aerobulk/src/mod_blk_neutral_10m.f90 INFO: extra f90 options: '-fdefault-real-8 -ffree-line-length-200 -O3 -w' INFO: gfortran:f90: ./src/aerobulk/src/mod_aerobulk_compute.f90 INFO: extra f90 options: '-fdefault-real-8 -ffree-line-length-200 -O3 -w' INFO: gfortran:f90: ./src/aerobulk/src/mod_aerobulk.f90 INFO: extra f90 options: '-fdefault-real-8 -ffree-line-length-200 -O3 -w' INFO: gfortran:f90: ./src/mod_aerobulk_wrap.f90 INFO: compiling Fortran sources INFO: Fortran f77 compiler: /srv/conda/envs/notebook/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops Fortran f90 compiler: /srv/conda/envs/notebook/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops Fortran fix compiler: /srv/conda/envs/notebook/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops INFO: compile options: '-Ibuild/src.linux-x86_64-3.9/build/src.linux-x86_64-3.9/src -I/srv/conda/envs/notebook/lib/python3.9/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.9/numpy/distutils/include -I/srv/conda/envs/notebook/include/python3.9 -c' extra options: '-Jbuild/temp.linux-x86_64-3.9/aerobulk -Ibuild/temp.linux-x86_64-3.9/aerobulk' INFO: extra f90 options: '-fdefault-real-8 -ffree-line-length-200 -O3 -w' INFO: gfortran:f90: build/src.linux-x86_64-3.9/src/mod_aerobulk_wrap-f2pywrappers2.f90 creating build/lib.linux-x86_64-3.9 creating build/lib.linux-x86_64-3.9/aerobulk INFO: /srv/conda/envs/notebook/bin/gfortran -Wall -g -Wall -g -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/srv/conda/envs/notebook/lib -Wl,-rpath-link,/srv/conda/envs/notebook/lib -L/srv/conda/envs/notebook/lib build/temp.linux-x86_64-3.9/build/src.linux-x86_64-3.9/src/mod_aerobulk_wrapmodule.o build/temp.linux-x86_64-3.9/build/src.linux-x86_64-3.9/build/src.linux-x86_64-3.9/src/fortranobject.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_const.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_phymbl.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_skin_coare.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_skin_ecmwf.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_blk_andreas.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_common_coare.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_blk_coare3p0.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_blk_coare3p6.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_blk_ecmwf.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_blk_ncar.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_blk_neutral_10m.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_aerobulk_compute.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_aerobulk.o build/temp.linux-x86_64-3.9/src/mod_aerobulk_wrap.o build/temp.linux-x86_64-3.9/build/src.linux-x86_64-3.9/src/mod_aerobulk_wrap-f2pywrappers2.o -L/srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/lib/../lib -L/srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/lib/../lib -lgfortran -o build/lib.linux-x86_64-3.9/aerobulk/mod_aerobulk_wrap.cpython-39-x86_64-linux-gnu.so /srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lgcc collect2: error: ld returned 1 exit status error: Command "/srv/conda/envs/notebook/bin/gfortran -Wall -g -Wall -g -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/srv/conda/envs/notebook/lib -Wl,-rpath-link,/srv/conda/envs/notebook/lib -L/srv/conda/envs/notebook/lib build/temp.linux-x86_64-3.9/build/src.linux-x86_64-3.9/src/mod_aerobulk_wrapmodule.o build/temp.linux-x86_64-3.9/build/src.linux-x86_64-3.9/build/src.linux-x86_64-3.9/src/fortranobject.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_const.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_phymbl.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_skin_coare.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_skin_ecmwf.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_blk_andreas.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_common_coare.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_blk_coare3p0.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_blk_coare3p6.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_blk_ecmwf.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_blk_ncar.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_blk_neutral_10m.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_aerobulk_compute.o build/temp.linux-x86_64-3.9/src/aerobulk/src/mod_aerobulk.o build/temp.linux-x86_64-3.9/src/mod_aerobulk_wrap.o build/temp.linux-x86_64-3.9/build/src.linux-x86_64-3.9/src/mod_aerobulk_wrap-f2pywrappers2.o -L/srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/lib/../lib -L/srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/lib/../lib -lgfortran -o build/lib.linux-x86_64-3.9/aerobulk/mod_aerobulk_wrap.cpython-39-x86_64-linux-gnu.so" failed with exit status 1 INFO: ########### EXT COMPILER OPTIMIZATION ########### INFO: Platform : Architecture: x64 Compiler : unix-like CPU baseline : Requested : 'min' Enabled : SSE SSE2 SSE3 Flags : -msse -msse2 -msse3 Extra checks: none CPU dispatch : Requested : 'max -xop -fma4' Enabled : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL Generated : none INFO: CCompilerOpt.cache_flush[817] : write cache to path -> /tmp/pip-install-bmalvxsf/aerobulk-python_ec8fa81c00564cf48c71241905fd8ba7/build/temp.linux-x86_64-3.9/ccompiler_opt_cache_ext.py ---------------------------------------- ERROR: Command errored out with exit status 1: /srv/conda/envs/notebook/bin/python3.9 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bmalvxsf/aerobulk-python_ec8fa81c00564cf48c71241905fd8ba7/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bmalvxsf/aerobulk-python_ec8fa81c00564cf48c71241905fd8ba7/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-gwdc2ps9/install-record.txt --single-version-externally-managed --compile --install-headers /srv/conda/envs/notebook/include/python3.9/aerobulk-python Check the logs for full command output.

But on my local machine I was able to install and run this version with pip install -i https://test.pypi.org/simple/ aerobulk-python --no-binary :all:, so I assume I could make it work on the pangeo cloud too given the right tinkering.

I have in parallel tried to create a conda-forge recipe, but I did not manage to actually get the code checked out using git release archives (this suggests that I might get around these issues by simply basing the recipe on a pypi tarball).

So if my assumptions above are correct, I will distribute a tarball to pypi next (which might or might not work depending on your local setup), but might be able to use that as source for the conda-recipe.

Any thought/tips are much appreciated.

andersy005 commented 2 years ago

Distribute a tarball to pypi, which can then be used by individual user to build the package locally (requiring fortran compiler/setup on their end)

A while ago, I put together a GitHub Action workflow for building manylinux wheels for https://github.com/cspencerjones/xlayers. You may find this GitHub Action workflow useful if you ever decide to publish manylinux binary wheels in addition to the standard tarball.

https://github.com/cspencerjones/xlayers/blob/master/.github/workflows/pythonpublish.yaml

jbusecke commented 2 years ago

I think I have figured this out in the publish CI, and it works with pangeo-forge (which was the most important aspect). Closing this now.