Installation from source not working

[woutdenolf]

ubuntu 20.04 python 3.8.8 pip 21.1.1 setuptools 56.1.0

python setup.py build

Traceback (most recent call last):
  File "setup.py", line 103, in <module>
    setup(
  File "...lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "...lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "...lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "...lib/python3.8/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "...lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "...lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "...lib/python3.8/site-packages/numpy/distutils/command/build_clib.py", line 122, in run
    dispatch_hpath = os.path.join(self.get_finalized_command("build_src").build_src, dispatch_hpath)
  File "...lib/python3.8/distutils/cmd.py", line 103, in __getattr__
    raise AttributeError(attr)
AttributeError: build_src

self.get_finalized_command("build_src") returns a NullCommand instance due to

    cmdclass={
        'build_clib': BuildClib,
        'build_src': NullCommand,
    },

[woutdenolf]

When removing the NullCommand

cmdclass={
        'build_clib': BuildClib,
        #'build_src': NullCommand,
    },

the following exception occurs

Traceback (most recent call last):
  File "setup.py", line 106, in <module>
    setup(
  File "/users/denolf/virtualenvs/oasys1/ubuntu_20_04/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/users/denolf/virtualenvs/oasys1/ubuntu_20_04/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/users/denolf/virtualenvs/oasys1/ubuntu_20_04/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/users/denolf/virtualenvs/oasys1/ubuntu_20_04/lib/python3.8/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/users/denolf/virtualenvs/oasys1/ubuntu_20_04/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/users/denolf/virtualenvs/oasys1/ubuntu_20_04/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/users/denolf/virtualenvs/oasys1/ubuntu_20_04/lib/python3.8/site-packages/numpy/distutils/command/build_clib.py", line 88, in run
    if not self.libraries:
  File "/users/denolf/virtualenvs/oasys1/ubuntu_20_04/lib/python3.8/distutils/cmd.py", line 103, in __getattr__
    raise AttributeError(attr)
AttributeError: libraries

[woutdenolf]

@srio @lucarebuffi Any suggestions? It's for testing this issue: https://github.com/silx-kit/silx/issues/3454

[srio]

Hello, dont' try to recompile everything... Oasys is installed in NICE, just enter "oasys" If you want to install in another machine, follow https://www.aps.anl.gov/Science/Scientific-Software/OASYS

The original question on silx is not specfic to oasys/shadowOui but all apps using ImageView, as the behavoiur of the line profile changed a lot.

Many thanks, Manuel

[t20100]

@woutdenolf silx-kit/silx#3454 can be reproduced without OASYS, see on the issue.

[woutdenolf]

I can build everything under python 3.7.8.

So not urgent but you'll have to fix this at some point I suppose.

[AbbyGi]

I'm trying to install this on my Mac and I'm encountering a similar issue.

I brew installed gfortran and I'm trying to get version 18.5.30 from PyPi.

Mac version 11.0.1 numpy version 1.20.3 python version 3.7.10

[mrakitin]

Hi @woutdenolf, @srio, we were working with @AbbyGi on the compilation of the package. The problem is actually with the numpy.distutils API, changed in the latest versions, so that the missing attribute build_src causes the installation issue. A workaround I've found was to use numpy 1.16, with that version the installation went well. I think the Shadow3 installer should tae into account the changed API in the latest numpy versions.

[woutdenolf]

Instead of doing custom things, maybe we can use a more standard way to do fortran bindings like this (see docs)

[srio]

Thanks to all for your useful remarks. Yes, I agree that shadow3 needs an update. The pip installable version is too old. Right now I compile the devel-gfortran branch that contains changes suggested by @tshchoonj to work with new versions of gfortran. I compiled it using conda and python3.8 but I do not remember the numpy version I used. I will look into your comments and try to fix these issues soon. If you get something working or suggest some modifications in the setup, I would appreciate. Thanks a lot, M

[mrakitin]

Hello @srio, thanks for the quick response! I created a conda feedstock to build the conda package for Shadow3. Here is the feedstock: https://github.com/nsls-ii-forge/shadow3-feedstock (check the meta.yaml for the details of the build and run dependencies). The resulting package is available from https://anaconda.org/nsls2forge/shadow3 (installable with conda install -c nsls2forge shadow3). I used the latest released version at PyPI as a source. Probably worth releasing an up-to-date version, so that can be built with conda. For the NumPy version, I used v1.16 for the packaging step, but the run dependency can be newer.

The longer-term plan is to port that feedstock to conda-forge, so it can be more widely used. I invite you to co-maintain that future feedstock after our team handles the initial feedstock creation.

A side note: the advantage of using conda is that the "heavy" dependencies like xraylib can be installed with no pain of compilation.

[srio]

Thanks a lot Maksim for your help. It looks brilliant. I will look into that as soon as possible. Indeed, it is very good to have it in conda, and your help is very appreciated as I am completely ignorant in conda packaging. I fully agree to have a shared future feedstock with full functionality. Did you use the devel-gfortran branch ?

[mrakitin]

Thanks, Manuel! I haven't tried packaging from that branch. The feedstocks are generally using the tagged versions from GitHub, or releases from PyPI (source distributions, .tar.gz files, not wheels). So the conda version is based on https://files.pythonhosted.org/packages/ee/4d/b506675348eac8b4a7f48dec9e2a9881cded37856a9b4bee618ca82051ca/shadow3-18.5.30.tar.gz from https://pypi.org/project/shadow3/18.5.30/#files.

I can try to package the current version from devel-gfortran in a separate branch to see how it goes.

[mrakitin]

@srio, I can confirm that the version from the devel-gfortran branch builds successfully. See https://github.com/nsls-ii-forge/shadow3-feedstock/pull/3 for the updates to the feedstock, and the corresponding build logs. I guess the next step would be to upload a new version to PyPI based on that branch, and we can build the corresponding conda package.

[lucarebuffi]

As a note, I successfully installed OASYS + Shadow + SRW in python3.8 on a Mac by using numpy 1.18 and fabio 0.11

[CFGrote]

I built shadow3 in devel-gfortran following the installation notes in README.md. numpy is v. 1.19.1.However, importing in python (3.9) failed:

>>> from Shadow import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "shadow3/Shadow/__init__.py", line 8, in <module>
    from Shadow.ShadowLibExtensions import OE, Source, Beam, CompoundOE, IdealLensOE
  File "shadow3/Shadow/ShadowLibExtensions.py", line 11, in <module>
    import Shadow.ShadowLib as ShadowLib
ImportError: libgfortran.so.4: cannot open shared object file: No such file or directory

Trying to install libgfortran4 from conda leads to multiple inconsistencies.

Any suggestion?

[srio]

My suggestion is to use python 3.7, the version used in Oasys. We are working in supporting python ? 3.7 but still there are problems, with shadow and other Oasys dependencies. Sorry!

[robnagler]

@CFGrote the particular error is due to the version of Fortran, not Python. We haven't been able to build Shadow with gfortran8+, but you can download the binary (pip install) and build libgfortran.so.4 by building gfortran 7.5.

We tried the gfortran8-fixes PR, but it still was getting errors so we rolled back to the above solution.

[CFGrote]

Thanks, I managed to compile with gfortran-7. I then ran some tests in tests/. They seem to go through but I get a

Fortran runtime error: Array rank of PUT is not 1.

at the end of each test log.

[robnagler]

@CFGrote We don't recompile shadow with gfortran7. Instead we build gfortran7, and just take the libgfortran.so.4 from the compile and install it. The compiler on the system is gfortran 10.3.1. However, we pip install shadow3 as a wheel.

[mrakitin]

Hi @srio, I would like to follow up on the release based on the devel-gfortran branch discussed above. Do you plan to make the release soon? Thanks!

[srio]

Hi @mrakitin. We have been working on that and we have a version pretty much ready. It is the devel-gfortran-yb66 branch. Please use this branch that contains the most updated version (not yet merged with master) and wheels to be used in oasys.

[mrakitin]

Hi Manuel / @srio, thanks for the update! I've tried building off that branch with the latest gfortran 10.3, and it worked OK for Linux and macOS: https://github.com/conda-forge/shadow3-feedstock/pull/6. I am currently excluding the Windows builds due to the failures in the past, but now the package may be ready for Windows. Is it? Are there special instructions on how to build Shadow3 for Windows?

The conda-forge rules encourage using the released artifacts. Do you plan to tag a new version and push it to PyPI soon? I can then update that PR, and build the released version for conda-forge. Thanks!

[srio]

Dear Maksim,

yes, I managed to build the windows version using the recipe that I summarized in the setup.py and I copy here:

# Notes creating windows wheels:
#
# cd src\windows_gfortran
# compile_fortran.bat
# cd ..\..
# python setup.py build --compiler=cygwin
# python setup.com bdist_wheel
# rename dist\...whl to .zip, open it and add the linshadow3.dll and libshadow3c.dll in the Shadow directory. Rename back to .whl

Note that I used the gcc and gfortran compilers! It is possible to compile it with visual studio including the intel-fortran compiler. However, I noticed some crashes and I want to keep the distributed version created with gnu compilers.

Can I test your created conda packages? When can I find them?

We found a problem in the linux version (not in windows and mac), where the option to use a surface mesh does not work. I am looking at that.

regards, M

[mrakitin]

Thank you for the updates, @srio! I created a dev branch in the conda-forge feedstock, that can be used to build development versions for testing: https://github.com/conda-forge/shadow3-feedstock/tree/dev. The artifacts are uploaded to https://anaconda.org/conda-forge/shadow3.

The package can be installed as follows:

$ conda install -c conda-forge/label/shadow3_dev shadow3

(Linux/OSX builds only, no Windows yet)

[srio]

Most (if not all) of these problems should be fixed in the new release 22.8.20.
@mrakitin: would you please update your conda installers for this version? It should be stable for a while...

[mrakitin]

@srio, thanks a lot for the update! This version was built automatically yesterday via https://github.com/conda-forge/shadow3-feedstock/pull/9. It's available in the conda-forge channel now:

[mrakitin]

Two questions I have at the moment: 1) I am building it with gfortran v7. Does the build support newer versions? I can try it, but I wonder if you tested it with the newer versions. 2) What's the state of the Windows builds? I am currently skipping them in the conda feedstock.

[srio]

I am building it with gfortran v7. Does the build support newer versions? I can try it, but I wonder if you tested it with the newer versions.

No problem with newer versions.

What's the state of the Windows builds? I am currently skipping them in the conda feedstock.

I created all wheels for windows. I used mingw64 compilers (gcc and gfortran) and python from python.org for creating the wheels, but it also works with miniconda. The wheels created with python.org work well with python from miniconda. The only problem I had is when I installed cygwin or mingw in conda: I recommend not to do it, and use mingw64 installed externally. Have a look at the specific comments for windows in setup.py.

Thanks for this!

[mrakitin]

@srio, I updated the feedstock back in August, and the 2022.8.24 version was built with the latest compilers from the conda-forge toolchain (https://github.com/conda-forge/shadow3-feedstock/pull/10 and https://github.com/conda-forge/shadow3-feedstock/pull/12). With these updates, it's compatible with scipy 1.7+ (previously due to the gfortran 4 restriction newer versions of scipy were conflicting with the shadow3 package).

I tried experimenting with Windows builds but excluded them, as there were failures, and I did not have the bandwidth to fix them for that platform. If there are volunteers, they are welcome to help.

[srio]

Thanks Maksim.

In windows I built it in my PC. We are on the way of testing it in other machines to track all dependencies. On the waiting list!

Rgds, M