How do I force pip packages to be installed in a specific order?

jcgoble3 commented 3 years ago

As a temporary (and potentially permanent) workaround for astropy (#349), we are attempting to use pyfits for part of our project. We also need numpy and scipy, so my build.gradle currently contains this:

        python {
            pip {
                install "numpy==1.17.4"
                install "scipy==1.4.1"
                install "pyfits==3.4"
            }
        }

The Chaquopy plugin correctly fetches the native wheels from your repo for numpy and scipy, and the sdist of pyfits from pypi.org. The next thing it tries to do is run setup.py bdist_wheel for pyfits, which fails because the setup.py contains a function numpy_extension_hook that requires numpy to already be installed, which it isn't because Chaquopy has only downloaded the wheel.

How do I force Chaquopy to completely install numpy before attempting to process pyfits? I have tried removing pyfits from build.gradle, building, and then re-adding it, but the installation is run anew each time. I have also tried removing numpy from build.gradle and letting it be installed as a dependency of pyfits, but that also fails in the same manner. I basically need a two-stage installation, where I install numpy completely in the first stage and then begin the second stage where pyfits is installed.

Any ideas?

Here is the complete output of gradlew.bat build (produced with Gradle plugin 4.0.1, but fails identically with 4.0.0):

> Configure project :app
Warning: This version of Chaquopy has not been tested with Android Gradle plugin versions beyond 4.0.0. If you experience problems, please edit the version of com.android.tools.build:gradle in your top-level build.gradle file. See https://chaquo.com/chaquopy/doc/current/versions.html.

> Task :app:generateDebugPythonRequirements
Chaquopy: Installing for arm64-v8a
Looking in indexes: https://pypi.org/simple, https://chaquo.com/pypi-7.0
Collecting numpy==1.17.4
  Using cached https://chaquo.com/pypi-7.0/numpy/numpy-1.17.4-3-cp38-cp38-android_21_arm64_v8a.whl
Collecting scipy==1.4.1
  Using cached https://chaquo.com/pypi-7.0/scipy/scipy-1.4.1-0-cp38-cp38-android_21_arm64_v8a.whl
Collecting pyfits==3.4
  Using cached https://files.pythonhosted.org/packages/45/98/d6d25932e6a82fa8456d38ab307bfb8945a1e1dd4e896730555e3b61cfc5/pyfits-3.4.tar.gz
Collecting chaquopy-openblas>=0.2.20 (from numpy==1.17.4)
  Using cached https://chaquo.com/pypi-7.0/chaquopy-openblas/chaquopy_openblas-0.2.20-5-py3-none-android_21_arm64_v8a.whl
Collecting chaquopy-libgfortran>=4.9 (from scipy==1.4.1)
  Using cached https://chaquo.com/pypi-7.0/chaquopy-libgfortran/chaquopy_libgfortran-4.9-0-py3-none-android_21_arm64_v8a.whl
Collecting chaquopy-libcxx>=7000 (from scipy==1.4.1)
  Using cached https://chaquo.com/pypi-7.0/chaquopy-libcxx/chaquopy_libcxx-7000-2-py3-none-android_21_arm64_v8a.whl
Building wheels for collected packages: pyfits
  Running setup.py bdist_wheel for pyfits: started
  Running setup.py bdist_wheel for pyfits: finished with status 'error'
  Complete output from command C:\UserPrograms\Python3.8\python.exe -u -S -c "import setuptools, tokenize;__file__='C:\\Users\\jcgob\\AppData\\Local\\Temp\\pip-install-o2m2ka8w\\pyfits\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\jcgob\AppData\Local\Temp\pip-wheel-wqdlxqps --python-tag py3:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-3.8
  creating build\lib.win-amd64-3.8\pyfits
  copying pyfits\card.py -> build\lib.win-amd64-3.8\pyfits
  copying pyfits\column.py -> build\lib.win-amd64-3.8\pyfits
  copying pyfits\convenience.py -> build\lib.win-amd64-3.8\pyfits
  copying pyfits\core.py -> build\lib.win-amd64-3.8\pyfits
  copying pyfits\diff.py -> build\lib.win-amd64-3.8\pyfits
  copying pyfits\file.py -> build\lib.win-amd64-3.8\pyfits
  copying pyfits\fitsrec.py -> build\lib.win-amd64-3.8\pyfits
  copying pyfits\header.py -> build\lib.win-amd64-3.8\pyfits
  copying pyfits\py3compat.py -> build\lib.win-amd64-3.8\pyfits
  copying pyfits\util.py -> build\lib.win-amd64-3.8\pyfits
  copying pyfits\verify.py -> build\lib.win-amd64-3.8\pyfits
  copying pyfits\version.py -> build\lib.win-amd64-3.8\pyfits
  copying pyfits\_numpy_hacks.py -> build\lib.win-amd64-3.8\pyfits
  copying pyfits\_release.py -> build\lib.win-amd64-3.8\pyfits
  copying pyfits\__init__.py -> build\lib.win-amd64-3.8\pyfits
  creating build\lib.win-amd64-3.8\pyfits\_compat
  copying pyfits\_compat\odict.py -> build\lib.win-amd64-3.8\pyfits\_compat
  copying pyfits\_compat\weakref.py -> build\lib.win-amd64-3.8\pyfits\_compat
  copying pyfits\_compat\__init__.py -> build\lib.win-amd64-3.8\pyfits\_compat
  creating build\lib.win-amd64-3.8\pyfits\extern
  Command "C:\UserPrograms\Python3.8\python.exe -u -S -c "import setuptools, tokenize;__file__='C:\\Users\\jcgob\\AppData\\Local\\Temp\\pip-install-o2m2ka8w\\pyfits\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\jcgob\AppData\Local\Temp\pip-wheel-wqdlxqps --python-tag py3" failed with error code 1 in C:\Users\jcgob\AppData\Local\Temp\pip-install-o2m2ka8w\pyfits\
  copying pyfits\extern\six.py -> build\lib.win-amd64-3.8\pyfits\extern
  copying pyfits\extern\__init__.py -> build\lib.win-amd64-3.8\pyfits\extern
  creating build\lib.win-amd64-3.8\pyfits\hdu
  copying pyfits\hdu\base.py -> build\lib.win-amd64-3.8\pyfits\hdu
  copying pyfits\hdu\compressed.py -> build\lib.win-amd64-3.8\pyfits\hdu
  copying pyfits\hdu\groups.py -> build\lib.win-amd64-3.8\pyfits\hdu
  copying pyfits\hdu\hdulist.py -> build\lib.win-amd64-3.8\pyfits\hdu
  copying pyfits\hdu\image.py -> build\lib.win-amd64-3.8\pyfits\hdu
  copying pyfits\hdu\nonstandard.py -> build\lib.win-amd64-3.8\pyfits\hdu
  copying pyfits\hdu\streaming.py -> build\lib.win-amd64-3.8\pyfits\hdu
  copying pyfits\hdu\table.py -> build\lib.win-amd64-3.8\pyfits\hdu
  copying pyfits\hdu\__init__.py -> build\lib.win-amd64-3.8\pyfits\hdu
  creating build\lib.win-amd64-3.8\pyfits\scripts
  copying pyfits\scripts\fitscheck.py -> build\lib.win-amd64-3.8\pyfits\scripts
  copying pyfits\scripts\fitsdiff.py -> build\lib.win-amd64-3.8\pyfits\scripts
  copying pyfits\scripts\fitsheader.py -> build\lib.win-amd64-3.8\pyfits\scripts
  copying pyfits\scripts\fitsinfo.py -> build\lib.win-amd64-3.8\pyfits\scripts
  copying pyfits\scripts\__init__.py -> build\lib.win-amd64-3.8\pyfits\scripts
  creating build\lib.win-amd64-3.8\pyfits\tests
  copying pyfits\tests\test_checksum.py -> build\lib.win-amd64-3.8\pyfits\tests
  copying pyfits\tests\test_core.py -> build\lib.win-amd64-3.8\pyfits\tests
  copying pyfits\tests\test_diff.py -> build\lib.win-amd64-3.8\pyfits\tests
  copying pyfits\tests\test_division.py -> build\lib.win-amd64-3.8\pyfits\tests
  copying pyfits\tests\test_groups.py -> build\lib.win-amd64-3.8\pyfits\tests
  copying pyfits\tests\test_hdulist.py -> build\lib.win-amd64-3.8\pyfits\tests
  copying pyfits\tests\test_header.py -> build\lib.win-amd64-3.8\pyfits\tests
  copying pyfits\tests\test_image.py -> build\lib.win-amd64-3.8\pyfits\tests
  copying pyfits\tests\test_nonstandard.py -> build\lib.win-amd64-3.8\pyfits\tests
  copying pyfits\tests\test_structured.py -> build\lib.win-amd64-3.8\pyfits\tests
  copying pyfits\tests\test_table.py -> build\lib.win-amd64-3.8\pyfits\tests
  copying pyfits\tests\test_uint.py -> build\lib.win-amd64-3.8\pyfits\tests
  copying pyfits\tests\test_util.py -> build\lib.win-amd64-3.8\pyfits\tests
  copying pyfits\tests\util.py -> build\lib.win-amd64-3.8\pyfits\tests
  copying pyfits\tests\__init__.py -> build\lib.win-amd64-3.8\pyfits\tests
  creating build\lib.win-amd64-3.8\pyfits\tests\data
  copying pyfits\tests\data\arange.fits -> build\lib.win-amd64-3.8\pyfits\tests\data
  copying pyfits\tests\data\ascii.fits -> build\lib.win-amd64-3.8\pyfits\tests\data
  copying pyfits\tests\data\blank.fits -> build\lib.win-amd64-3.8\pyfits\tests\data
  copying pyfits\tests\data\checksum.fits -> build\lib.win-amd64-3.8\pyfits\tests\data
  copying pyfits\tests\data\comp.fits -> build\lib.win-amd64-3.8\pyfits\tests\data
  copying pyfits\tests\data\fixed-1890.fits -> build\lib.win-amd64-3.8\pyfits\tests\data
  copying pyfits\tests\data\memtest.fits -> build\lib.win-amd64-3.8\pyfits\tests\data
  copying pyfits\tests\data\o4sp040b0_raw.fits -> build\lib.win-amd64-3.8\pyfits\tests\data
  copying pyfits\tests\data\random_groups.fits -> build\lib.win-amd64-3.8\pyfits\tests\data
  copying pyfits\tests\data\scale.fits -> build\lib.win-amd64-3.8\pyfits\tests\data
  copying pyfits\tests\data\stddata.fits -> build\lib.win-amd64-3.8\pyfits\tests\data
  copying pyfits\tests\data\table.fits -> build\lib.win-amd64-3.8\pyfits\tests\data
  copying pyfits\tests\data\tb.fits -> build\lib.win-amd64-3.8\pyfits\tests\data
  copying pyfits\tests\data\tdim.fits -> build\lib.win-amd64-3.8\pyfits\tests\data
  copying pyfits\tests\data\test0.fits -> build\lib.win-amd64-3.8\pyfits\tests\data
  copying pyfits\tests\data\zerowidth.fits -> build\lib.win-amd64-3.8\pyfits\tests\data
  running build_ext
  running pre_hook stsci.distutils.hooks.numpy_extension_hook for command build_ext

  Numpy is required to build this package.
  Please install Numpy on your system first.

  hook stsci.distutils.hooks.numpy_extension_hook raised exception: 1

 Traceback (most recent call last):
    File "c:\users\jcgob\appdata\local\temp\pip-install-o2m2ka8w\pyfits\.eggs\stsci.distutils-0.3.7-py3.8.egg\stsci\distutils\hooks.py", line 406, in numpy_extension_hook
      import numpy
    ModuleNotFoundError: No module named 'numpy'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "c:\users\jcgob\appdata\local\temp\pip-install-o2m2ka8w\pyfits\.eggs\d2to1-0.2.12.post1-py3.8.egg\d2to1\util.py", line 504, in run_command_hooks
      hook_obj(cmd_obj)
    File "c:\users\jcgob\appdata\local\temp\pip-install-o2m2ka8w\pyfits\.eggs\stsci.distutils-0.3.7-py3.8.egg\stsci\distutils\hooks.py", line 414, in numpy_extension_hook
      sys.exit(1)
  SystemExit: 1

  ----------------------------------------
ERROR: Failed to install pyfits==3.4 from https://files.pythonhosted.org/packages/45/98/d6d25932e6a82fa8456d38ab307bfb8945a1e1dd4e896730555e3b61cfc5/pyfits-3.4.tar.gz.
For assistance, please raise an issue at https://github.com/chaquo/chaquopy/issues.
Chaquopy: Exit status 1

> Task :app:generateDebugPythonRequirements FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:generateDebugPythonRequirements'.
> Process 'command 'py'' finished with non-zero exit value 1

  To view full details in Android Studio:
  * In version 3.6 and newer, click the 'Build: failed' caption to the left of this message.
  * In version 3.5 and older, click the 'Toggle view' button to the left of this message.
  * Then scroll up to see the full output.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 35s
15 actionable tasks: 1 executed, 14 up-to-date

jcgoble3 commented 3 years ago

As an additional note, pyfits contains a C component, but it is optional and we are intending to use it without that component, i.e. as a pure Python package.

mhsmith commented 3 years ago

setup.py scripts can't import other packages being installed at the same time. There's no way that could work anyway, because those packages contain native components built for Android, not for your build machine.

If the extension is truly optional, then you can probably work around this by making a small change to the pyfits source code:

Download the sdist from PyPI, and unpack it.
Edit setup.cfg to remove the reference to numpy_extension_hook. You might also have to remove the [extension=pyfits.compression] block.
Then either:
- Run setup.py bdist_wheel, copy the resulting wheel into your app source tree, and change your build.gradle file to install it using a local filename, as shown in the documentation; OR
- Run setup.py sdist, and use the resulting sdist in the same way.

jcgoble3 commented 3 years ago

numpy and the optional C component inside pyfits (i.e. compression) are independent of each other. numpy IS needed, but the compression component is not.

I'll give this a try next week (I'm off work for the weekend), but I can't guarantee it will work if numpy_extension_hook is removed (I haven't investigated what it does).

mhsmith commented 3 years ago

It looks like the only thing numpy_extension_hook does is locate the native Numpy header files. Once you remove the extension, I don't think those headers will be required anymore. Of course, Numpy will still be needed at runtime, but that's no problem.

jcgoble3 commented 3 years ago

Yep, that looks right. I have another project for today and am off tomorrow but will try this later this week unless I get bored today.

mhsmith commented 3 years ago

I hope you managed to solve your problem. If you need more help, please reopen this issue and give details.

jcgoble3 commented 3 years ago

Apologies for failing to check back. I ultimately cloned the pyfits repo from GitHub, checked out the 3.4 tag, removed the C files and the numpy_extension_hook, and created a local wheel to install from. That ended up being successful, and everything is now working. Thanks for the help!

chaquo / chaquopy

How do I force pip packages to be installed in a specific order? #351