Astropy & Photutils Libraries

[dkarakay]

I am trying to build Astropy (https://pypi.org/project/astropy/5.0/#files) and Photutils (https://pypi.org/project/photutils/) libraries for my Android application. I am trying to follow the steps on this webpage (https://github.com/chaquo/chaquopy/tree/master/server/pypi) to create android based whl files for my libraries.

I am trying to build whl files, but I saw this warning. I am not sure about the version 0.0.0 statement. Is it not going to create whl file? Or is the whl output corrupted? What should I do to create versions for my Android app?

Thank you.

My setup:

• Mac M1 Pro
• Python 3.8.16
• Chaquopy version 14.02 (if it is suitable for this repo as well)
• I do not use Docker

[dkarakay]

I installed NDK but I cannot find the "linux-x86_64" directory. I only have "darwin-x86_64" under "sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/" What should I do to change os.environ['AR']?

Do I need a Linux such as Ubuntu to build without a problem?

[dkarakay]

You can just ignore my old message. I used Ubuntu this time and I am stuck here:

Thank you in advance.

[mhsmith]

I installed NDK but I cannot find the "linux-x86_64" directory. I only have "darwin-x86_64" under "sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/"

As the README says, the tool only runs on Linux, not Mac.

[mhsmith]

The "version 0.0.0" issue was caused by build-wheel currently disabling PEP 517, which astropy depends on in order to get its version number from setuptools-scm. I was able to work around this using the following meta.yaml:

package:
  name: astropy
  version: 5.1

requirements:
  build:
    - cython 0.29.32
    - extension-helpers 1.0.0
    - setuptools-scm 7.1.0

And the following patch, which you should put in astropy/patches/chaquopy.patch:

--- src-original/setup.py       2022-05-24 19:43:02.000000000 +0000
+++ src/setup.py        2023-02-13 12:07:48.893006417 +0000
@@ -65,4 +65,5 @@
 from setuptools import setup  # noqa
 from extension_helpers import get_extensions  # noqa

-setup(ext_modules=get_extensions())
+from setuptools_scm import get_version
+setup(ext_modules=get_extensions(), version=get_version())

This allowed the build to complete, though I haven't tested it.

It looks like the build used NumPy headers from the build environment, which may cause problems. If so, you may need to follow the example in packages/pandas by adding numpy to the requirements file, and adding another small patch to allow the build to use it. See the scikit-learn notes in the README for how to get the numpy wheel files.

[mhsmith]

I was not able to reproduce your last error: --fix-cortex-a53-843419 is only supported on AArch64 targets. If you need more help, please post the following, and as text, not screenshots:

• Your meta.yaml file
• Any patches you've applied
• The full build log, including the build-wheel command line. You can attach it as a file if it's too large to paste.

[dkarakay]

This allowed the build to complete, though I haven't tested it.

I did not use any patch while trying to build it. What is the patch for and is your example enough for me?

It looks like the build used NumPy headers from the build environment, which may cause problems. If so, you may need to follow the example in packages/pandas by adding numpy to the requirements file, and adding another small patch to allow the build to use it. See the scikit-learn notes in the README for how to get the numpy wheel files.

In fact, astropy requires NumPy, extension-helpers and pyerfa. I tried to add NumPy and extension-helpers in my requirements.txt file and I was able to pass Gradle build without any error. However, I need to export a whl for Pyerfa too. Therefore while trying to build the whl file, I installed Numpy, extension-helpers and pyerfa using pip beforehand to overcome errors.

My aim is to utilize photutils (https://photutils.readthedocs.io/en/stable/install.html) library which depends on Numpy and Astropy. I will try your meta.yml file and your patch and share the results.

Thank you for your help and your detailed reply!

[dkarakay]

Also, I checked the requirements for astropy and saw that the Numpy version should be higher (>1.20) than the original one for Python 3.8 (1.19.5.0). If I want to work with Python 3.9 or Python 3.10 (totally fine for me), then I have a problem with SciPy since it does not have any version for Python 3.9 or Python 3.10. What do you think I should do? Should I work to generate a 3.9 version for SciPy or another way around? Is this achievable? Thank you

[dkarakay]

I tried to build scipy with the existing meta.yml file and patches with Python 3.9:

The log I got:

log_scipy_39.txt

[mhsmith]

It's not possible to build SciPy with the current version of the tool, because it doesn't have Fortran support yet.

However, we actually do have some newer NumPy wheels for Python 3.8: see pypi-extra. The reason they're not in the main repository is because they're incompatible with our old version of TensorFlow (https://github.com/chaquo/chaquopy/issues/559#issuecomment-989318791), but they're otherwise fine, so you should be able to use them for this.

I'm starting to lose track of all the different things you're building, so if you need any more help, please make a GitHub fork of this repository and push your changes to it, so I can see them all in one place.

[dkarakay]

Thank you for sharing! These NumPy whl files will be fine for me. I will try to create whls for these libraries:

• pyerfa
• astropy
• photutils

This is my aim. I will try to build android versions of these libraries. I will let you know with my GitHub fork soon. Thank you again

[dkarakay]

I tried to build this for astropy but which ABI did you use?

[dkarakay]

You can find my fork here:

Fork

This is the error I encountered:

  ld: error: /home/deniz/Android/Sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/lib/arm-linux-androideabi/21/crtbegin_so.o is incompatible with elf64-x86-64
  ld: error: /home/deniz/dev/fork/chaquopy/server/pypi/packages/astropy/build/5.1/cp38-cp38-android_21_armeabi_v7a/requirements/chaquopy/lib/libpython3.8.so is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/astropy_wcs.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/astropy_wcs_api.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/distortion.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/distortion_wrap.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/docstrings.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/pipeline.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/pyutil.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/sip.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/sip_wrap.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/str_list_proxy.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/unit_list_proxy.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/util.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/wcslib_auxprm_wrap.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/wcslib_celprm_wrap.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/wcslib_prjprm_wrap.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/wcslib_tabprm_wrap.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/wcslib_wrap.o is incompatible with elf64-x86-64
  ld: error: build/temp.linux_armv7l-cpython-38/astropy/wcs/src/wcslib_wtbarr_wrap.o is incompatible with elf64-x86-64
  ld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
  error: command '/home/deniz/Android/Sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/armv7a-linux-androideabi21-clang' failed with exit code 1
error
  ERROR: Failed building wheel for astropy
  Running setup.py clean for astropy
  Running command /home/deniz/.virtualenvs/chaq/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/deniz/dev/fork/chaquopy/server/pypi/packages/astropy/build/5.1/cp38-cp38-android_21_armeabi_v7a/src/setup.py'"'"'; __file__='"'"'/home/deniz/dev/fork/chaquopy/server/pypi/packages/astropy/build/5.1/cp38-cp38-android_21_armeabi_v7a/src/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
  /home/deniz/.virtualenvs/chaq/lib/python3.8/site-packages/setuptools/config/setupcfg.py:520: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
    warnings.warn(msg, warning_class)
  Chaquopy: clean command disabled
Failed to build astropy
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-ufj0dngm'
ERROR: Failed to build one or more wheels
Exception information:
Traceback (most recent call last):
  File "/home/deniz/.virtualenvs/chaq/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
    status = self.run(options, args)
  File "/home/deniz/.virtualenvs/chaq/lib/python3.8/site-packages/pip/_internal/commands/wheel.py", line 171, in run
    raise CommandError(
pip._internal.exceptions.CommandError: Failed to build one or more wheels
build-wheel.py: Error: Command returned exit status 1

[mhsmith]

I've created a pull request #846 based on your fork, with an additional change to add numpy to the requirements file. With this recipe, I was able to complete the build for Python 3.8 on all 4 ABIs, though I still haven't tested the result.

See above for where to get the numpy wheel files: https://github.com/chaquo/chaquopy/issues/812#issuecomment-1429334730

If you have any more problems, please save the FULL build log to a file, and attach it by dragging it into the comment box.

[dkarakay]

It is good to know! I will try it as soon as possible. I will also need pyerfa and photutils libraries to complete my system. After trying astropy, I will try to compile these libraries. Thank you for your help

[dkarakay]

I encountered errors while trying to build astropy on my own. I tried Armeabi, arm64 and other versions but did not solve it. Is it related to my NDK version? I am using: 22.1.7171670 astropy_armeabi_log.txt

[dkarakay]

Could you share the whl files so that I can test the astropy

[mhsmith]

Sorry, I'm very busy at the moment so I won't have time to look at this for the next few weeks.

[dkarakay]

I see thank you for your help. I will try to utilize different NDK versions maybe I can solve it

[mhsmith]

The cause of the failure is probably the -L/usr/lib/x86_64-linux-gnu in the last linker command. For some reason, that argument doesn't appear when I do the build myself. Your NDK version is correct, so that's not the problem.

Anyway, I've copied the wheels I built to https://chaquo.com/pypi-test/, so you can use them by adding an --extra-index-url line to your build.gradle file as shown here.

[dkarakay]

Thank you for your reply! I will test those and will let you know about the outcome. In addition, I want to build Photutils that is based on Astropy

[dkarakay]

Sorry for the late reply, I tried to test build Astropy, but encountered this error:

"Failed to install pyerfa>=2.0 from https://files.pythonhosted.org/packages/a7/51/b1b8770853d82726dfa5d1079de29c32f144e10eb65b3852b1cd2b39f341/pyerfa-2.0.0.3.tar.gz#sha256=d77fbbfa58350c194ccb99e5d93aa05d3c2b14d5aad8b662d93c6ad9fff41f39 (from astropy==5.1.0)."

The error comes from the dependency of astropy, namely pyerfa. Is there an active development on pyerfa?

I tried to build pyerfa on my fork

This is the error I got pyerfa.log I do not what I should add like you did in previous ones so that I cannot bother you.

My goal is to test the wheel's status by attempting one of the samples provided by the Astropy package.

[dkarakay]

I also need to have one last package Photutils that depends on Astropy. I tried to build on my forked repo, and add your wheel files in dist/astropy folders, but it seems the system cannot detect the imported wheel files. https://github.com/dkarakay/chaquopy/blob/master/server/pypi/photutils_armeabi_log.txt

I solved it, I forgot to remove 0 original version was 5.1. Updated error log can be found: https://github.com/dkarakay/chaquopy/blob/master/server/pypi/photutils_armeabi_log.txt

[mhsmith]

Your pyerfa log contains the same error message as astropy ("is incompatible with elf64-x86-64"), and probably has the same cause (-L/usr/lib/x86_64-linux-gnu appearing on the linker command line).

I don't know why this is happening to you, but it may have something to do with this:

Therefore while trying to build the whl file, I installed Numpy, extension-helpers and pyerfa using pip beforehand to overcome errors.

In general, you should not pip install the requirements of a package before building it. Usually these requirements aren't needed at build time at all. And if they are, they'll often need to be Android builds, not the Linux builds which pip install will give you.

The only packages that need to be available at build time are those which are listed in setup_requires in setup.py, or [build-system] requires in pyproject.toml. These packages should be listed in the build section of the meta.yaml file if they're pure-Python (e.g. extension-helpers), or the host section if they're not (e.g. numpy). Usually you can tell the difference by looking at the "Download files" section of the package's PyPI page. A pure-Python package will have a single none-any wheel, while a non-pure package will have many different wheels for each combination of Python version and operating system.

So before you continue, please delete your virtual environment and re-create it from server/pypi/requirements.txt.

Meanwhile, I've updated #846 to add numpy as a host requirement of pyerfa, exactly as I did for astropy. Your photutils log shows that it's including a native NumPy header, so you'll have to do something similar for that package.

I've copied my (untested) pyerfa wheels to https://chaquo.com/pypi-test/.

[dkarakay]

I believe pyerfa works well after I used your wheels. I tried to delete and recreate a new environment based on server/pypi/requirements.txt and I did not install any dependent library such as numpy etc, but could not solve the problem. Still, I got the same errors. I tried to build photutils as you explained but still got same errors. I updated my fork to add host, build and a similar patch to astropy

Apart from that, I tried to test astropy on mobile but encountered such error:

The code that I used original was from here

import astropy.coordinates as coord
import astropy.units as u

def rv_to_gsr(c, v_sun=None):
    """Transform a barycentric radial velocity to the Galactic Standard of Rest
    (GSR).

    The input radial velocity must be passed in as a

    Parameters
    ----------
    c : `~astropy.coordinates.BaseCoordinateFrame` subclass instance
        The radial velocity, associated with a sky coordinates, to be
        transformed.
    v_sun : `~astropy.units.Quantity`, optional
        The 3D velocity of the solar system barycenter in the GSR frame.
        Defaults to the same solar motion as in the
        `~astropy.coordinates.Galactocentric` frame.

    Returns
    -------
    v_gsr : `~astropy.units.Quantity`
        The input radial velocity transformed to a GSR frame.

    """
    if v_sun is None:
        v_sun = coord.Galactocentric().galcen_v_sun.to_cartesian()

    gal = c.transform_to(coord.Galactic)
    cart_data = gal.data.to_cartesian()
    unit_vector = cart_data / cart_data.norm()

    v_proj = v_sun.dot(unit_vector)

    return c.radial_velocity + v_proj

def main():
    coord.galactocentric_frame_defaults.set("latest")

    icrs = coord.SkyCoord(
        ra=258.58356362 * u.deg,
        dec=14.55255619 * u.deg,
        radial_velocity=-16.1 * u.km / u.s,
        frame="icrs",
    )

    v_sun = coord.Galactocentric().galcen_v_sun.to_cartesian()

    gal = icrs.transform_to(coord.Galactic)
    cart_data = gal.data.to_cartesian()
    unit_vector = cart_data / cart_data.norm()

    v_proj = v_sun.dot(unit_vector)

    rv_gsr = icrs.radial_velocity + v_proj
    print(rv_gsr)

    rv_gsr = rv_to_gsr(icrs)
    print(rv_gsr)
    return rv_gsr

I encountered such error, I checked but could not find the error. It seems the error is based on Astropy, but although I checked the mentioned file I did not find the error.

ValueError: 'm / (s)' did not parse as unit:
[Errno 2] No such file or directory: '/data/ data/com.denizkarakay.stellandroid/files/ chaquopy/AssetFinder/requirements/ astropy/units/format/generic_parsetab.py'
If this is meant to be a custom unit, define it with 'u.def_unit'. To have it recognized inside a file reader or other code, enable it with 'u.add_enabled_units'. For details, see https://docs.astropy.org/en/latest/units /combining_and_defining.html

It would be great if you are able to build Photutils for me. Although I am not sure about Astropy code, I have more experience and knowledge about Photutils.

Thank you for your support

[mhsmith]

It looks like the Astropy error is caused by it attempting to read its own Python code. This can be worked around by adding extractPackages("astropy") to your build.gradle file, as explained here.

[mhsmith]

Your photutils recipe worked first time for me, although, as I explained here, there's no need for photutils to have a host requirement for astropy, because it's not listed in [build-system] requires or setup_requires.

I've updated #846, and copied my (untested) wheels to https://chaquo.com/pypi-test/.

Looking at your photutils build log, it doesn't mention applying the patch. Are you sure you've tried to build it since adding the patch file? Without the patch, the numpy include directory will be silently omitted because of this code in extension-helpers.

[dkarakay]

Hi @mhsmith, I tested astropy and after adding extractPackages "astropy" everything seems fine. I was trying your photutils wheels. I am trying to run a custom script which we try to detect the stars.

After running it I encountered such error:

Process: com.denizkarakay.stellandroid, PID: 19541
com.chaquo.python.PyException: ImportError: Convolution C extension is missing. Try re-building astropy.
    at <python>.astropy.convolution.convolve.<module>(convolve.py:30)
    at <python>.java.chaquopy.import_override(import.pxi:60)
    at <python>.astropy.convolution.<module>(__init__.py:4)
    at <python>.java.chaquopy.import_override(import.pxi:60)
    at <python>.photutils.utils._convolution.<module>(_convolution.py:9)
    at <python>.java.chaquopy.import_override(import.pxi:60)
    at <python>.photutils.detection.daofinder.<module>(daofinder.py:15)
    at <python>.java.chaquopy.import_override(import.pxi:60)
    at <python>.photutils.detection.<module>(__init__.py:8)
    at <python>.java.chaquopy.import_override(import.pxi:60)
    at <python>.photutils.<module>(__init__.py:17)
    at <python>.java.chaquopy.import_override(import.pxi:60)
    at <python>.star_detector.<module>(star_detector.py:7)
    at <python>.importlib._bootstrap._call_with_frames_removed(<frozen importlib._bootstrap>:219)
    at <python>.importlib._bootstrap_external.exec_module(<frozen importlib._bootstrap_external>:843)
    at <python>.java.android.importer.exec_module(importer.py:554)
    at <python>.java.android.importer.exec_module(importer.py:606)
    at <python>.importlib._bootstrap._load_unlocked(<frozen importlib._bootstrap>:671)
    at <python>.importlib._bootstrap._find_and_load_unlocked(<frozen importlib._bootstrap>:975)
    at <python>.importlib._bootstrap._find_and_load(<frozen importlib._bootstrap>:991)
    at <python>.importlib._bootstrap._gcd_import(<frozen importlib._bootstrap>:1014)
    at <python>.importlib.import_module(__init__.py:127)
    at <python>.chaquopy_java.Java_com_chaquo_python_Python_getModuleNative(chaquopy_java.pyx:129)
    at com.chaquo.python.Python.getModuleNative(Native Method)
    at com.chaquo.python.Python.getModule(Python.java:84)
    at com.denizkarakay.stellandroid.MainActivity.onCreate$lambda$5(MainActivity.kt:185)
    at com.denizkarakay.stellandroid.MainActivity.$r8$lambda$CS3izk6H0rs3tSCu4sIu2HZ9OrI(Unknown Source:0)
    at com.denizkarakay.stellandroid.MainActivity$$ExternalSyntheticLambda4.onClick(Unknown Source:2)
    at android.view.View.performClick(View.java:7448)
    at com.google.android.material.button.MaterialButton.performClick(MaterialButton.java:1211)
    at android.view.View.performClickInternal(View.java:7425)
    at android.view.View.access$3600(View.java:810)
    at android.view.View$PerformClick.run(View.java:28305)
    at android.os.Handler.handleCallback(Handler.java:938)
    at android.os.Handler.dispatchMessage(Handler.java:99)
    at android.os.Looper.loop(Looper.java:223)
    at android.app.ActivityThread.main(ActivityThread.java:7656)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947)

As it can be seen, this is the problem:

com.chaquo.python.PyException: ImportError: Convolution C extension is missing. Try re-building astropy.

The function that I utilized inside photutils using Convolution and I need to utilize convolution. Did you encounter such error or do you have any suggestions to overcome?

Thank you in advance

[dkarakay]

I did not encounter such error while trying to run everything on the same environment on my local machine.

[mhsmith]

Thanks for the report: I've created #892 to track the general issue. Meanwhile, you can work around it by running the following before you import photutils:

import astropy
import ctypes
ctypes.CDLL(f"{astropy.__path__[0]}/convolution/_convolve.so")

[dkarakay]

It is working right now! Thank you for your support and your help! I believe after some testing, you can add Astropy and Photutils libraries to package repository as well!

[mhsmith]

--fix-cortex-a53-843419 is only supported on AArch64 targets.

ld: error: /home/deniz/Android/Sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/lib/arm-linux-androideabi/21/crtbegin_so.o is incompatible with elf64-x86-64

These were both caused by #909, which is fixed in the current version of build-wheel.

[mhsmith]

Thanks for the report: I've created #892 to track the general issue.

Based on https://github.com/astropy/astropy/pull/14035, it looks like this might no longer be a problem in Astropy 5.2, so I'll try building 5.2.2, which is the last version to support Python 3.8.

[mhsmith]

Unfortunately Astropy 5.2 requires NumPy 1.20, so we can't release it into the main repository for Python 3.8. So I'll go back to Astropy 5.1.1 and patch it in the same way as spectrum (#892).

[mhsmith]

The Photutils tutorial, which I'm using to write the tests, requires SciPy. So that means I can only release it for Python 3.8, and to be compatible with our current SciPy version, I'll have to go back to Photutils 1.1.0.

I'll leave the builds of Photutils 1.8.0 in pypi-extra, but I have not tested them.

[mhsmith]

The following packages are now in the public repository:

• pyerfa 2.0.0.3 for Python 3.8, 3.9 and 3.10
• astropy 5.1.1 for Python 3.8, 3.9 and 3.10
• photutils 1.1.0 for Python 3.8 only

As mentioned above, to use astropy you must add the line extractPackages("astropy") to your build.gradle file. See here for more details.