Trying to build a wheel file for netcdf4 and getting errors

[githubweather]

I am trying to build an app which requires python scripts that need to import netcdf4. When I do "pip install netcdf4" as seen in the python block in the build.gradle file below and try to build the app in Android Studio, I get:

"CCompiler.compile: Chaquopy cannot compile native code" among other errors.

As such it was recommended to me by mhsmith that I build a wheel file for netcdf4 following the instructions at: https://github.com/chaquo/chaquopy/tree/master/server/pypi

For reference, I am creating the wheel file on this VM: Linux debian 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20) x86_64 GNU/Linux

I have had to make a few minor mods to the build-wheel.py file and get to this part of build-wheel.py --

run(f"{self.pip} wheel --no-deps "
            # --no-clean doesn't currently work: see env/python/sitecustomize.py.
            #
            # We pass -v unconditionally, because we always want to see the build process
            # output. --global-option=-vv enables additional distutils logging.
            f"-v {'--global-option=-vv ' if self.verbose else ''}"
            f"-e .")

and it outputs a string of errors such as the following --

Building wheel for numpy (setup.py): still running...
    Building wheel for numpy (setup.py): finished with status 'error'
    error: subprocess-exited-with-error

    × python setup.py bdist_wheel did not run successfully.
    │ exit code: 1
    ╰─> [4890 lines of output]
        Running from numpy source directory.
        blas_opt_info:
        blas_mkl_info:
        customize UnixCCompiler
          libraries mkl_rt not found in ['/home/me/PYTHON3.8/lib', '/usr/loc
al/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
          NOT AVAILABLE

        blis_info:
        customize UnixCCompiler
          libraries blis not found in ['/home/me/PYTHON3.8/lib', '/usr/local
/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
          NOT AVAILABLE

        openblas_info:
openblas_info:
        customize UnixCCompiler
        customize UnixCCompiler
          libraries openblas not found in ['/home/me/PYTHON3.8/lib', '/usr/l
ocal/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
          NOT AVAILABLE

        atlas_3_10_blas_threads_info:
        Setting PTATLAS=ATLAS
        customize UnixCCompiler
          libraries tatlas not found in ['/home/me/PYTHON3.8/lib', '/usr/loc
al/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu']
          NOT AVAILABLE
etc.
...

However, I do see libmkl_rt.so.2 in /home/me/PYTHON3.8/lib. How can I make build-wheel.py see it? Via "LDFLAGS", or "LD_LIBRARY_PATH"? Regarding openblas, I built it 'by hand' and it installed to /opt/OpenBLAS/lib. Again, how to make build-wheel.py see this path?

Below is the portion of build.gradle that should be relevant--

python{
          version "3.8"
       //     version "3.11" -- this version fails gradle syncing so stick with v3.8
       //testing     version "3.9"
            pip {
                install "numpy"
    //            install "netcdf4-python"
                install "netcdf4"
            }
        }

ndk {
            abiFilters "armeabi-v7a", "arm64-v8a", "x86", "x86_64"
        }

plugins {
    id 'com.android.application' version '7.2.2' apply false
    id 'com.android.library' version '7.2.2' apply false
    id 'com.chaquo.python' version '14.0.2' apply false
}

Build log error from Android Studio:

Let me know if I need to add more to make this understandable.

Thanks much.

[mhsmith]

Please create a fork of this repository, and post the link here. Make sure you include:

• The changes you made to build-wheel.py
• Your meta.yaml file for netcdf4
• Any patches you applied to netcdf4

If anyone else wants this package too, let us know by clicking the thumbs-up button above.

[githubweather]

Took a second... Created a fork of the chaquopy GitHub site. It is at located at https://github.com/githubweather/chaquopy. Uploaded my meta.yaml, and a file called seeit.build_wheel.changes which is from the command 'diff build-wheel.py.orig build-wheel.py > seeit.build_wheel.changes'. The latter file, build-wheel.py being the file with my changes. Also created a file called README.githubweather which is a general summary of what I stated here.

[githubweather]

Forgot to add - I have made no edits to the netcdf4 source to date.

[mhsmith]

Your changes to build-wheel are very difficult to understand in this format. Please actually modify build-wheel in the repository and commit it, instead of adding a separate patch file. It would also be useful if your changes included comments explaining why they were necessary. But I suspect most of them actually won't be necessary if you follow the instructions below correctly.

Based on a quick look at the netcdf4 setup.py, you should probably be listing numpy as a host requirement rather than a build one. This will require you to download the Android NumPy wheel into the dist directory, as described in the scikit-learn section of server/pypi/README.md.

Since this is a Python interface to a native library, and the library doesn't appear to come bundled with the Python package, you'll have to build the library as a separate wheel. The two wheels will have the same relationship as pyzmq and chaquopy-libzmq, as described in the README.

Finally, it looks like this package may require the hdf5 library. We've already built this for Android, so you should add that to the host requirements section and download it in the same way as NumPy. See packages/h5py for an existing package that does this.

Sorry about how complicated this is, but some packages are just like that.

[githubweather]

Greatly appreciate your assistance with this project. Currently trying to implement your suggestions. Downloaded numpy-1.23.3-0-cp310-cp310_21_arm64_v8a.whl and placed in server/pypi/dist/numpy. Edited meta.yaml, moving numpy reference to 'host' section as '- numpy 1.23.3-0'. Also tried '- numpy 1.23.3' and keep getting cannot find wheel file for requirement numpy 1.23.3-0 (or numpy 1.23.3). What am I doing wrong?

[mhsmith]

numpy 1.23.3 should work correctly. But this wheel file is for Python 3.10 and arm64-v8a. Does that match the settings you're passing to build-wheel.py?

If you need more help, please post the full build log, including the build-wheel.py command line, and your current meta.yaml file.

[githubweather]

I was able to get things right regarding the whl files for numpy and h5py since I needed the ones for python 3.8 and arm64_v8a. Thx for that advice. Now I'm trying to grasp the other part you mentioned --

"Since this is a Python interface to a native library, and the library doesn't appear to come bundled with the Python package, you'll have to build the library as a separate wheel. The two wheels will have the same relationship as pyzmq and chaquopy-libzmq, as described in the README."

Which README are you referring to?

[mhsmith]

This one: https://github.com/chaquo/chaquopy/blob/master/server/pypi/README.md

[githubweather]

Based on your suggestions I have broken things out as follows --

chaquopy-netcdf4 directory -- meta.yaml --

{% set version = "4.9.2" %}

package:
  name: chaquopy-netcdf4
  version: {{ version }}

source:
  git_url: https://github.com/Unidata/netcdf-c.git
  git_rev: v{{ version }}

build:
  number: 1

build.sh --

#!/bin/bash
set -eu

configure --host=aarch64-linux-gnu --disable-netcdf4 --disable-dap --disable-byterange
make all install

======================================================================================== netcdf4-python directory -- meta.yaml --

{% set version = "1.6.2" %}

package:
  name: netcdf4-python
  version: {{ version }}

source:
  git_url: https://github.com/Unidata/netcdf4-python.git
  git_rev: v{{ version }}

build:
  number: 1

requirements:
  build:
    - cftime 1.6.2
  host:
    - numpy 1.17.4
    - h5py 2.10.0
    - chaquopy-netcdf 4.9.2
    - python

======================================================================================= in /dist, I have /numpy and h5py/ containing the relevant (python 3.8 and arm64_v8a) wheel files. Also, have gone back to the original build-wheel.py (not edited by me).
My kickoff script is: build-wheel.py --python 3.8 --abi arm64-v8a chaquopy-netcdf4

The current errors are: undefined symbols compress2, inflate, inflateEnd referenced by H5Zdeflate.c.

These look to be required by the native netcdf-c code and are defined in zlib (current version is 1.2.13). How can I include zlib in the build?

[githubweather]

I am so sorry. I thought I had inadvertently closed this issue earlier but now find it is not. Good. Pls disregard the 'new' issue of the same title that I created.

Now, back to this issue, since I failed with the pasting of my build.log file (it was too many characters), is there any. way to attach a file?

[mhsmith]

I pasted my build log and did a preview. It looked like it pasted ok, but then I got a message indicating my comment (build.log) had too many characters. Is it possible to attach a build log?

You can attach any file by dragging it into the comment box.

[githubweather]

build.log

[githubweather]

The build.log above comes from executing: "build-wheel.py --python 3.8 --abi arm64-v8a chaquopy-netcdf4". Let me know how I can incorporate zlib into the native build of netcdf-c. In a 'regular' build on my VirtualBox Debian virtual machine I would add the LDFLAGS and LIB flags to configure. But not sure what to do within chaquopy. Does a 'wheel' file need to be created for zlib? Thanks for any clues.

[mhsmith]

zlib comes with Android itself, so all that should be necessary is to add -lz to the linker command line. I was able to do this by adding export LIBS=-lz to the start of build.sh.

However, I then got this error:

/bin/bash ../libtool  --tag=CC   --mode=link /home/smith/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android21-clang  -I/home/smith/git/chaquo/chaquopy/server/pypi/packages/chaquopy-netcdf
4/build/4.9.2/py3-none-android_21_arm64_v8a/requirements/chaquopy/include -fno-strict-aliasing   -module -shared -export-dynamic    -version-info 0:0:0 -rpath /home/smith/git/chaquo/chaquopy/server/pypi/packages/chaquopy-netcdf4/
build/4.9.2/py3-none-android_21_arm64_v8a/src/plugins/.libs -L/home/smith/git/chaquo/chaquopy/server/pypi/packages/chaquopy-netcdf4/build/4.9.2/py3-none-android_21_arm64_v8a/requirements/chaquopy/lib -Wl,--exclude-libs,libgcc.a -
Wl,--exclude-libs,libgcc_real.a -Wl,--exclude-libs,libunwind.a -Wl,--build-id=sha1 -Wl,--no-rosegment -lm -Wl,--no-undefined -o lib__nczstdfilters.la  NCZstdfilters.lo  -lm -lz
libtool: link: /home/smith/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android21-clang -shared  -fPIC -DPIC  .libs/NCZhdf5filters.o   -L/home/smith/git/chaquo/chaquopy/server/pypi/packages
/chaquopy-netcdf4/build/4.9.2/py3-none-android_21_arm64_v8a/requirements/chaquopy/lib -lm -lz  -Wl,--exclude-libs -Wl,libgcc.a -Wl,--exclude-libs -Wl,libgcc_real.a -Wl,--exclude-libs -Wl,libunwind.a -Wl,--build-id=sha1 -Wl,--no-r
osegment -Wl,--no-undefined   -Wl,-soname -Wl,lib__nczhdf5filters.so.0 -o .libs/lib__nczhdf5filters.so.0.0.0
ld: error: undefined symbol: nc_inq_var
>>> referenced by NCZhdf5filters.c
>>>               .libs/NCZhdf5filters.o:(NCZ_shuffle_modify_parameters)

ld: error: undefined symbol: nc_inq_type
>>> referenced by NCZhdf5filters.c
>>>               .libs/NCZhdf5filters.o:(NCZ_shuffle_modify_parameters)
clang: error: linker command failed with exit code 1 (use -v to see invocation)

This probably means that one of its libraries requires another one of its libraries, and it's not bothering to provide an -l option because it assumes they'll be loaded in a specific order at runtime. In such cases, the easiest solution is just to remove the -Wl,--no-undefined option, as is already done in chaquopy-hdf5/build.sh.

I also changed the netcdf build.sh file to be more consistent with the libzmq example. If you do need to deviate from the example, it would be useful if you explain why.

Here's the resulting build.sh file:

#!/bin/bash                                                                                                                                                                                                                                                                                                                                                       
set -eu

# Some internal libraries can't be built with this flag.                                                                                                                                                                                                                                                                                                          
LDFLAGS=$(echo $LDFLAGS | sed 's/-Wl,--no-undefined//')

./configure --host=$HOST --disable-netcdf4 --disable-dap --disable-byterange
make -j $CPU_COUNT all
make install prefix=$PREFIX

Finally, I've just pushed an update to build-wheel to let it detect COPYRIGHT as a license filename.

With those changes, the build of chaquopy-netcdf4 succeeds. However, I'm suspicious about some of those --disable flags you've added, especially --disable-netcdf4. If they're really necessary, please explain why.

[githubweather]

Thanks for this awesome progress. I tested build.sh without the --disable-netcdf4 flag and it yields "can't find or link to the hdf5 library". Apparently. netcdf uses hdf5 in some cases for compression, etc. Seems would definitely be optimal if hdf5 is part of the build. What would be the best way to incorporate hdf5 in this build?

[mhsmith]

See above: https://github.com/chaquo/chaquopy/issues/823#issuecomment-1480777699.

[githubweather]

yes - thanks for all your comments. I was trying to explain why I added the --disable-netcdf4 flag. When I looked this flag up (at the UCAR netcdf-c site) it is supposedly equivalent to adding the --enable-hdf5 flag, but I have not verified this yet. I noticed there is a wheel file for hdf5 which I added to this build. Not sure if it changes the build at all.

[githubweather]

Now I'm back on my Android Studio project and am trying to include the *.whl file generated thx to a lot of help from you. I have something wrong in getting the AS project to 'see' the .whl file now. I have uploaded the wheel file to my GitHub site (githubweather). Here are the particulars in the AS code:

python{
          version "3.8"
            pip {
                options "--extra-index-url","https://github.com/githubweather/chaquopy/blob/master/chaquopy_netcdf4-4.9.2-1-py3-none-android_21_arm64_v8a.whl"
                install "chaquopy-netcdf4"
            }
        }

The resulting error upon build attempt: "Could not find a version that satisfies the requirement chaquopy-netcdf4"

I clearly have not captured the nomenclature needed in the AS code. Pls let me know what I've got wrong here.

[mhsmith]

I'm not sure whether a GitHub website link will work with --extra-index-url, but in any case, it must point to a location laid out in the same way as the server/pypi/dist directory, with one subdirectory per package.

Rather than putting it on a web server, the easiest way to test your package would be to copy your dist directory to the machine where you're building your app, and then use --extra-index-url with the path to that directory.

Alternatively, if your abiFilters ONLY contains arm64-v8a, then you can put the path to the .whl file directly on the install line, with no need for an --extra-index-url at all.

[githubweather]

I went with your tip in paragraph 2 above. My AS code now looks like:

python{
          version "3.8"
            pip {
                options "--extra-index-url","file:///Users/me/TEMPY/home/me/WHEEL-TEST/chaquopy/server/pypi/dist/"
                install "chaquopy-netcdf4"
            }
        }

where chaquopy-netcdf4 is the directory under /Users/me/TEMPY/home/me/WHEEL-TEST/chaquopy/server/pypi/dist/ and where the wheel file resides. When I try to rebuild, it simply says:

"Exception:" with no information as to what the exception is. I hope I'm getting closer but still am missing something...

[mhsmith]

That should be fine, but you could also try removing the file:// prefix. If that doesn't work, please post the full build log.

By the way, to mark code blocks you need three backticks like this: ```, not two.

[githubweather]

I see a build log in the Android Studio window (Build Output tab). Was wondering where/if this same build log information is written to a file? When I run "adb logcat" it does not capture the build information that I see in the Android Studio window.

[mhsmith]

adb is for runtime logs only, not build logs.

The build log is shown in the Build Output window, and it isn't written to a file as far as I know. Unfortunately, the window defaults to showing individual error messages, which are often useless without context. To see the full build log, click the 'Build ... failed' caption to the left of the log.

[githubweather]

[githubweather]

Pls note the screen capture of the build log above.

[mhsmith]

Like I said, click the "Build ... failed" line in bold, then you'll see the full log in the text view to the right. If you don't see anything, you may have shrunk the text view to nothing, in which case you'll need to drag it out again.

If you need more help, please post:

• The full log
• The python section of your build.gradle file
• A listing of the directories underneath your --extra-index-url

[githubweather]

I see that when I right-click the "Build ... failed" line in bold, it allows one to "show warnings" and/or "show successful steps". The screenshot I posted were the warnings. I could screenshot the warnings and the successful steps. Would it be more hopeful if I post both?

Here's the python section of build.gradle:

python{
          version "3.8"
            pip {
                options "--extra-index-url","/Users/me/TEMPY/home/me/WHEEL-TEST/chaquopy/server/pypi/dist/"
                install "chaquopy-netcdf4"
            }
        }

Underneath the "--extra-index-url" directory is the directory "chaquopy-netcdf4". And in that directory is the wheel file, chaquopy_netcdf4-4.9.2-1-py3-none-android_21_arm64_v8a.whl

[githubweather]

should read "helpful" rather than "hopeful" in post above.

[githubweather]

There is one more part to the Android Studio build log that I am also seeing that is not in the screen capture above:

The index url "/Users/me/TEMPY/home/me/WHEEL-TEST/chaquopy/server/pypi/dist" seems invalid, please provide a scheme.

[githubweather]

Testing this recently created netcdf wheel file in my Android Studio project but getting an error. I will post the relevant build.gradle file and the python script call that is failing. Note that the Android Studio project is building perfectly.

          version "3.8"
            pip {
               install "numpy"
               install "/Users/me/TEMPY/home/me/WHEEL-TEST/chaquopy/server/pypi/dist/chaquopy-netcdf4/chaquopy_netcdf4-4.9.2-1-py3-none-android_21_x86.whl"
            }
        }
        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
        ndk {
            abiFilters "x86"
        }

The error:

04-19 18:47:22.513  4780  4780 E AndroidRuntime: Caused by: com.chaquo.python.PyException: ModuleNotFoundError: No module named 'netCDF4'

And this is the code in the Python script causing the error:

from netCDF4 import Dataset

Thanks for any ideas on how to proceed.

[mhsmith]

You've installed the underlying library chaquopy-netcdf4, but you need to install the Python netcdf4 package as well.

Also, you should match PyPI and call the Python package netCDF4 (note capitalization), rather than netcdf4-python. That way, any other packages that depend on it will be able to find it.

[githubweather]

I installed the wheel file built for chaquopy-netcdf4 plus found a wheel file for netcdf4-python. Here is the resulting build screen grab from AS --

Here is the right hand side that got cut off in the image grab above --

i.e., it looks like netcdf4 (python interface) depends on cftime. Do I need to look for a cftime wheel file next? Or would that be something to build?

Here is the relevant gradle portion currently--

python{
          version "3.8"

            pip {
            //    options "--extra-index-url","file:///Users/skirby/TEMPY/home/skirby/WHEEL-TEST/chaquopy/server/pypi/dist/"
  //test              options "--extra-index-url","/Users/skirby/TEMPY/home/skirby/WHEEL-TEST/chaquopy/server/pypi/dist/"

                // A requirement specifier, with or without a version number:
               install "numpy"

                install "/Users/skirby/TEMPY/home/skirby/WHEEL-TEST/chaquopy/server/pypi/dist/chaquopy-netcdf4/chaquopy_netcdf4-4.9.2-1-py3-none-android_21_arm64_v8a.whl"
               install "/Users/skirby/TEMPY/home/skirby/WHEEL-TEST/chaquopy/server/pypi/dist/netCDF4/netCDF4-1.6.3-cp38-cp38-macosx_11_0_arm64.whl"
            }
        }

Note - I got the python interface to netCDF4 wheel file at: [https://pypi.org/project/netCDF4/1.6.3/#files)]

[githubweather]

This is the meta.yaml for the netcdf4-python package. I do have cftime listed in it --

package:
  name: netcdf4-python
  version: {{ version }}

source:
  git_url: https://github.com/Unidata/netcdf4-python.git
  git_rev: v{{ version }}

build:
  number: 1

requirements:
  build:
    - cftime 1.6.2
  host:
    - numpy 1.17.4
    - h5py 2.10.0
    - hdf5 1.10.2
#hdf5 is new above

#    - chaquopy-netcdf 4.4.1.1
    - chaquopy-netcdf 4.9.2
    - python

I don't completely understand the. difference between the "build" requirements and the "host" requirements in the meta.yaml. I tried cftime under "build", then under "host", but it is still erroring out saying unable to build native code cftime.

[githubweather]

I tried one more thing. In the meta.yaml for the netcdf4-python package I moved 'cftime 1.6.2' to the 'host:' section and created dist/cftime/ and put 'cftime-1.6.2-cp38-cp38-macosx_11_0_arm64.whl' downloaded from https://pypi.org/project/cftime/#files there. Rebuilt the wheel file for chaquopy-netcdf4 package and copied the resultant wheel file to my Mac to re-attempt the build in Android Studio but it is still erroring out as in the screen grabs seen above. I feel like I am very close but clearly am missing something critical. Pls let me know where I am going wrong here. Thx.

[mhsmith]

it looks like netcdf4 (python interface) depends on cftime. Do I need to look for a cftime wheel file next? Or would that be something to build?

Yes, it looks like you'll have to build it. But there's probably no need to add it to the meta.yaml file.

If you need help with any more build errors, please post the text log on the right side, not the tree on the left side. See my comment above, and this screenshot of what it's supposed to look like.

[githubweather]

Thanks for that. I am still using the wheel file for cftime I found at: https://pypi.org/project/cftime/#files. Hopefully that is ok. And also I am using this wheel file for netcdf4-python https://files.pythonhosted.org/packages/99/0d/92ae1e4da528f742a349bc1b0ad7259982e376be90ea258d6dc01f8bb781/netCDF4-1.6.3-cp38-cp38-macosx_10_9_x86_64.whl, however, one caveat:: the error I kept seeing in the APK launch (the APK is building without error) is "No such module netCDF4._netCDF4". As such, I cloned this: https://github.com/Unidata/netcdf4-python.git and found netCDF4/_netCDF4.pyx in the tree. Then used Cython to create netCDF4/_netCDF4.c. Then used "gcc -c _netCDF4.c ... -o _netCDF4" to generate netCDF4/_netCDF4. Created a new wheel file via "python3 setup.py bdist_wheel", using the unzipped contents of:netCDF4-1.6.3-cp38-cp38-macosx_10_9_x86_64.whl and adding the netCDF4._netCDF4 module (built as described) to this wheel. Here are the contents of this new wheel file netCDF4-1.6.3-cp38-cp38-linux_x86_64.whl -- show.netCDF4.wheel.log (please note the existence of the module netCDF4._netCDF4)

Here is the "python" portion of the build.gradle file: show-build.gradle.log

The last attachment is the "logcat" when I launch the APK (sorry it is a bit large) -- show.logcat.log Please look for the 05/10 portion, in particular the error "no such module netCDF4._netCDF4. This is puzzling since the attachment above (show.netCDF4.wheel.log) shows that this module exists.

One last thing I need to add (and I do apologize) because long entries can be tiring to look at, but, since I absolutely could not get an x86_64-based emulator to launch in Android Studio (it always hangs), I have moved completely to Android SDK command-line tools for the emulator launching and building/launching of the APK.

Thx for any clues as I am a bit puzzled currently...

[mhsmith]

Thanks for that. I am still using the wheel file for cftime I found at: https://pypi.org/project/cftime/#files. Hopefully that is ok. And also I am using this wheel file for netcdf4-python https://files.pythonhosted.org/packages/99/0d/92ae1e4da528f742a349bc1b0ad7259982e376be90ea258d6dc01f8bb781/netCDF4-1.6.3-cp38-cp38-macosx_10_9_x86_64.whl

No, neither of those are OK, because those wheels are for Linux and macOS. The only wheels you can install with Chaquopy are either android wheels from our build-wheel tool, or pure-Python wheels tagged as none-any.

[githubweather]

ok - thx for that clarification. So I am now stepping back and trying to build an android cftime wheel file using your build-wheel tool. I have tried 2 different versions of meta.yaml (without success) as follows: meta.yaml.cftime.errors-with-no-such-file-_cftime.c.log

This meta.yaml is yielding this error upon execution of build-wheel.py -- show.no_cftime.c.log

This tells me Cython needs to be introduced so I tried to in meta.yaml as follows-- meta.yaml.cftime.no-such-cython-wheel-file.log

However, this Cython wheel file is not found. Please fill me in as to which Cython wheel to reference or do I have the meta.yaml wrong? This version of the meta.yaml file yields this via build-wheel.py-- show.no-cython-wheel-file.log

This is the build.sh file used in both cases-- build.sh.cftime.log

In both cases this is how I invoked build-wheel.py

build-wheel.py --python 3.8 --abi x86_64 chaquopy-cftime

Thx for your patience as I try to get this right...

[vmitro]

Try this:

(build-wheel) vmitro@v3629:~/projects/chaquopy/server/pypi/packages/cftime$ cat meta.yaml
package:
  name: cftime
  version: 1.6.2

build:
  number: 0

source:
  git_url: https://github.com/Unidata/cftime.git
  git_rev: v1.6.2rel

requirements:
  build:
    - Cython 0.29.32
  host:
    - python
    - numpy 1.24.2

and

(build-wheel) vmitro@v3629:~/projects/chaquopy/server/pypi/packages/cftime$ cat patches/chaquopy.patch
diff -ur src-orig/setup.py src/setup.py
--- src-orig/setup.py   2023-05-05 17:12:39.915982717 +0000
+++ src/setup.py        2023-05-05 17:12:53.956387448 +0000
@@ -2,6 +2,10 @@

 import os
 import sys
+# Chaquopy
+import builtins
+sys.path.insert(0, os.path.abspath("../requirements"))  # For numpy.distutils
+builtins.__NUMPY_SETUP__ = True  # Prevent the rest of NumPy from being imported so that this setup.py can work with what it only needs
 import numpy

 from setuptools import Command, Extension, setup

Note: I'm on my phone so formating bzw. indentation could be wring, but IIRC it worked (although I also have a newer numpy version built, please correct the version number in your meta.yaml). You need to create two folders and two text files like it says above.

Edit: updated directory name to suite the Chaquopy naming cnvention.

[mhsmith]

@vmitro: Thanks, that all looks reasonable.

@githubweather: You're using too many chaquopy prefixes in the package names in meta.yaml. The rules are very simple:

• If you're building a Python package, use exactly the same name as its entry on PyPI, including capitalization. Otherwise, other packages that depend on it won't be able to find it.
• If you're building a non-Python library, prefix its name with chaquopy, to prevent clashes with any other packages that may be on PyPI.

The pypi/packages subdirectory name should be identical, except that it should be normalized if necessary according to PEP 503. This is necessary to allow the pkgtest test harness to work.

[githubweather]

Thx @vmitro and @mhsmith. I have updated my meta.yaml file, including the numpy version, as you mentioned. Also created the chaquopy.patch file as recommended. My current issue is with the build.sh file. This is the build.sh file --

build.sh.cftime.may172023.log

For reference here is the chaquopy.patch file, as in the info you provided --

chaquopy-patch.may172023.log

Here is the result of running build-wheel.py -- show.build-wheel-cftime.may172023.log

Lastly, when I clone cftime.git, create the configure file (as done in build.sh), and run configure, this is the issue-- error-when-running-configure-may172023.log

[vmitro]

@githubweather: I don't think you need a build.sh script at all. Try running the build-wheel.py script without it (e.g. mv build.sh build.sh.old).

[githubweather]

Tried that. But still get the errors upon running build-wheel.py ---

build-wheel.py: mkdir -p /home/skirby/WHEEL-TEST/chaquopy/server/pypi/packages/chaquopy-cftime/build/1.6.2/cp38-cp38-android_21_x86_64/requirements/chaquopy/lib
build-wheel.py: unzip -q -d /home/skirby/WHEEL-TEST/chaquopy/server/pypi/packages/chaquopy-cftime/build/1.6.2/cp38-cp38-android_21_x86_64/requirements/chaquopy /home/skirby/WHEEL-TEST/chaquopy/maven/com/chaquo/python/target/3.8.16-0/target-3.8.16-0-x86_64.zip include/* jniLibs/*
caution: filename not matched:  include/*
caution: filename not matched:  jniLibs/*
build-wheel.py: Error: Command returned exit status 11

[vmitro]

Strange, it should not fail so early in the build process. What's the content of the directory for Python target?

(build-wheel) vmitro@v3629:~/projects/chaquopy$ ls maven/com/chaquo/python/target/3.10.6-1/
target-3.10.6-1-arm64-v8a.zip    target-3.10.6-1-x86.zip
target-3.10.6-1-armeabi-v7a.zip  target-3.10.6-1-x86_64.zip
target-3.10.6-1-stdlib-pyc.zip

Adjust when needed, you target Python 3.8 IIRC.

Edit: It would maybe beneficial if you didn't deviate from the standard peocedure; could you try from scratch by reseting the repo, only copying the Python 3.8 target and adding the cftime directory to packages?

[githubweather]

Yes - not sure what is happening. I created a new cloned tree of chaquopy (good idea btw). And the same thing is happening. My chaquopy-netcdf4 package still builds perfectly (wheel file generated just fine). And the chaquopy-cftime tree is dying on the build-wheel.py call.

Here is the meta.yaml file for chaquopy-cftime --

{% set version = "1.6.2" %}

package:
  name: cftime
  version: {{ version }}

build:
  number: 1

source:
  git_url: https://github.com/Unidata/cftime.git
  git_rev: v{{ version }}

requirements:
  build:
    - Cython 0.29.34
  host:
    - python
    - numpy 1.19.5

Here is the tail end of the build-wheel.py call --

build-wheel.py: unzip -q -d /home/skirby/CHAQUOPY-RETRY/chaquopy/server/pypi/packages/chaquopy-cftime/build/1.6.2/cp38-cp38-android_21_x86_64/requirements/chaquopy /home/skirby/CHAQUOPY-RETRY/chaquopy/maven/com/chaquo/python/target/3.8.16-0/target-3.8.16-0-x86_64.zip include/* jniLibs/*
caution: filename not matched:  include/*
caution: filename not matched:  jniLibs/*
build-wheel.py: Error: Command returned exit status 11

Here is the maven directory --

ls -l maven/com/chaquo/python/target/3.8.16-0
total 5572
-rwxr-x--- 1 skirby skirby 5701885 May 18 12:27 target-3.8.16-0-x86_64.zip

and the patch file --

--- src-orig/setup.py   2023-05-16 17:26:57.704081052 -0600
+++ src/setup.py        2023-05-16 17:29:08.581181155 -0600
@@ -2,6 +2,10 @@

 import os
 import sys
+#Chaquopy
+import builtins
+sys.path.insert(0,os.path.abspath("../requirements"))  # For numpy.distutils
+builtins.__NUMPY_SETUP__ = True # Prevent the rest of NumPY from being imported so that this setup.py can work with what it only needs
 import numpy

 from setuptools import Command, Extension, setup

[vmitro]

Here is the maven directory

ls -l maven/com/chaquo/python/target/3.8.16-0
total 5572
-rwxr-x--- 1 skirby skirby 5701885 May 18 12:27 target-3.8.16-0-x86_64.zip

Have you built the zip yourself or have you downloaded it from the Maven repo? Because the sizes don't match. The one in the repo is 5571890 bytes long and yours is 5701885.

[githubweather]

@vmitro - excellent catch! Indeed the maven file size mismatch was the key. The issue was I had been downloading the maven file to my Mac which would unzip it automatically. Thus, I then had to re-compress to a zip file so apparently it was getting mangled. A straight download to my VM fixed things. Also you were right about not needing a build.sh file, although it is somewhat unclear to me as to why one is not needed? I still have a bit of testing to go, i.e., including this cftime wheel file in my package build which includes netCDF4, but this is a big step forward. Thx.

[vmitro]

@githubweather I've got amazing powers of observation, haha.

Now the reason why it doesn't need a separate build script is because the package's setup.py has all the info how to compile and package it. I've got a bunch of text explaining this and similar stuff, check out my chaquopy-experimental repo, more precisely, the .md files in ./server/pypi.

[githubweather]

Indeed - thx again for catching that maven issue! Makes good sense regarding why a build.sh is not needed.

I 'think' I'm on the last part of this -- now trying to build the netcdf4-python wheel file.

When I run build-wheel.py I am seeing this:

build-wheel.py: Error: /home/me/WHEEL-TEST/chaquopy/server/pypi/packages/chaquopy-netcdf4-python/build/1.6.3/cp38-cp38-android_21_x86_64/fix_wheel/netCDF4/_netCDF4.so is linked against unknown library 'libm.so.6'.

When I do ldd on _netCDF4.so, I get, in part:

linux-vdso.so.1 (0x00007ffec2571000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9e4e4e7000)

Indeed when this _netCDF4.so is. built this is the call:

/home/skirby/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/x86_64-linux-android21-clang -shared -L/home/skirby/WHEEL-TEST/chaquopy/server/pypi/packages/chaquopy-netcdf4-python/build/1.6.3/cp38-cp38-android_21_x86_64/requirements/chaquopy/lib -Wl,--exclude-libs,libgcc.a -Wl,--exclude-libs,libgcc_real.a -Wl,--exclude-libs,libunwind.a -Wl,--build-id=sha1 -Wl,--no-rosegment -lm -Wl,--no-undefined -lpython3.8 -I/home/skirby/WHEEL-TEST/chaquopy/server/pypi/packages/chaquopy-netcdf4-python/build/1.6.3/cp38-cp38-android_21_x86_64/requirements/chaquopy/include -idirafter /home/skirby/WHEEL-TEST/chaquopy/server/pypi/packages/chaquopy-netcdf4-python/build/1.6.3/cp38-cp38-android_21_x86_64/requirements/chaquopy/include/python3.8 build/temp.linux_x86_64-cpython-38/src/netCDF4/_netCDF4.o -L/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu/hdf5/serial -Wl,--enable-new-dtags,-R/usr/lib/x86_64-linux-gnu -Wl,--enable-new-dtags,-R/usr/lib/x86_64-linux-gnu/hdf5/serial -lnetcdf -lhdf5_hl -lhdf5 -lpthread -lsz -lz -ldl -lm -lcurl -o build/lib.linux_x86_64-cpython-38/netCDF4/_netCDF4.cpython-38-x86_64-linux-gnu.so

(Note: later on in the build-wheel.py run, _netCDF4.cpython-38-x86_64-linux-gnu.so is renamed to _netCDF4.so)

So that the problem is that -L/usr/lib/x86_64-linux-gnu is in the call above, however, the ldd on _netCDF4.so, shows that the libm.so.6 is actually in /lib/x86_64-linux-gnu. I somehow need to have build-wheel.py add -L/lib/x86_64-linux-gnu so that it can find libm.so.6. What's the best way to do. this? Thx.

[mhsmith]

The problem here is the /usr/lib references on the clang command line. This causes Linux libraries to be linked into the Android build, which will never work.

Most likely, there's something in a build script (such as setup.py) which detects that it's building on Linux, and adds these arguments to the command unconditionally. You'll need to edit the build script to remove these arguments. Since build-wheel already adds all the necessary -L arguments to LDFLAGS, you probably won't need to replace them with anything.

For how to go about doing making these edits, see "If any changes are needed to make the build work" in the README.

I also notice that you haven't renamed your package to follow this comment. Please do that, or you will have even more problems.