search

nicogodet / telemac-mascaret-feedstock

https://anaconda.org/nicogodet/opentelemac
2 stars 2 forks source link

Python / Numpy compilation and cross compatibility #13

Open tomsail opened 10 months ago

tomsail commented 10 months ago

EDIT (after applying patch provided by @nicogodet, see details below):

Recapitulative table:

python \ numpy 1.19 1.20 1.21 1.22 1.23 1.24 1.25 1.26
3.8 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :x: :x:
3.9 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:
3.10 :x: :x: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:
3.11 :x: :x: :x: :x: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark:
3.12 :x: :x: :x: :x: :x: :x: :x: :x:

:x: Compiling fails :large_orange_diamond: API fails :heavy_check_mark: everything works

TODO:

  1. check the triangulation mtri.triangulation.Triangulation/mtri.Triangulation to apply depending on the numpy version

Note: for the API test, a simple test was done :

mamba create -n "$env_name" opentelemac python="$py" numpy="$np" -y
mamba activate "$env_name"
TELEMAC_DIR="<LOCAL_DIR_THAT_CONTAINS_EXAMPLES>"
example_dir="$TELEMAC_DIR/examples/python3/TelApy_api"
destination_dir="$CONDA_PREFIX/opentelemac/examples/telemac2d"
mkdir -p "$destination_dir"
cp -r "$TELEMAC_DIR/examples/telemac2d/gouttedo/" "$destination_dir/"
cd "$example_dir" || exit
python t2d.py

@nicogodet validate_telemac.py does not work because package does not contain the examples. I f you have a fix for this, I'm keen to hear.

nicogodet commented 10 months ago

What's the error message when it fails ?

Compiling main is PIA (sha256 will change multiple time a day), I would stick to released version with patch.

To generate a patch, you can

f2py.patch

tomsail commented 10 months ago

What's the error message when it fails ?

Mainly during one of the API steps, i.e. 


Building the Python API 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    ~> Wrap_api built
    ~> Compiling Modules api
    ~> First part of f2py for api passed

It can be during the wrap, the first or second step, it depends.. I haven't printed out the whole compilation process, just the outcome of it.

Compiling main is PIA (sha256 will change multiple time a day), I would stick to released version with patch.

To generate a patch, you can

  • git checkout v8p4 branch
  • create a new branch from v8p4 : v8p4-patch-f2py
  • cherry pick the merge commit
  • git diff v8p4..v8p4-patch-f2py > f2py.patch

f2py.patch

Thanks that's a great hack! I had seen in one of your meta.yaml that you could include the patch in the conda build process, like so correct?

source:
  url: <source-url>
  sha256: <source-sha256>
  patches:
    - f2py.patch

I will finish the first matrix build I started and add a second table with the patch fix

nicogodet commented 10 months ago

I opened https://github.com/conda-forge/staged-recipes/pull/24529 We need to manually specify python and numpy version in conda_build_config.yaml and add condition on patch depending numpy version.

nicogodet commented 10 months ago

With patch in PR, il was able to build on py3.11 and numpy 1.25 locally but it fails with same recipe on azure pipeline

tomsail commented 10 months ago

what is the error on azure? With the patch locally, doing the conda build, I have (with python 3.10, numpy 1.25) 

  - completed: .../opentelemac/sources/api/api_run_t3d.f                     
    - completed: .../opentelemac/sources/api/api_run_wac.f                     
    - completed: .../opentelemac/sources/api/api_interface.f                   
/home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/sources/api/example_cpl_t2d_sis.f:146:30:

  146 |      &                      0,0,IERR)
      |                              1
Error: Non-variable expression in variable definition context (actual argument to INTENT = OUT/INOUT) at (1)
/home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/sources/api/example_cpl_t2d_sis.f:146:72:

  146 |      &                      0,0,IERR)
      |                                                                        1
Error: More actual than formal arguments in procedure call at (1)
/home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/sources/api/example_cpl_t2d_sis.f:164:30:

  164 |      &                      0,0,IERR)
      |                              1
Error: Non-variable expression in variable definition context (actual argument to INTENT = OUT/INOUT) at (1)
/home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/sources/api/example_cpl_t2d_sis.f:164:72:

  164 |      &                      0,0,IERR)
      |                                                                        1
Error: More actual than formal arguments in procedure call at (1)
    - completed: .../opentelemac/sources/api/homere_api.F                      
Traceback (most recent call last):\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ ]  99%  | 1s
  File "/home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/scripts/python3/compile_telemac.py", line 142, in <module>
    main()
  File "/home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/scripts/python3/compile_telemac.py", line 129, in main
    compile_cmdf(options.ncsize, modules, options.verbose)
  File "/home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/scripts/python3/compilation/compil_tools.py", line 1298, in compile_cmdf
    raise TelemacException(\
utils.exceptions.TelemacException: 
        +> failed: .../opentelemac/sources/api/example_cpl_t2d_sis.f
... The following command failed for the reason in the listing
mpif90 -c -O2 -fPIC -cpp -fconvert=big-endian -frecord-marker=4 -DHAVE_I16 -DHAVE_MPI -DHAVE_MUMPS -DHAVE_GOTM -DHAVE_AED2  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/utils/special  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/utils/damocles  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/utils/parallel  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/utils/hermes  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/utils/bief  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/nestor  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/utils/ad  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/tomawac  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/sisyphe  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/waqtel  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/khione  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/gaia  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/telemac2d  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/utils/gretel  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/utils/partel  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/telemac3d  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/artemis  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/api    /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/sources/api/example_cpl_t2d_sis.f
nicogodet commented 10 months ago

Retry with patch from https://github.com/conda-forge/staged-recipes/blob/65a32bc7fdb6112b6c841249e5eed27642f212cd/recipes/opentelemac/patches/f2py.patch

tomsail commented 10 months ago

I have this error, would you know why?

Using : 

{% set name = "opentelemac" %}
{% set version = "v8p4r1" %}
{% set build = 0 %}

package:
  name: {{ name|lower }}
  version: {{ version }}

source:
  # - url: https://gitlab.pam-retd.fr/otm/telemac-mascaret/-/archive/{{ version }}/telemac-mascaret-{{ version }}.tar.gz
  - url: https://github.com/nicogodet/otm-sources/releases/download/{{ version }}/telemac-mascaret-{{ version }}.tar.gz
    folder: opentelemac
    # sha256: b48a0ee5cb43a68bc7255357544810ea67ea918a5ffd7fd61eeed41be4cf7b71
    sha256: 4dc0fdf59c7befff277b51ef8bec1eaac9b4b12db071b8914739ea2156532ee5
    patches:
      - patches/f2py.patch
  - path: ../src


INFO:conda_build.source:Found source in cache: telemac-mascaret-v8p4r1_b48a0ee5cb.tar.gz
Extracting download
Applying patch: /home/tomsail/work/python/feedstocks/telemac-mascaret-feedstock/recipe/f2py.patch
Applying patch: /home/tomsail/work/python/feedstocks/telemac-mascaret-feedstock/recipe/f2py.patch with args:
['-Np1', '-i', '/tmp/tmp7sw0rlpr/f2py.patch.native', '--binary']
checking file scripts/python3/compilation/compil_tools.py
checking file scripts/python3/execution/get.py
checking file scripts/python3/telapy/api/api_module.py
checking file scripts/python3/telapy/api/hermes.py
checking file sources/api/api.cmdf
checking file sources/api/api_interface.f
checking file sources/api/example_cpl_t2d_sis.f
patch unexpectedly ends in middle of line
/home/tomsail/miniconda3/bin/patch: **** malformed patch at line 769:  

Warning: failed to download source.  If building, will try again after downloading recipe dependencies.
Error was: 
Command '['/home/tomsail/miniconda3/bin/patch', '--no-backup-if-mismatch', '--batch', '-Np1', '-i', '/tmp/tmp7sw0rlpr/f2py.patch.native', '--binary', '--dry-run']' returned non-zero exit status 2.
Failed to download or patch source. Please see build log for info.```
nicogodet commented 10 months ago

Check you have an empty line at end of patch file

tomsail commented 10 months ago

It worked! I'll matrix build overnight and update the table tomorrow. I'll also test all versions with validate_telemac.py or some basic tests so to see how the compilation and API performed.

This will give us insight for the constraints to put on numpy on conda-forge. Bravo @nicogodet, great stuff

tomsail commented 10 months ago

@sebourban @nicogodet please have a look at the recap table above (first post) basic API runs fine, even 1.22/1.23 version ! (see test run above)

full linux versions (MPI/MUMPS/API/AED/GOTM) are available on my anaconda channel for the python x numpy versions in the table above.

nicogodet commented 10 months ago

@nicogodet validate_telemac.py does not work because package does not contain the examples. I f you have a fix for this, I'm keen to hear.

You can add a test folder inside you recipe dir with examples and refere to this folder during test phase https://github.com/nicogodet/telemac-mascaret-feedstock/commit/8f744a1894a0e1981cb428cf39853e48f64ce3b5#diff-f3725a55bf339595bf865fec73bda8ac99f283b0810c205442021f29c06eea9a