SWxTREC / pymsis

Python interface to the NRLMSIS codes
MIT License
23 stars 5 forks source link

Handling builds with meson #21

Closed greglucas closed 2 years ago

greglucas commented 2 years ago

This adds the ability to build with meson. Everything works locally for me on mac and Linux, but of course I don't have a Windows box and that is always the one causing failures... :(

  1. can't parse readme.rst due to bad characters... Avoid that for now: 8d970051403abed2637ac533143e30f8572a4684
  2. It seems like everything is built, but then running pytest fails because the libraries can't be found.

https://github.com/SWxTREC/pymsis/runs/8281587728?check_suite_focus=true#step:8:21

ImportError while importing test module 'D:\a\pymsis\pymsis\tests\test_msis.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests\test_msis.py:5: in <module>
    from pymsis import msis
C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pymsis\msis.py:5: in <module>
    from pymsis import msis00f, msis20f, msis[21](https://github.com/SWxTREC/pymsis/runs/8281587728?check_suite_focus=true#step:8:22)f
E   ImportError: DLL load failed while importing msis00f: The specified module could not be found.

The install appears to build the wheel successfully... https://github.com/SWxTREC/pymsis/runs/8281587728?check_suite_focus=true#step:6:1543

WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: D:\a\pymsis\pymsis\.mesonpy-7o1ars04\install\hostedtoolcache\windows\Python\3.9.13\x64\Lib\site-packages\pymsis\msis21f.cp39-win_amd64.dll.a (C:\hostedtoolcache\windows\Python\3.9.13\x64\Lib\site-packages\pymsis\msis21f.cp39-win_amd64.dll.a)
  Copying files to wheel...
  [0/14] pymsis/msis00f.cp39-win_amd64.pyd
  [1/14] pymsis/msis00f.cp39-win_amd64.dll.a
  [2/14] pymsis/msis20f.cp39-win_amd64.pyd
  [3/14] pymsis/msis20f.cp39-win_amd64.dll.a
  [4/14] pymsis/msis21f.cp39-win_amd64.pyd
  [5/14] pymsis/msis21f.cp39-win_amd64.dll.a
  [6/14] pymsis/__init__.py
  [7/14] pymsis/msis.py
  [8/14] pymsis/msis2.0.parm
  [9/14] pymsis/msis21.parm
  [10/14] pymsis/tests/__init__.py
  [11/14] pymsis/tests/test_msis.py
  [12/14] pymsis/tests/test_regression.py
  [13/14] pymsis/tests/msis2.0_test_ref_dp.txt
  [14/14] pymsis/tests/msis2.1_test_ref_dp.txt
  Preparing metadata (pyproject.toml): finished with status 'done'

closes #19

greglucas commented 2 years ago

@eli-schwartz I decided to try and learn how to use meson and gave it a go. I think I'm close to getting this to build, but wondering if you have any experience with Python Windows builds and if I'm doing something wrong here. Any other advice is welcome too. (taking advantage of your previous comment ;) https://github.com/SWxTREC/pymsis/pull/16#issuecomment-1220956807 )

eli-schwartz commented 2 years ago

I'm not a Windows user myself, but I think the error message emitted by python here is saying that there are missing DLL dependencies of the module. Unfortunately, the error message doesn't say what those dependencies are.

Is it possible to use something like dumpbin or Dependency Walker on the .pyd and see what it thinks it depends on?

rgommers commented 2 years ago

I had a closer look. The "Could not tell if file was meant for purelib or platlib" warnings are gone now, and the .pyd files end up in the correct location. I'm fairly sure that the missing DLL will be libgfortran, because that is needed for extensions from Fortran code and it's not clear to me how it will be found (GITHUB_PATH may not add it to the Windows library search path - although hard to be sure about that).

To build redistributable wheels, you need delvewheel to vendor the missing DLLs. See cibw_before_build_win.sh and repair_windows.sh at https://github.com/scipy/scipy/tree/main/tools/wheels.

I'll also point out that SciPy uses a different Mingw-w64 build, the rtools one: https://github.com/scipy/scipy/blob/main/.github/workflows/windows.yml#L42. That uses the Windows UCRT rather than the older runtime libraries, maybe that matters too.

greglucas commented 2 years ago

I must have overlooked that Scipy builds the libraries statically on Windows! https://github.com/scipy/scipy/blob/a082eb415192f3075921fbf3e539f2b82a53811e/scipy/meson.build#L5-L7 That is the key here and I'm able to get the Windows build working now: https://github.com/SWxTREC/pymsis/actions/runs/3266720215 Specifically the latest commit: https://github.com/SWxTREC/pymsis/pull/21/commits/e2f0c0daef8c450985eca6272a938bcb8a8adbbb

Thank you very much for taking a look @eli-schwartz and @rgommers, your comments definitely pointed me in the right direction here! It might be good to put this into the f2py / scipy -> meson migration notes. I'll try and make some PRs for docs updates if I can think of things that would help others in the next week or so.

rgommers commented 2 years ago

Nice, glad that got resolved!

eli-schwartz commented 2 years ago

Aha, excellent timing on my part. :D

greglucas commented 2 years ago

No problem at all, @eli-schwartz! Thanks so much for commenting, I'll add that in another commit before I make a release. I've got a few more things to do before releasing a new version. The meson part of this was all pretty nice to work with, so thanks for your work on that!