NOAA-GSL / VxIngest

Other
2 stars 0 forks source link

create wheel build for NCEPLIBS-bufr #358

Closed randytpierce closed 3 months ago

randytpierce commented 3 months ago

We need to create a wheel build for the NCEPLIBS-bufr so that our images builds do not re-compile the fortran code all the time.

randytpierce commented 3 months ago

Wow, nothing confusing about this.... (wrong)! I have added a new directory at the top level NCEPLIBS-bufr that contains a pyproject.toml for a poetry managed package. The NCEPLIBS-bufr has it's own cmake build environment that we have decided to build as an external dependency. The build steps are documented in the README.md in the NCEPLIBS-bufr directory. Once it is built and tested the essential library and python package files are to be copied into the NCEPLIBS-bufr ncepbufr directory. Those steps are also documented in the README. From there the build and install is handled by poetry and will be taken care of by the CI github actions.

randytpierce commented 3 months ago

After discussing this issue Ian and I have decided on the following approach to finalize this. 1) We will have a third_party top level directory that will contain an NCEPLIBSbufr wheel build poetry project. 2) That project will contain a build_wheel.sh script that will

  1. clone the NCEPLIBSbufr repo at a specified version into a temporary area (make sure it is gitignored)
  2. add or check for any required dependencies
  3. build the NCEPLIBSbufr stuff
  4. run the tests
  5. copy the pyproject stuff into the temporary area
  6. copy the required .so file and ncep python wrapper code into the temporary pyproject area
  7. build the wheel file for the running architecture - naming the wheel appropriately
  8. copy the wheel into the controlled dist_wheel area
  9. clean up the temporary area 3) the wheel_dist directory will be controlled and will contain all the versions and architectures we intend to support
randytpierce commented 3 months ago

Ok, I needed to make a new branch and a new PR. The new branch contains a third_party directory and that contains an NCEPLIBS-bufr directory which contains a pyproject.toml, a build.sh script, and a wheel_dist directory. The build script createsa tmp directory, downloads and installs the NCEPLIBS-bufr project, and then builds a python wheel to wrap up the NCEPLIBS-bufr stuff in a python installable. I've run it on a mac ARM laptop, and a linux ARM container, and a linux_x86_64 EC2 instance to create python wheels for those, and I intend to run it on a MAC x86_64. I am only waiting for review before merging the branch and closing the PR.

randytpierce commented 3 months ago

This is finished in the NCEPLIBS-bufr-petry-new feature branch. I had to implement a flag for building with local test data. Waiting on approval of PR.

randytpierce commented 3 months ago

merged and closed