Move install location for python packages to "lib/pythonX.Y/site-packages"

[srherbener]

The purpose of moving the install location is to be compatible with spack-stack. The issue comes up with the setting of PYTHONPATH in the lua module file for bufr-query. spack-stack insists on inserting the site-packages directory into the install directory despite the fact that the cmake config wasn't placing it there.

I couldn't figure out a simple way to tell spack-stack (or spack) to not insert the site-packages directory and ended up using a spack patch mechanism which applies the exact same changes as in this PR.

Using "lib/pythonX.Y/site-packages" appears to be a standard location for any custom packages (in regard to what is shipped with the OS) built on a platform, and in the context of the OS, the spack-stack built packages seem to me to qualify as "custom".

[srherbener]

@rmclaren, I can't seem to assign reviewers. I was thinking of you, @emilyhcliu and @CoryMartin-NOAA, but I'll leave it to your discretion. Thanks!

[rmclaren]

@srherbener The updated CMakeLists.txt file does not make a usable python binding when doing a regular build as it does not copy in all the relevant python files (__init__.py files and encoders directory with __init__.py files) into the build/lib/python3.12/site-packages/bufr directory. Compare install path with build path.

[srherbener]

@srherbener The updated CMakeLists.txt file does not make a usable python binding when doing a regular build as it does not copy in all the relevant python files (__init__.py files and encoders directory with __init__.py files) into the build/lib/python3.12/site-packages/bufr directory. Compare install path with build path.

Good catch @rmclaren! These exact same changes work in spack-stack but when I try building bufr-query standalone, I get the broken result you reported. Sorry about that. I'll get this fixed, and thanks for checking this.

[srherbener]

I noticed that there are no ctests that check the correctness of the python install in the build area - which would have caught this mistake. I can add a ctest that runs python and simply tries to import the bufr package. Does this sound good?

[srherbener]

I noticed that there are no ctests that check the correctness of the python install in the build area - which would have caught this mistake. I can add a ctest that runs python and simply tries to import the bufr package. Does this sound good?

I take this back - there are a lot of ctests but they didn't run when I tried running ctest after the build. What am I doing wrong?

[srherbener]

I see the problem now. The ecbuild output contains this:

-- Downloading: /Users/stephenh/projects/BUFR_QUERY/build/test/bufr_query-0.0.1.tgz
-- [download 0% complete]
-- Could not download test files, not building tests

I'll keep debugging.

[srherbener]

The reason these changes worked in spack-stack is because spack-stack does not run ctests by default, and only the install area matters (which is working).

[srherbener]

Looks like the issue is that this file is missing: https://ftp.emc.ncep.noaa.gov/static_files/public/bufr_query-0.0.1.tgz

Here's what I see when I go to https://ftp.emc.ncep.noaa.gov/static_files/public in Chrome:

<h1 style="color: rgb(0, 0, 0); font-family: Times; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Index of /static_files/public</h1>
Name | Last modified | Size
-- | -- | --

Parent Directory |   | -
GDASApp/ | 08-Mar-2024 21:00 | -
NCEPLIBS-g2/ | 21-Jun-2024 12:53 | -
UFS/ | 23-Feb-2024 18:13 | -
WW3/ | 23-Feb-2024 15:56 | -
bufr-11.6.0.tgz | 23-Feb-2024 14:20 | 86M
bufr-12.1.0.tgz | 02-May-2024 21:39 | 87M
bufr-12.1.0.tgz.1 | 22-Feb-2024 22:36 | 87M
g2-testdata/ | 26-Jun-2023 17:27 | -
gsi/ | 13-Jan-2023 15:11 | -
obsproc/ | 21-Oct-2021 18:50 | -

Index of /static_files/public
[Name](https://ftp.emc.ncep.noaa.gov/static_files/public/?C=N;O=D)  [Last modified](https://ftp.emc.ncep.noaa.gov/static_files/public/?C=M;O=A) [Size](https://ftp.emc.ncep.noaa.gov/static_files/public/?C=S;O=A)
[Parent Directory](https://ftp.emc.ncep.noaa.gov/static_files/)     -
[GDASApp/](https://ftp.emc.ncep.noaa.gov/static_files/public/GDASApp/)  08-Mar-2024 21:00   -
[NCEPLIBS-g2/](https://ftp.emc.ncep.noaa.gov/static_files/public/NCEPLIBS-g2/)  21-Jun-2024 12:53   -
[UFS/](https://ftp.emc.ncep.noaa.gov/static_files/public/UFS/)  23-Feb-2024 18:13   -
[WW3/](https://ftp.emc.ncep.noaa.gov/static_files/public/WW3/)  23-Feb-2024 15:56   -
[bufr-11.6.0.tgz](https://ftp.emc.ncep.noaa.gov/static_files/public/bufr-11.6.0.tgz)    23-Feb-2024 14:20   86M
[bufr-12.1.0.tgz](https://ftp.emc.ncep.noaa.gov/static_files/public/bufr-12.1.0.tgz)    02-May-2024 21:39   87M
[bufr-12.1.0.tgz.1](https://ftp.emc.ncep.noaa.gov/static_files/public/bufr-12.1.0.tgz.1)    22-Feb-2024 22:36   87M
[g2-testdata/](https://ftp.emc.ncep.noaa.gov/static_files/public/g2-testdata/)  26-Jun-2023 17:27   -
[gsi/](https://ftp.emc.ncep.noaa.gov/static_files/public/gsi/)  13-Jan-2023 15:11   -
[obsproc/](https://ftp.emc.ncep.noaa.gov/static_files/public/obsproc/)  21-Oct-2021 18:50   -

[rmclaren]

@srherbener Yes, that file is not available on the ftp server at the moment.

[srherbener]

I just pushed the fix for the build area issue. It appears to be working now - I tried a simple test which worked:

Stephens-MBP:build stephenh$ DYLD_LIBRARY_PATH=$PWD/lib PYTHONPATH=$PWD/lib/python3.10/site-packages python3
Python 3.10.13 (main, Jul 18 2024, 12:26:26) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bufr
>>> quit()
Stephens-MBP:build stephenh$ 

[rmclaren]

@srherbener thanks Steve!