NOAA-GSL / VxIngest

Other
2 stars 0 forks source link

fix pyproj error on python-slim-bookworm based container #251

Closed randytpierce closed 11 months ago

randytpierce commented 11 months ago

Although the integration and unit tests seem to pass on the python-slim-bookworm based container, the grib-builder-parent code is throwing an exception when using pyproj to perform transformations on the grid. There are two problems with this: first - what is this pyproj error? There is no error when running this code under macosx, but there is a problem when running it in a container under python-slim-bookworm regardless of where the container is running (of course). second - why is this failure not being caught by the grib2 integration test?

This is a critical problem to solve.

randytpierce commented 11 months ago

The reason that the tests didn't catch this was because I inadvertantly left out the grib_builder data checking integration test. ;(

randytpierce commented 11 months ago

I added a test specifically for pyproj (it uses couchbase as well) named grib_to_cb/test/test_unit_proj.py. python -m pytest grib2_to_cb/test/test_unit_proj.py

I also created two new tests which are the same as the above named test_unit_proj-cb.py and test_unit_proj_nocb.py and I am putting them into their own container. The only difference between the two is that the test_unit_proj_cb.py imports the couchbase cluster. I expect that one to fail and the one that does not import couchbase to pass. This gives me something succinct to give to the CB people.

randytpierce commented 11 months ago

I am trying to start the container as root (remove the user commands in Dockerfile and compose.yaml) and then find a configuration that works. Found that changing the order of the imports in test_unit_proj_cb.py changes the problems in an informative way. If pyproj is first i.e. import pyproj from couchbase.cluster import Cluster Then the couchbase import causes an abort. If the imports are reversed i.e. from couchbase.cluster import Cluster import pyproj then the couchbase import is fine but pyproj gets an std::bad_cast exception which is undoubtedly coming from eccodes. In an attempt to get further I uninstall couchbase with pip and then install apt-get install cmake apt-get install libssl-dev and then pip install couchbase --no-binary :all: To try to build couchbase from source, but it appears to hang on "Installing build dependencies ..."

randytpierce commented 11 months ago

Actually do ... apt-get install git python3-dev python3-pip python3-setuptools cmake build-essential libssl-dev then pip install couchbase --no-binary couchbase