search

NeoGeographyToolkit / StereoPipeline

The NASA Ames Stereo Pipeline is a suite of automated geodesy & stereogrammetry tools designed for processing planetary imagery captured from orbiting and landed robotic explorers on other planets.
Apache License 2.0
478 stars 168 forks source link

Problem finding GDAL's supporting data #347

Closed oleg-alexandrov closed 2 years ago

oleg-alexandrov commented 2 years ago

The shipped ASP build has a problem. The command:

/path/to/ASP/bin/point2las --t_srs 'EPSG:4326' run/run-PC.tif

fails with the error:

Error: GdalIO: Unable to open EPSG support file gcs.csv. Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files. (code = 4)

Yet, gcs.csv is there where it should be, in $GDAL_DATA, which is set as /path/to/ASP/share/gdal.

I tracked this down. The strange thing is that somehow, under the hood, libgdal from the shipped ASP reaches back into the original miniconda directory where it used to live before being packaged and shipped, and peeks into the directory share/epsg_csv in there.

Copying that directory so that it exists in the "share" directory of the shipped build fails to fix the problem. It really insists the original directory share/epsg_csv must exist. If I rename that one, it does not work.

To put it in different words, a packaged build must be self-contained. It must not remember where exactly it was built and with what libraries it was built. It must find all that it needs in the package itself. Yet, somehow libgdal remembers to search for its supporting data not in the package, but back before it was packaged.

I made sure the environment is clean, that there are no hidden paths burned into the libgdal executable, but that does not help.

rbeyer commented 2 years ago

Yikes, I wonder if this is really a GDAL Issue?

oleg-alexandrov commented 2 years ago

Hard to day. It surely tries to peek in a directory named "share/epsg_csv", which is not advertised by the error message, which complains about "share/gdal". So some hidden lookup exists.

But I did not build gdal in that local directory it tries to peek into, I only installed it there with conda. So, conda likely did the packaging right, but when we package ASP in a tarball we potentially miss some important details, but I don't know what those should be.

The binaries are clean, ldd shows the right thing, but there could be some magic text somewhere gdal counts on which was not translated well during us shipping things.

dshean commented 2 years ago

I experienced this same issue in September. Checked the PATH used by ASP wrapper scripts was pointing to the gdal exectuables shipped by ASP and GDAL_DATA was set correctly, no luck. Using full PROJ string was the workaround.

This ties in with the larger issue of supporting updated GDAL3/PROJ8 in ASP and ISIS - there will inevitably be more issues as GDAL/PROJ in user environment (presumably installed via conda) diverges further from the outdated ASP GDAL/PROJ. It will be nice to avoid shipping a separate GDAL build with ASP. But I realize this will require some major work, and ISIS dependency is a limitation.

The gcs.csv is no longer used by GDAL >2.5: https://trac.osgeo.org/gdal/wiki/rfc73_proj6_wkt2_srsbarn. I believe ASP still ships GDAL 2.4.1.

oleg-alexandrov commented 2 years ago

I found the problem, and it was a bug by me. My bad. I was not setting the path to GDAL data correctly.

ASP has three different paths to set (ISIS, Qt plugins, and GDAL), and when doing all that I misread the documentation for this one. This was not caught because somehow GDAL was falling back to a good location, which exists on our test machines but not on user machines.

This will be fixed in tomorrow's build.