search

ajolma / Geo-GDAL-FFI

Perl FFI to GDAL using FFI::Platypus
Other
4 stars 4 forks source link

Tests fail on FreeBSD as NITF is not found #51

Closed shawnlaffan closed 1 year ago

shawnlaffan commented 1 year ago

It seems that the FreeBSD port does not include the NITF driver. This is causing failures under Cirrus CI for downstream dependencies.

From https://cirrus-ci.com/task/6207322767228928?logs=install#L10610 :

Driver 'NITF' not found. at /.cpanm/work/1682383834.41121/Geo-GDAL-FFI-0.09/blib/lib/Geo/GDAL/FFI.pm line 1483.
    Geo::GDAL::FFI::GetDriver("NITF") called at t/00.t line 141
# Tests were run but no plan was declared and done_testing() was not seen.
# Looks like your test exited with 2 just after 12.

The test should check if NITF is available. This is probably most easily done by using an eval{} in the if block for those tests. https://github.com/ajolma/Geo-GDAL-FFI/blob/287215b902504234ac297dd55cfa2540f5ccaf15/t/00.t#L143-160

shawnlaffan commented 1 year ago

Or we could grep the list returned from GetDrivers to see if it includes NITF.

ajolma commented 1 year ago

Skip the metadata test if NITF driver is not found.

shawnlaffan commented 1 year ago

@ajolma - any chance of a new version on CPAN? I'm getting CI failures on Cirrus while this is not fixed.

shawnlaffan commented 1 year ago

Building from github results in new drivers missing.

This looks like an issue with the freshports build, and that the CI system is not picking up the newer port in which it should be fixed. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269708

ajolma commented 1 year ago

I'll update the FFI to gdal 3.7, so it will take some time to familiarize myself again how some things are done.

ajolma commented 1 year ago

https://github.com/ajolma/Geo-GDAL-FFI/actions/runs/5302840083/jobs/9598058927 @shawnlaffan do you see what's the problem here?

shawnlaffan commented 1 year ago

It's usually an issue with the rpaths in libgdal.so. The log indicates the system cannot find libgdal or libgeos_c, but those might be false negatives.

Can you uncomment the ldd debug line? If any of the contents are empty then it is probably a build issue with Alien::gdal or one of the dependencies.

https://github.com/ajolma/Geo-GDAL-FFI/blob/1ff3c70ea637a3e8117a66b962764e884a4dc332/.github/workflows/linux_share_build.yml#L90

shawnlaffan commented 1 year ago

It can also be useful to add some feedback about alien installs in an early test. I do this in Biodiverse as part of a more general feedback process.

https://github.com/shawnlaffan/biodiverse/blob/161e37bee5453f7f4ec4414f06b768c77f25ca21/t/00-load.t#L64-L85

ajolma commented 1 year ago

Seems to me the cache may be missing the .so but why, I don't know. https://github.com/ajolma/Geo-GDAL-FFI/actions/workflows/linux_share_build.yml

shawnlaffan commented 1 year ago

It might be under a lib64 dir instead of lib. If so then that's likely related to the issue.

ldd `perl -MAlien::gdal -E'print q{}, Alien::gdal->dist_dir, q{/lib64/libgdal.so}'`
shawnlaffan commented 1 year ago

It might also be one of the other aliens has its dynamic libs under a lib64 dir.

I have code to work around the issue but it is not yet in a release (or on the master branch).
https://github.com/shawnlaffan/perl-alien-gdal/commit/9a2db9fe7d6ce118e29a4fab021ca33640b4efe1

shawnlaffan commented 1 year ago

It might also be one of the other aliens has its dynamic libs under a lib64 dir.

I have code to work around the issue but it is not yet in a release (or on the master branch).
https://github.com/shawnlaffan/perl-alien-gdal/commit/9a2db9fe7d6ce118e29a4fab021ca33640b4efe1

ajolma commented 1 year ago

Now the package reports dynamic libs if attching to them fails. See for example https://github.com/ajolma/Geo-GDAL-FFI/actions/runs/5333035822/jobs/9662989026#step:9:169 Installing deps: Alien::gdal is up to date. (1.36) In Build/tests: Error: Can't attach to GDAL methods. Problem with GDAL dynamic libs: '/Users/runner/perl5/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-gdal/lib/libgdal.dylib /Users/runner/perl5/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-gdal/lib/libgdal.33.dylib /Users/runner/perl5/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-gdal/lib/libgdal.33.3.7.0.dylib /Users/runner/perl5/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-gdal/share/java/libgdalalljni.dylib /Users/runner/perl5/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-geos-af/lib/libgeos.dylib /Users/runner/perl5/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-geos-af/lib/libgeos.3.11.2.dylib /Users/runner/perl5/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-geos-af/lib/libgeos_c.dylib /Users/runner/perl5/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-geos-af/lib/libgeos_c.1.dylib /Users/runner/perl5/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-geos-af/lib/libgeos_c.1.17.2.dylib /Users/runner/perl5/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.dylib /Users/runner/perl5/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.25.dylib /Users/runner/perl5/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.25.9.2.1.dylib'? at t/00.t line 11.

shawnlaffan commented 1 year ago

That ldd result looks like it is missing many of the dependencies.

Running it on my local machine using Ubuntu on WSL2 I get:

ldd `perl -MAlien::gdal -E'print q{}, Alien::gdal->dist_dir, q{/lib/libgdal.so}'`
        linux-vdso.so.1 (0x00007ffe6698b000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f33ae726000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f33ae70a000)
        libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f33ae677000)
        libxml2.so.2 => /lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f33ae4bb000)
        libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f33ae1e5000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f33ae1bc000)
        libtiff.so.5 => /lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f33ae13b000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f33ae10d000)
        libjpeg.so.8 => /lib/x86_64-linux-gnu/libjpeg.so.8 (0x00007f33ae088000)
        libpng16.so.16 => /lib/x86_64-linux-gnu/libpng16.so.16 (0x00007f33ae04e000)
        libsqlite3.so.0 => /home/user/perl5/perlbrew/perls/perl-5.36.0/lib/site_perl/5.36.0/x86_64-linux/auto/share/dist/Alien-gdal/lib/../../Alien-sqlite/lib/libsqlite3.so.0 (0x00007f33adf1a000)
        libgeos_c.so.1 => /home/user/perl5/perlbrew/perls/perl-5.36.0/lib/site_perl/5.36.0/x86_64-linux/auto/share/dist/Alien-gdal/lib/../../Alien-geos-af/lib/libgeos_c.so.1 (0x00007f33adeca000)
        libproj.so.25 => /home/user/perl5/perlbrew/perls/perl-5.36.0/lib/site_perl/5.36.0/x86_64-linux/auto/share/dist/Alien-gdal/lib/../../Alien-proj/lib/libproj.so.25 (0x00007f33adac5000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f33adaa2000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f33ad8be000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f33ad76f000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f33ad754000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f33ad562000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f33afdfe000)
        libnghttp2.so.14 => /lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f33ad539000)
        libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f33ad518000)
        librtmp.so.1 => /lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f33ad4f6000)
        libssh.so.4 => /lib/x86_64-linux-gnu/libssh.so.4 (0x00007f33ad488000)
        libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f33ad475000)
        libssl.so.1.1 => /lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f33ad3e2000)
        libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f33ad395000)
        libldap_r-2.4.so.2 => /lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f33ad33f000)
        liblber-2.4.so.2 => /lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f33ad32c000)
        libbrotlidec.so.1 => /lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x00007f33ad31e000)
        libicuuc.so.66 => /lib/x86_64-linux-gnu/libicuuc.so.66 (0x00007f33ad138000)
        libwebp.so.6 => /lib/x86_64-linux-gnu/libwebp.so.6 (0x00007f33ad0ce000)
        libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f33ad025000)
        libjbig.so.0 => /lib/x86_64-linux-gnu/libjbig.so.0 (0x00007f33ad012000)
        libgeos.so.3.11.1 => /home/user/perl5/perlbrew/perls/perl-5.36.0/lib/site_perl/5.36.0/x86_64-linux/auto/share/dist/Alien-gdal/lib/../../Alien-geos-af/lib/libgeos.so.3.11.1 (0x00007f33acd5e000)
        libunistring.so.2 => /lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f33acbdc000)
        libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f33aca06000)
        libhogweed.so.5 => /lib/x86_64-linux-gnu/libhogweed.so.5 (0x00007f33ac9cf000)
        libnettle.so.7 => /lib/x86_64-linux-gnu/libnettle.so.7 (0x00007f33ac993000)
        libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f33ac90f000)
        libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f33ac832000)
        libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f33ac801000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f33ac7fa000)
        libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f33ac7eb000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f33ac7cd000)
        libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f33ac7b0000)
        libgssapi.so.3 => /lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f33ac76b000)
        libbrotlicommon.so.1 => /lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x00007f33ac748000)
        libicudata.so.66 => /lib/x86_64-linux-gnu/libicudata.so.66 (0x00007f33aac87000)
        libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f33aab4f000)
        libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f33aab39000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f33aab32000)
        libheimntlm.so.0 => /lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f33aab26000)
        libkrb5.so.26 => /lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f33aaa93000)
        libasn1.so.8 => /lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f33aa9eb000)
        libhcrypto.so.4 => /lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f33aa9b3000)
        libroken.so.18 => /lib/x86_64-linux-gnu/libroken.so.18 (0x00007f33aa99a000)
        libffi.so.7 => /lib/x86_64-linux-gnu/libffi.so.7 (0x00007f33aa98e000)
        libwind.so.0 => /lib/x86_64-linux-gnu/libwind.so.0 (0x00007f33aa964000)
        libheimbase.so.1 => /lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f33aa950000)
        libhx509.so.5 => /lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f33aa902000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f33aa8c7000)
shawnlaffan commented 1 year ago

Can you try changing the Alien::gdal install to use the issue_23_runtime_props branch? And make it verbose?

cpanm -v https://github.com/shawnlaffan/perl-alien-gdal.git@issue_23_runtime_props
shawnlaffan commented 1 year ago

FWIW, running with Alien::gdal and Geo::GDAL::FFI from cpan results in a successful build on my canary system.
https://github.com/shawnlaffan/geo-gdal-ffi-canary/actions/runs/5345035309

shawnlaffan commented 1 year ago

This was a false positive due to the FreeBSD build missing several features.