search

conda-forge / gdal-feedstock

A conda-smithy repository for gdal.
BSD 3-Clause "New" or "Revised" License
30 stars 63 forks source link

add libgdal run requirement on libgrpc #806

Closed akrherz closed 1 year ago

akrherz commented 1 year ago

Checklist

refs conda-forge/rasterio-feedstock#274

It seems like we need libgrpc participating here as a run dependency as we get this via libgdal -> tiledb -> libgoogle-cloud -> libgrpc and need to maintain pinning.

conda-forge-webservices[bot] commented 1 year ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

akrherz commented 1 year ago

@conda-forge-admin please rerender

akrherz commented 1 year ago

Oye, now I wonder if tiledb needs this treatment first.

akrherz commented 1 year ago

This smells like not the right approach, as we would need to be pinning libgrpc in tiledb, libgdal, and rasterio for this to "work", but maybe that is how it is supposed to be?

xylar commented 1 year ago

I would check with @h-vetinari, who is pretty expert in these things. Also, see this comment: https://github.com/conda-forge/protobuf-feedstock/issues/198#issuecomment-1714603007

h-vetinari commented 1 year ago

Yeah, the issue is in tiledb, which has a variant that depends on google-cloud-cpp, which itself depends on abseil. It's also pretty clear in the log there (c.f. latest build on main):

WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libcurl.so.4 found in ['libcurl']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libcurl'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
   INFO (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libbz2.so.1.0 found in conda-forge::bzip2-1.0.8-h7f98852_4
   INFO (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/liblz4.so.1 found in conda-forge::lz4-c-1.9.3-h9c3ff4c_1
   INFO (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libz.so.1 found in conda-forge::libzlib-1.2.13-hd590300_5
   INFO (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libzstd.so.1 found in conda-forge::zstd-1.5.2-hfc55251_7
   INFO (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libssl.so.3 found in conda-forge::openssl-3.1.2-hd590300_0
   INFO (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libcrypto.so.3 found in conda-forge::openssl-3.1.2-hd590300_0
   INFO (tiledb,lib/libtiledb.so.2.16): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/libdl.so.2 found in CDT/compiler package conda-forge::sysroot_linux-64-2.12-he073ed8_16
   INFO (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libgoogle_cloud_cpp_storage.so.2 found in conda-forge::libgoogle-cloud-2.12.0-h840a212_1
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_cord.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_cordz_info.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_cord_internal.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_cordz_functions.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_exponential_biased.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_cordz_handle.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_synchronization.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_stacktrace.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_graphcycles_internal.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_symbolize.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_debugging_internal.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_malloc_internal.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_demangle_internal.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_crc_cord_state.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_crc32c.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_crc_internal.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_crc_cpu_detect.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_str_format_internal.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
   INFO (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libgoogle_cloud_cpp_rest_internal.so.2 found in conda-forge::libgoogle-cloud-2.12.0-h840a212_1
   INFO (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libgoogle_cloud_cpp_common.so.2 found in conda-forge::libgoogle-cloud-2.12.0-h840a212_1
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_time.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_strings.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_strings_internal.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_int128.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_civil_time.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_time_zone.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_bad_variant_access.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_base.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_spinlock_wait.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_bad_optional_access.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_throw_delegate.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_raw_logging_internal.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libabsl_log_severity.so.2301.0.0 found in ['libabseil']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libabseil'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (tiledb,lib/libtiledb.so.2.16): Needed DSO lib/libcrc32c.so.1 found in ['libcrc32c']
WARNING (tiledb,lib/libtiledb.so.2.16): .. but ['libcrc32c'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)

I would suggest to rebuild tiledb with an abseil-dependence, that should already fix most of the bleeding I think. Possible too to patch existing tiledb builds (with gcs support) to depend on libabseil 230125. Once the former has been added, the migrator should pass by to rebuild it for the new abseil. PS. not sure what's going on with the curl builds there, 7.x and 8.x are ABI-compatible.

CC @conda-forge/tiledb

ocefpaf commented 1 year ago

It's also pretty clear in the log there (c.f. latest build on main)

Anyone that parses those logs at the top of their head and says "pretty clear" definitely achieved a higher level of consciousness :smile:

ocefpaf commented 1 year ago

PS: if that affects a single build of tiled I prefer that we mark it as broken until we figure out a better approach.

h-vetinari commented 1 year ago

Anyone that parses those logs at the top of their head and says "pretty clear" definitely achieved a higher level of consciousness

😜

My trick is opening the raw log (because the Azure Pipelines Search is broken except for windows jobs), and then search for the string "WARNING (", which pretty much only ever shows up in the conda link check.

PS: if that affects a single build of tiled I prefer that we mark it as broken until we figure out a better approach.

It probably affects more tiledb builds (depending on when the gcp variant was added), just that the breakage didn't show up until a new abseil version was released that surfaced the implicit dependence.

akrherz commented 1 year ago

Thanks @h-vetinari , does this mean that libgdal needs the same dependency too or just fixing it for tiledb will fix it for "upstream" packages of it?

h-vetinari commented 1 year ago

does this mean that libgdal needs the same dependency too or just fixing it for tiledb will fix it for "upstream" packages of it?

I think you mean downstream (i.e. packages depending on tiledb)? In any case, that'll depend whether tiledb uses abseil types in its own external C-API or whether they just use abseil internally. In any case rebuilding tiledb is the first step, and then the link check here should give an indication if anything from gdal also depends on abseil.

akrherz commented 1 year ago

In any case rebuilding tiledb is the first step

thanks again, conda-forge/tiledb-feedstock#209 is in. The reproducer is trivial if you want to see it in the wild

$ mamba create -n repro python=3.11 rasterio
$ python -c 'import rasterio'
h-vetinari commented 1 year ago

thanks again, conda-forge/tiledb-feedstock#209 is in

I had apparently misinterpreted "in" to mean "merged"... 🤔

h-vetinari commented 1 year ago

So the tiledb PR is merged now. Ideally this will already solve most problems.

However, it would be good to add a run-requirement on libabseil 20230125 for all gcs-enabled tiledb-builds since https://github.com/conda-forge/tiledb-feedstock/pull/184, to avoid the solver picking one of those builds when trying to solve for newer abseil.

h-vetinari commented 1 year ago

FYI: I retriggered CI to see if the problem persists (it doesn't 🥳), but that doesn't mean that this PR should be merged -- it should not be necessary as gdal has no explicit dependence of any kind on abseil/grpc etc., only a transitive one through tiledb (and this is confirmed by the link check in the currently running CI as well).

akrherz commented 1 year ago

it should not be necessary as gdal has no explicit dependence of any kind on abseil/grpc etc.

Agreed, thank you for all the help.