Dependency TODOs: enablements & clean-ups

[h-vetinari]

From looking at the list of third-party deps, we could still do better with:

• [ ] azure bindings
  • [x] unix
  • [ ] windows: https://github.com/apache/arrow/issues/41990
• [x] ~jemalloc~ (not possible?)
• [ ] opentelemetry
  • [x] https://github.com/conda-forge/cpp-opentelemetry-sdk-feedstock/issues/38
  • [ ] https://github.com/conda-forge/arrow-cpp-feedstock/pull/1048
• [x] orc for windows
  • [x] https://github.com/conda-forge/orc-feedstock/issues/54
  • [x] enabled for libarrow (as of https://github.com/conda-forge/arrow-cpp-feedstock/commit/db46b2c2a04f8b301da7b6e9a1f04452a5f75698)
  • [x] https://github.com/apache/arrow/issues/36026
  • [x] https://github.com/apache/orc/pull/1882
  • [x] https://github.com/conda-forge/arrow-cpp-feedstock/pull/1086
  • [x] https://github.com/conda-forge/arrow-cpp-feedstock/pull/1424
  • [x] https://github.com/conda-forge/pyarrow-feedstock/pull/122
  • [x] https://github.com/apache/arrow/pull/41767
• [x] nlohmann_json (as of https://github.com/conda-forge/arrow-cpp-feedstock/commit/c548fc22965eefb7351e90a1a8a0f97ea8096ce4)
• [ ] substrait (not yet in conda-forge)
• [x] ucx (as of https://github.com/conda-forge/arrow-cpp-feedstock/commit/c548fc22965eefb7351e90a1a8a0f97ea8096ce4)

Switch to shared builds for (if/once available):

• [ ] google-cloud-cpp (win; not yet available in cf)
• [x] gflags (win; available, but apparently symbols aren't needed; clarified as of https://github.com/conda-forge/arrow-cpp-feedstock/commit/bda7682f93eb55b7b1c2ef66506e98de8bdfbe8c)
• [x] glog (win; available, but apparently symbols aren't needed; clarified as of https://github.com/conda-forge/arrow-cpp-feedstock/commit/bda7682f93eb55b7b1c2ef66506e98de8bdfbe8c)
• [x] grpc (win; available as of 1.55, which requires protobuf >=4.23; fixed as of https://github.com/conda-forge/arrow-cpp-feedstock/commit/025b77a45fcc89c609eb7cccee1b4840b331305b)
• [x] thrift (needed fixes for CMake metadata; fully cleaned up as of https://github.com/conda-forge/arrow-cpp-feedstock/commit/2af3d6096a5cd3b6b6ab17aa5def0073b565e6d2)

Bindings:

• [ ] skyhook (needs librados)
• [ ] tensorflow (though we should use libtensorflow, if at all, because it's a very heavy dependency to pull in)

[xhochy]

We cannot unvendor jemalloc as the build is with special options. Unverdoring it would have an impact on the default allocator.

[jakirkham]

Would it be possible to just build a variant of the jemalloc package with options needed?

[jakirkham]

Would not count on ucx being supported on other ~architectures~ OSes. Would also like to understand why that is relevant

[h-vetinari]

Would also like to understand why that is relevant

I have a maximalist approach to packaging the bindings offered by a given library -- if the library offers integration for it, why shouldn't we package it?[^1]

Users rely on conda-forge because a lot of these packages are really hard to build from source, so if a binding is not supported by us, it's effectively unusuable (even moreso if the distributed wheels also turn it off).

[^1]: whether to break a package into separate outputs is a separate question

[jakirkham]

Sorry I should clarify, what I mean is why are other OSes relevant? Asking since ucx likely will remain Linux only (possibly macOS at some point and basically no chance of Windows support)

[h-vetinari]

Sorry I should clarify, what I mean is why are other OSes relevant? Asking since ucx likely will remain Linux only (possibly macOS at some point and basically no chance of Windows support)

Not relevant indeed, unless they become: a) available for the respective platform & b) are supported by arrow on that platform.