search

TileDB-Inc / tiledbsoma-feedstock

A conda-smithy repository for tiledbsoma.
BSD 3-Clause "New" or "Revised" License
3 stars 3 forks source link

tiledbsoma 1.8.0 pre-check #91

Closed johnkerl closed 4 months ago

johnkerl commented 4 months ago

Following our established procedure

See also #93 which addresses issues which have happened since 1.7.2 -- things involving our dependencies -- which have nothing to do with 1.8.0

See also https://github.com/single-cell-data/TileDB-SOMA/issues/2219 for tracking.

johnkerl commented 4 months ago

[sc-41030]

shortcut-integration[bot] commented 4 months ago

This pull request has been linked to Shortcut Story #41030: Update projects to TileDB 2.20.

jdblischak commented 4 months ago

Currently failing with similar conda solver errors as we previously identified in https://github.com/TileDB-Inc/tiledbsoma-feedstock/pull/89#issuecomment-1965013717

We are getting conda solver errors in multiple feedstocks, so there must be some systemic problem. I'm hoping that this commit merged earlier this morning (which migrates to libgoogle-cloud 2.21) will help ease the solver problems 🤞

johnkerl commented 4 months ago

I'm hoping that this commit merged earlier this morning (which migrates to libgoogle-cloud 2.21) will help ease the solver problems 🤞

@jdblischak are we waiting for some sort of propagation? 🤔

jdblischak commented 4 months ago

are we waiting for some sort of propagation?

Yes, I'm hoping that maybe the new tiledb 2.20.1 binaries that were built against libgoogle-cloud 2.21 might help with the current solver issue. They were uploaded more than 2 hours ago, so should be available now

https://anaconda.org/conda-forge/tiledb/files?version=2.20.1

(note that the build numbers are weird (two different sets with build number 1) because of all the merge commits we've done recently in tiledb-feedstock. Fortunately the migration to libgoogle-cloud 2.21 also changed the build hash, so new binaries were uploaded)

jdblischak commented 4 months ago

Welp that didn't fix it. The latest failed CI run used the latest tiledb built against libgoogle-cloud 2.21

    libgoogle-cloud:          2.21.0-h31df0ca_1           conda-forge
jdblischak commented 4 months ago

The solver error from the latest run was decently informative. We know it has to do with libabseil, libcurl, and libgoogle-cloud-storage

    Could not solve for environment specs
    The following packages are incompatible
    ├─ libabseil ==20240116.1 cxx17_h59595ed_1 is requested and can be installed;
    ├─ libcurl ==8.5.0 hca28451_0 is requested and can be installed;
    ├─ libgoogle-cloud-storage ==2.21.0 hc7a4891_2 is requested and can be installed;
    └─ libtiledbsoma ==1.8.0 hd763302_0 is not installable because it requires
       └─ tiledb >=2.20.1,<2.21.0a0  but there are no viable options
          ├─ tiledb 2.20.1 would require
          │  └─ libcurl >=7.88.1,<8.0a0 , which conflicts with any installable versions previously reported;
          ├─ tiledb 2.20.1 would require
          │  └─ libgoogle-cloud-storage >=2.17.0,<2.18.0a0 , which conflicts with any installable versions previously reported;
          └─ tiledb 2.20.1 would require
             └─ libabseil >=20230802.1,<20230803.0a0 , which conflicts with any installable versions previously reported.
johnkerl commented 4 months ago

https://anaconda.org/conda-forge/google-cloud-cpp/files updated 4 days ago ... 🤔

johnkerl commented 4 months ago

Weird thing is 2.17 doesn't exist at all at https://anaconda.org/conda-forge/google-cloud-cpp/files ... 🤔

jdblischak commented 4 months ago

https://anaconda.org/conda-forge/libgoogle-cloud-storage/files?version=2.17.0

johnkerl commented 4 months ago

OK I may be mixing google-cloud-cpp and google-cloud-storage ... OTOH from the last raw log:

johnkerl@exade[prod][][~]$ grep -C1 google-cloud Desktop/raw.txt
2024-02-28T22:31:52.2156877Z     libgomp:                  13.2.0-h807b86a_5           conda-forge
2024-02-28T22:31:52.2157237Z     libgoogle-cloud:          2.21.0-h31df0ca_1           conda-forge
2024-02-28T22:31:52.2157609Z     libgoogle-cloud-storage:  2.21.0-hc7a4891_1           conda-forge
2024-02-28T22:31:52.2157976Z     libgrpc:                  1.60.1-h74775cd_0           conda-forge
--
2024-02-28T22:34:23.3632616Z     libgomp:                   13.2.0-h807b86a_5           conda-forge
2024-02-28T22:34:23.3633104Z     libgoogle-cloud:           2.17.0-h31df0ca_2           conda-forge
2024-02-28T22:34:23.3634545Z     libgoogle-cloud-storage:   2.17.0-hc7a4891_2           conda-forge
2024-02-28T22:34:23.3635111Z     libgrpc:                   1.60.1-h74775cd_0           conda-forge
--
2024-02-28T22:36:32.5382008Z     libgomp:                   13.2.0-h807b86a_5           conda-forge
2024-02-28T22:36:32.5382437Z     libgoogle-cloud:           2.17.0-h31df0ca_2           conda-forge
2024-02-28T22:36:32.5382872Z     libgoogle-cloud-storage:   2.17.0-hc7a4891_2           conda-forge
2024-02-28T22:36:32.5383298Z     libgrpc:                   1.60.1-h74775cd_0           conda-forge
--
2024-02-28T22:38:44.8621716Z     ├─ libcurl ==8.5.0 hca28451_0 is requested and can be installed;
2024-02-28T22:38:44.8622037Z     ├─ libgoogle-cloud-storage ==2.21.0 hc7a4891_2 is requested and can be installed;
2024-02-28T22:38:44.8622353Z     └─ libtiledbsoma ==1.8.0 hd763302_0 is not installable because it requires
--
2024-02-28T22:38:44.8623815Z           ├─ tiledb 2.20.1 would require
2024-02-28T22:38:44.8624167Z           │  └─ libgoogle-cloud-storage >=2.17.0,<2.18.0a0 , which conflicts with any installable versions previously reported;
2024-02-28T22:38:44.8636944Z           └─ tiledb 2.20.1 would require

🤔

johnkerl commented 4 months ago

[sc-42200]

shortcut-integration[bot] commented 4 months ago

This pull request has been linked to Shortcut Story #42200: tiledbsoma 1.8.0.

jdblischak commented 4 months ago

The osx-64 build is now failing because the r-tiledbsoma compilation is failing with a bunch of macOS SDK errors. Maybe we need to adjust MACOSX_SDK_VERSION and/or MACOSX_DEPLOYMENT_TARGET?

https://github.com/TileDB-Inc/tiledbsoma-feedstock/blob/6663e5111ccd0a74dd5e4f01781918f4bf52879d/recipe/conda_build_config.yaml#L1-L4

Here is the final part of the errors:

/Users/runner/miniforge3/conda-bld/tiledbsoma_1709247111523/_build_env/bin/../include/c++/v1/__filesystem/path.h:780:25: note: 'operator basic_string' has been explicitly marked unavailable here
  _LIBCPP_HIDE_FROM_ABI operator string_type() const { return __pn_; }
                        ^
In file included from RcppExports.cpp:4:
In file included from ./../inst/include/tiledbsoma_types.h:21:
In file included from /Users/runner/miniforge3/conda-bld/tiledbsoma_1709247111523/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include/tiledbsoma/tiledbsoma:51:
/Users/runner/miniforge3/conda-bld/tiledbsoma_1709247111523/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include/tiledbsoma/soma/soma_collection.h:105:44: error: 'path' is unavailable: introduced in macOS 10.15 - see https://conda-forge.org/docs/maintainer/knowledge_base.html#newer-c-features-with-old-sdk
              std::string(std::filesystem::path(uri).filename()),  // group name
                                           ^
/Users/runner/miniforge3/conda-bld/tiledbsoma_1709247111523/_build_env/bin/../include/c++/v1/__filesystem/path.h:442:24: note: 'path' has been explicitly marked unavailable here
class _LIBCPP_TYPE_VIS path {
                       ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make: *** [/Users/runner/miniforge3/conda-bld/tiledbsoma_1709247111523/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/R/etc/Makeconf:200: RcppExports.o] Error 1
ERROR: compilation failed for package ‘tiledbsoma’
* removing ‘/Users/runner/miniforge3/conda-bld/tiledbsoma_1709247111523/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/R/library/tiledbsoma’

In the osx-arm64 build, r-tiledbsoma compiles fine. The failure is due to tiledbsoma-py being unable to find libtiledbsoma.so in the conda env, and then failing to build it from source:

running build_ext
Checking: /Users/runner/miniforge3/conda-bld/tiledbsoma_1709247012541/work/dist/lib/libtiledbsoma.dylib exists: False
dlopen(libtiledbsoma.dylib, 6): image not found
Build with TileDB: /Users/runner/miniforge3/conda-bld/tiledbsoma_1709247012541/work/dist
Building Release build
./bld: line 88: cmake: command not found

My guess is that this might be due to https://github.com/single-cell-data/TileDB-SOMA/pull/1937 since it updated how tiledbsoma-py interacts with an existing libtiledbsoma.so

jdblischak commented 4 months ago

From the nightly feedstock builds, the R compilation error on osx-64 started on Feb 12th. And the osx-arm64 builds started failing on Feb 16th, so I am more confident that it was caused by https://github.com/single-cell-data/TileDB-SOMA/pull/1937, which was merged earlier that day in https://github.com/single-cell-data/TileDB-SOMA/commit/f4999d8ea89f7fcfb710380e64f3572acd01d5ca

johnkerl commented 4 months ago

I've moved this PR back to draft status -- please ignore until I mark this ready for review again -- I'm trying to figure out (mainly via local builds) what worked about #93 that didn't work here ...

johnkerl commented 4 months ago

Oops, I missed comment https://github.com/TileDB-Inc/tiledbsoma-feedstock/pull/91#issuecomment-1972361731 ... 👀

johnkerl commented 4 months ago

@jdblischak FYI - libgoogle-cloud 2.17.* no longer solves -- this happened today -- I've removed that line

I've confirmed this locally (for linux) as well

johnkerl commented 4 months ago

@jdblischak @ihnorton linux is once again green.

OSX is not; and I've switched from 10.14 to 10.15 in an attempt to rid errors of the form

2024-03-01T22:51:51.7619640Z /Users/runner/miniforge3/conda-bld/tiledbsoma_1709332344079/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/include/tiledbsoma/soma/soma_sparse_ndarray.h:121:54: error: '~path' is unavailable: introduced in macOS 10.15 - see https://conda-forge.org/docs/maintainer/knowledge_base.html#newer-c-features-with-old-sdk
2024-03-01T22:51:51.7620440Z               std::string(std::filesystem::path(uri).filename()),  // array name
2024-03-01T22:51:51.7620860Z                                                      ^
2024-03-01T22:51:51.7621620Z /Users/runner/miniforge3/conda-bld/tiledbsoma_1709332344079/_build_env/bin/../include/c++/v1/__filesystem/path.h:505:3: note: '~path' has been explicitly marked unavailable here
2024-03-01T22:51:51.7622230Z   ~path() = default;
2024-03-01T22:51:51.7622590Z   ^

which is weird b/c of the error

error: '~path' is unavailable: introduced in macOS 10.15

and I am using 10.15 :(

johnkerl commented 4 months ago

... raw logs reveal 10.14 still appearing in log output. Trying a rerender ...

johnkerl commented 4 months ago

https://github.com/TileDB-Inc/tiledbsoma-feedstock/pull/91/commits/d416ebc97e31711753b21d280d547389f53ad7a4 🤞

jdblischak commented 4 months ago

Ugh, the osx-64 one failed with a spurious download error. That is the main one we need feedback on. I'm going to cancel and restart the CI jobs

johnkerl commented 4 months ago

See also https://github.com/single-cell-data/TileDB-SOMA/issues/2210

johnkerl commented 4 months ago

CI is green; this PR has done its job per our established procedure

See also notes on https://github.com/single-cell-data/TileDB-SOMA/issues/2210 that the cmake dependency in the build line here is a workaround not a fix -- build-from-source should not be happening in the first place

johnkerl commented 4 months ago

More advice from @jdblischak

jdblischak commented 4 months ago

The --no-build-isolation flag is working as intended. pip is no longer installing the build deps

xref: https://github.com/TileDB-Inc/tiledbsoma-feedstock/pull/95#issuecomment-1981596862

/home/conda/feedstock_root/build_artifacts/tiledbsoma_1709752782816/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/python -m pip install --no-deps --no-build-isolation . -vv
Using pip 24.0 from /home/conda/feedstock_root/build_artifacts/tiledbsoma_1709752782816/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.10/site-packages/pip (python 3.10)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-build-tracker-_creqxao
Initialized build tracking at /tmp/pip-build-tracker-_creqxao
Created build tracker: /tmp/pip-build-tracker-_creqxao
Entered build tracker: /tmp/pip-build-tracker-_creqxao
Created temporary directory: /tmp/pip-install-qfsfg8yk
Created temporary directory: /tmp/pip-ephem-wheel-cache-ly63dwyh
Processing /home/conda/feedstock_root/build_artifacts/tiledbsoma_1709752782816/work/apis/python
  Added file:///home/conda/feedstock_root/build_artifacts/tiledbsoma_1709752782816/work/apis/python to build tracker '/tmp/pip-build-tracker-_creqxao'
  Created temporary directory: /tmp/pip-modern-metadata-3e9ncd7i
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-3e9ncd7i/tiledbsoma.egg-info
  writing /tmp/pip-modern-metadata-3e9ncd7i/tiledbsoma.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-3e9ncd7i/tiledbsoma.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-modern-metadata-3e9ncd7i/tiledbsoma.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-3e9ncd7i/tiledbsoma.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-3e9ncd7i/tiledbsoma.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'py.typed'
  no previously-included directories found matching 'dist_links/libtiledbsoma/test/__pycache__'
  writing manifest file '/tmp/pip-modern-metadata-3e9ncd7i/tiledbsoma.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-3e9ncd7i/tiledbsoma-1.5.0rc0.post226.dev267876719.dist-info'
  Preparing metadata (pyproject.toml): finished with status 'done'
  Source in /home/conda/feedstock_root/build_artifacts/tiledbsoma_1709752782816/work/apis/python has version 1.5.0rc0.post226.dev267876719, which satisfies requirement tiledbsoma==1.5.0rc0.post226.dev267876719 from file:///home/conda/feedstock_root/build_artifacts/tiledbsoma_1709752782816/work/apis/python
  Removed tiledbsoma==1.5.0rc0.post226.dev267876719 from file:///home/conda/feedstock_root/build_artifacts/tiledbsoma_1709752782816/work/apis/python from build tracker '/tmp/pip-build-tracker-_creqxao'
Created temporary directory: /tmp/pip-unpack-q9ol_44z
Building wheels for collected packages: tiledbsoma
  Created temporary directory: /tmp/pip-wheel-shq_ssco
  Destination directory: /tmp/pip-wheel-shq_ssco
  Building wheel for tiledbsoma (pyproject.toml): started
  Running command Building wheel for tiledbsoma (pyproject.toml)
  running bdist_wheel
  Checking: /home/conda/feedstock_root/build_artifacts/tiledbsoma_1709752782816/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/libtiledbsoma.so exists: True
  running build
  running build_py