search

laminlabs / lamindb-setup

Setup & configure LaminDB.
Apache License 2.0
4 stars 2 forks source link

🎨 Make `get_stat_dir_s3` compatible with canonical storage policy (`s3:ListBucket` conditional on `prefix/`) #675

Closed bpenteado closed 6 months ago

bpenteado commented 6 months ago

Performance benchmarking: https://lamin.ai/laminlabs/lamindata/transform/krGp3hT1f78N5zKv

github-actions[bot] commented 6 months ago

🚀 Deployed on https://65ef9465a0f7c031d9626cc0--lamindb-setup-htry.netlify.app

falexwolf commented 6 months ago

Here is a directory from an LTS cellxgene census release that you can use for the performance check:

ln.UPath("s3://cellxgene-data-public/cell-census/2023-12-15/soma").view_tree(level=4)
falexwolf commented 6 months ago

For testing, you could e.g. use this one:

ln.UPath("s3://lamindb-dev-datasets/iris_studies/study0_raw_images").view_tree()
codecov[bot] commented 6 months ago

Codecov Report

Attention: Patch coverage is 92.85714% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 81.65%. Comparing base (0a170a3) to head (a58b50f).

Files Patch % Lines
lamindb_setup/core/upath.py 92.85% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #675 +/- ## ========================================== + Coverage 80.80% 81.65% +0.84% ========================================== Files 39 39 Lines 2371 2366 -5 ========================================== + Hits 1916 1932 +16 + Misses 455 434 -21 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

falexwolf commented 6 months ago

Yeah, if you run it in the "storage" group, you need anon, because there are no AWS creds in the runner env: https://github.com/laminlabs/lamindb-setup/blob/0a170a3b23d665fae9f4aaad6fab47943e5c3615/noxfile.py#L101-L103