Limit the listing scope by the storage directory specified in the URI

[mnrozhkov]

Description

When running the code below for the first time, it takes too much time to list. I guess it's because Datachain does listing all files in the bucket. I think it makes sense to limit it to the directory specified in the URI. Is it correct?

It becomes more crucial for CI/CD pipelines where I need to run a test script. Listing all files in the bucket is unnecessary.

Code to reproduce
(from examples/computer_vision/fashion_product_images/1-quick-start.ipynb)

# Create a DataChain

dc = (
    DataChain.from_storage(
        "gs://datachain-demo/fashion-product-images", type="image", anon=True
    )
    .filter(C("file.name").glob("*.jpg"))
    .save()
)

• There are 44K files inside /fashion-product-images
• Listing finds 270K files
• => it's 6x times slower

Output:

Version Info

No response

[shcheklein]

@mnrozhkov workaround is to put / at the end of the URL.

Improvement to the semantics is here: https://github.com/iterative/datachain/pull/108

[shcheklein]

Hey, @iterative/datachain please review https://github.com/iterative/datachain/pull/108 (cc @ilongin )

[shcheklein]

Should be fixed in the latest release.