search

dandi / dandi-archive

DANDI API server and Web app
https://dandiarchive.org
14 stars 10 forks source link

find files matching a regex #1002

Closed satra closed 2 years ago

satra commented 2 years ago

how do i best search for files matching a path regex? given the sizes of dandiset 26 and 108, i would prefer not having to get all the assets to filter it. with the hackathon coming up, this would be a good thing to be able to support filters at some endpoint.

sub-I46/*/*.ome.tiff
*.ngff

related #230

waxlamp commented 2 years ago

When is the hackathon?

satra commented 2 years ago

apr 21/22, but it would be good to build out tutorials and scripts a bit earlier.

jjnesbitt commented 2 years ago

how do i best search for files matching a path regex? given the sizes of dandiset 26 and 108, i would prefer not having to get all the assets to filter it. with the hackathon coming up, this would be a good thing to be able to support filters at some endpoint.

sub-I46/*/*.ome.tiff
*.ngff

related #230

To clarify, your example uses filename globbing, not regex. I'm assuming you'd intend to use this in a manner similar to your example, and so you'd want globbing support?

As far as feasibility, it should be easy. Django (really postgres) supports regex filtering, which globs can be easily converted to.

satra commented 2 years ago

indeed my current examples are globs. regex would be a superset. but just having globs would be achieve 99% of use cases i think.

jjnesbitt commented 2 years ago

indeed my current examples are globs. regex would be a superset. but just having globs would be achieve 99% of use cases i think.

My preference would be globs since that's both easier to understand for more people, as well as more natural when it comes to filenames.

yarikoptic commented 2 years ago

FWIW, seeing globs (or regex) support in API would be awesome. Echoing brief discussion in #230, I would have added path_type=(exact|glob|regex) to re-use path argument while giving it expandable (unlike a bool flag) list of possibilities to be treated like ;)

waxlamp commented 2 years ago

@satra, we're 80% confident we can get this done by the end of this week. That should give enough time to put together hackathon tutorials, right?

satra commented 2 years ago

That should give enough time to put together hackathon tutorials, right?

yup.

dandibot commented 2 years ago

:rocket: Issue was released in v0.2.13 :rocket:

yarikoptic commented 2 years ago

reincarnated in https://github.com/dandi/dandi-archive/issues/1160