Build an end to end command

[rom1504]

Url list -> filtering (dedup) -> downloading -> clip inference -> indexing -> back + front (subprocess or host with back too)

clip-retrieval end2end <url list> <config.json>

It would start a prefect UI with what's going on and wandb links for each subtask Then after a small while, it will start the back and front and open the demo in the browser

Build it with prefect, use a good config framework (fromconfig ?)

Would be ideal to make it incremental and schedulable too. Making it distributed potentially could also be interesting but not necessary.

[rom1504]

Could even consider having some optional stats computation and even trainings at the end. A full dataset construction pipeline starting from urls. Important point:

• Having well packaged and separated components
• having logging in each component, reporting to wandb
• Having a modern pipeline system that works both for experimentation and for scheduling
• expose all the components to make everything as simple as wanted
• Have a simple configuration system (could be half python half yaml) to let the users decide what they want to start
• Collect the necessary resources up front and inform the user

[rom1504]

result will look like this https://colab.research.google.com/github/criteo/autofaiss/blob/master/docs/notebooks/autofaiss_multimodal_search.ipynb but bette packaged

[rom1504]

https://github.com/rom1504/img2dataset#api

the config is important. Figure out a way to expose all options and yet do the right things by default with as few as possible necessary arguments to pass

[rom1504]

it should be possible to do url list -> index + metadata store at almost no memory usage and in one step when

• img2dataset provides an iterator (or a stream to be able to handle back pressure)
• clip retrieval inference can take an iterator and provides an iterator
• autofaiss can take an iterator and writes to disk part of indices

would look like this: clip-retrieval end2end <url list> <output path> that would use for laion400m:

• 2 Gbps of bandwidth
• 4GB of ram
• 16 cores
• 100 GB of disk for metadata, 16GB of disk for the index
• 4 days

interesting possibility but might not be that important vs working incrementally

[rom1504]

the basic is now done

next:

• [ ] configuration system
• [ ] incremental
• [ ] scheduling

[rom1504]

config modes:

• everything defined by the user
• lot of default
• guess everything

Each one will be useful for maximum convenience or configurability

[rom1504]

also consider the option of having end2end be an example and let people do their preferred config in python