When use clip-retrieval inference.worker, if the file format is webdataset and there are more than one tar file, only one file is read

rom1504 / clip-retrieval

Easily compute clip embeddings and build a clip retrieval system with them

https://rom1504.github.io/clip-retrieval/

MIT License

2.42k stars 213 forks source link

When use clip-retrieval inference.worker, if the file format is webdataset and there are more than one tar file, only one file is read #245

Closed AlanKang98 closed 1 year ago

AlanKang98 commented 1 year ago

And if use clip-retrieval inference, when reading multiple files from webdataset, the torch will rise the error: File "python3.7/site-packages/torch/utils/data/dataloader.py", line 233, in init if num_workers < 0: TypeError: '<' not supported between instances of 'str' and 'int'

simhermansson commented 1 year ago

I had the problem where it only read one tar file when using clip-retrieval inference. I realized you have to use double quotes around the path to the dataset. Like "path/to/data/{000..010}.tar" instead of path/to/data/{000..010}.tar. Make sure to use = when passing arguments also.