search

Veldrovive / embedding-dataset-reordering

Reorders the embeddings generated by CLIP to be in line with the webdatasets generated by img2dataset.
MIT License
5 stars 3 forks source link

Something Wrong with the reorder #1

Open LearningJack opened 2 years ago

LearningJack commented 2 years ago

issue Could you help me? I had run the clip-retrieval and then generate the embedding.npy.But when I run the reorder-embeddings, it occured invalid literal for int() with base 10: 'img_test/COCO_train2014_000000581909'. How can I solve this problem?

Veldrovive commented 2 years ago

The reorder only works with the webdataset option from img2dataset. It looks like you have folders for your shards, but the webdataset format has the images in tar files.

If this does not meet your use case you can label this an enhancement issue and I can work allowing this format as well.

Veldrovive commented 2 years ago

Actually, no you're right. There is something strange here. Can you list the commands and arguments you used so I can replicate this state?

LearningJack commented 2 years ago

Sure, the commands and arguments are clip-retrieval inference --input_dataset ./000000 --output_folder img_emb_test and then reorder-embeddings reorder --embeddings-folder img_emb_test/img_emb --metadata-folder img_emb_test/metadata/ --output-folder img_emb_test/order --index-width 6 --output_shard_width 6 Or you can see the above image I upload. I see the issue is my metadata contains folder names and string. But I do not konw how to solve it.

Veldrovive commented 2 years ago

I think the issue is actually that the API for clip-retrieval changed slightly and the key attribute I was using no longer exists so it is falling back to image_path which is not always reliable. This shouldn't take too long to figure out, it's just that I'm very busy right now. I'll try to get this fixed tonight.

Veldrovive commented 2 years ago

Sure, the commands and arguments are clip-retrieval inference --input_dataset ./000000 --output_folder img_emb_test and then reorder-embeddings reorder --embeddings-folder img_emb_test/img_emb --metadata-folder img_emb_test/metadata/ --output-folder img_emb_test/order --index-width 6 --output_shard_width 6 Or you can see the above image I upload. I see the issue is my metadata contains folder names and string. But I do not konw how to solve it.

I assume that the problem is that your input is not a webdataset. It looks like you are specifying a shard for your input_dataset which itself is indicative of an issue. This repo was purpose built to work with the flow from img2dataset -> clip-retrieval where img2dataset generates a webdataset. It is more difficult to make it work with a folder structure as well because clip-retrieval does not generate a key for the image. In order to figure it out for this exact situation, I would need to be able to replicate the state of your files which means I would need more detail on exactly how you are getting to this point.

If you did use img2dataset to generate your dataset, you need to go back and use the --output_format="webdataset" option.

The other option is to wait for Rom and I to go back and implement the system within clip-retrieval which is planned for the not too distant future.