rom1504 / embedding-reader

Efficiently read embedding in streaming from any filesystem
MIT License
92 stars 19 forks source link

aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed #62

Open DJLee68 opened 5 months ago

DJLee68 commented 5 months ago

Hi! I'm working on using this repo to download and utilize the clip embedding and meta-data of the laion data from https://the-eye.eu/public/AI/cah/laion5b/embeddings/laion2B-en/

However, while running a for loop that keeps batch processing, I'm getting an error like aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed. It doesn't happen at the beginning, it happens a few times in the middle of the for loop. (To be precise, the error is in this line)

I set the same parameters as the example code you provided https://github.com/rom1504/embedding-reader/blob/main/examples/aesthetic_inference.py (batch size, start, end, etc.)

My version of the library looks like this

aiohttp==3.9.3
fsspec==2024.3.1

Thank you in advance!