search

aimhubio / aim

Aim 💫 — An easy-to-use & supercharged open-source experiment tracker.
https://aimstack.io
Apache License 2.0
4.93k stars 297 forks source link

Large validation image fails to send to server #3154

Open jasonmads opened 3 weeks ago

jasonmads commented 3 weeks ago

🐛 Large validation image fails to send

Getting this error in my training logs when trying to send a validation image to server. It is a jpg image about 4mb in size. [aim.ext.transport.request_queue][WARNING] - Remote Server is unavailable, please check network connection: received 1009 (message too big); then sent 1009 (message too big).

To reproduce

Generate a large validation image

Expected behavior

No error and get the image in images section.

Environment

aim==3.19.3 and main(3.20.1) Python 3.11.6 and 3.10.12 pip 24.0 Ubuntu 22.04

Additional context

I did get things working doing the following. Is it the best way? Should there be some kind of config options? I can make a PR in cli/utils.py Added --ws-max-size 1073741824 to uvicorn command

in ext/transport/client.py added max_size=None to connect (self._ws)

alberttorosyan commented 3 weeks ago

Hey @jasonmads! Thanks for reporting this issue. Opening a PR would be great! Appreciate your contribution 🙌