Closed arky closed 6 months ago
Thanks for the heads up. That error message could definitely be friendlier.
It must be intermittent issues on the iNat server side. Have you noticed this only with create_observation()
, or also with read-only endpoints (like get_observations()
)? By default, pyinaturalist sends requests with a timeout of 10 seconds, and will retry failed requests up to 5 times with delays and exponential backoff. I could bump up the default timeout for create_observation()
, maybe something like 30 seconds?
If you want, you can modify these settings yourself with a ClientSession object. Example:
from pyinaturalist import create_observation, ClientSession
session = ClientSession(timeout=30, max_retries=10)
create_observation(
...,
session=session,
)
@JWCook Am in Southeast Asia with bit of latency, so sending a post payload with json, image and audio file causing the above error every other requests. Thanks for the tip on using clientsession with custom timeout and retries options, this should be documented in docs, it would have saved me valuable time.
I haven't noticed any timeouts on get_observations yet, perhaps I need to spend some time to do some metrics and report back here.
Good point about slower internet connections. I'll bump up the default timeout, and add some more info about that to the docs.
a post payload with json, image and audio file
It's also worth noting that each media file is uploaded in a separate request. So if, for example, you need to create an observation with 10 images and 10 audio files, you don't need to increase your timeout by 10x.
@JWCook What finally seems to work well is to send each observation every 10 seconds and handle any timeout errors using python exceptions.
If an exception happens, then wait for another additional 10-20 seconds before sending any new create_observations.
Perhaps this backing off strategy seems to work for now. My guess is that iNat servers might have write latency and might respond badly if you send too many requests and file payloads too frequently.
@JWCook After further testing, I strongly suspect the problem is with the add '{API_V1}/observation_sounds
endpoint which seems to cause most amount of write errors causing timeouts.
Hmm, in iNat's backend I don't think there's much difference in how it handles image uploads vs audio uploads. Could it be correlated with file size? Roughly how large are the files you're uploading? And are you still seeing the same type of error (TimeoutError('The write operation timed out')
)?
Also, there is already rate limiting built in (per_second
, per_minute
, and, per_day
parameters for ClientSession
), but it currently doesn't go any slower than 1 request per second. That's something I could fix, though.
@JWCook Am perplexed with this, there seems to be no correlation with the size of files. It is just that the server just gives up or stops responding triggering a time out error.
Will also play around with combination of per_* ClientSession options to see what's happening here.
Small update: if you pip install -U requests-ratelimiter
, you can now use floats as rate limit values (for <1 request per second). For example:
session = ClientSession(per_second=0.1)
Am using both per_second setting of requests-ratelimiter and tenacity's retry method to manage to upload. Still get timeout errors regularly but slowly managing to upload the backlog of observations.
@retry(wait=wait_exponential(multiplier=10, min=10, max=360) ```
@JWCook Shall we close this bug as this seems to be caused mainly by the iNat's servers capacity to handle audio files. What do you reckon?
Yeah, I don't think there's much more we can do about this on the client side. Sorry, I wish I could be of more help!
On the iNat server side, something that would help is support for uploading files in small chunks, and resuming failed partial uploads. As far as I know, that's not currently possible.
After thinking about it a bit more, it would be possible to set up an intermediate service that emulates this using a cloud storage provider. Something like:
Let me know if that's something you'd be interested in looking into. I may not have time to work on something like that right now, but I could at least give some pointers and examples.
Thank you @JWCook . Perhaps I'll reach out to iNat developers through NatGeo to improve their upload process as you suggested in future.
The problem
Sending create_observation fails intermittently. This might be due to slow response from API endpoint.
Expected behavior
Print an error about setting higher timeout or adding sleep between requests.
Steps to reproduce the behavior
Workarounds
Is there an existing workaround for this issue?
Environment