atviriduomenys / spinta

Spinta is a framework to describe, extract and publish data (a DEP Framework).
MIT License
10 stars 4 forks source link

Add requests timeout on push operation #662

Open sirex opened 1 week ago

sirex commented 1 week ago

Currently, spinta push uses requests to access remote Spinta server, but does not specify any timeouts.

requests documentation says:

Nearly all production code should use this parameter in nearly all requests. Failure to do so can cause your program to hang indefinitely. -- https://docs.python-requests.org/en/latest/user/quickstart/#timeouts

There are no "correct" timeout value, that is why requests does not specify timeout by default. In some cases it could be 1 second, in other cases 30 minutes.

Also there are two timeout options:

In case of read timeout error, it should be suggested, that a smaller --chunk-size could be used, to avoid timeouts.