lithops-cloud / lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
http://lithops.cloud
Apache License 2.0
317 stars 105 forks source link

Make delay between call status check configurable #1292

Closed tomwhite closed 4 months ago

tomwhite commented 6 months ago

The time between calls to internal_storage.get_call_status is hardcoded to 1 second (GET_RESULT_SLEEP_SECS):

https://github.com/lithops-cloud/lithops/blob/b199505955f11602331e4725bf1844f885e2f7f8/lithops/future.py#L201-L203

It would be useful to be able to control this so that it is the same as either

The use case is running tests with local storage where it would be good to set it to a small value such as 0.1s so the tests run faster.

JosepSampe commented 5 months ago

In #1338 I exposed wait_dur_sec and retries of the future methods, this can be useful in the case you manually call future.wait() or future.get_result().

However, note that in a normal lithops execution where you use the fexec.wait() method, the while loop you pointed out is never reached, since the internal monitoring system is responsible to track the available files, and only calls future.status() or future.get_result() when the status/result file is already in the storage backend.

I see that you've already discovered this in your project, but for the record in this issue: to reduce the time between checks, utilize the monitoring_interval configuration parameter, and adjust the wait_dur_sec in the fexec.wait() method.

We can discuss whether it's appropriate to automatically set the wait_dur_sec of the fexec.wait() to match or closely resemble the monitoring_interval config parameter. This way, users wouldn't need to set two different parameters for the same purpose.

We can also discuss if it s appropriate to default the monitoring_interval config parameter to 0.1 (or similar) in case of the localhost storage backend (only if it is not set by the user in the config)

JosepSampe commented 4 months ago

Closing as the root issue is addressed by multiple PRs. WE can open a separate PR if there is a new issue or alternative requirement