A single task hanging for 10 min on one provider causes agreements with other providers to be extended until it's done

maaktweluit commented 3 years ago

blender-yapapi-2021-05-25_12.19.09.log.gz Name: Artur Yapapi 0.6.0a Title: A single task hanging for 10 min on one provider causes agreements with other providers to be extended until it's done

    Task Task(id=2, data=10) timed out on descriptive-beast, time: 0:10:00.001003
    [2021-06-08 12:48:12,697 WARNING yapapi.summary] Worker for provider 'descriptive-beast' failed; reason: BatchTimeoutError()
    [2021-06-08 12:48:12,700 INFO yapapi.summary] Task started on provider 'changeable-statement', task data: 10
    [2021-06-08 12:48:15,597 INFO yapapi.summary] Terminated agreement with descriptive-beast
    [2021-06-08 12:48:16,745 INFO yapapi.summary] Accepted invoice from 'descriptive-beast', amount: 0.000364789440811111
    Task computed: Task(id=2, data=10), result: output_10.png, time: 0:00:11.778172
    [2021-06-08 12:48:24,484 INFO yapapi.summary] Task finished by provider 'changeable-statement', task data: 10
    [2021-06-08 12:48:24,485 INFO yapapi.summary] Computation finished in 618.5s
    [2021-06-08 12:48:24,485 INFO yapapi.summary] Negotiated 3 agreements with 3 providers
    [2021-06-08 12:48:24,485 INFO yapapi.summary] Provider 'likely-potato' computed 3 tasks
    [2021-06-08 12:48:24,485 INFO yapapi.summary] Provider 'changeable-statement' computed 3 tasks
    [2021-06-08 12:48:24,485 INFO yapapi.summary] Provider 'helpless-snow' did not compute any tasks
    [2021-06-08 12:48:24,485 INFO yapapi.summary] Provider 'descriptive-beast' did not compute any tasks
    [2021-06-08 12:48:24,485 INFO yapapi.summary] Activity failed 1 time on provider 'descriptive-beast'
    [2021-06-08 12:48:25,483 INFO yapapi.executor] Waiting for 2 workers to finish...
    [2021-06-08 12:48:26,959 INFO yapapi.executor] Waiting for Executor services to finish...
    6 tasks computed, total time: 0:10:20.936597
    [2021-06-08 12:48:26,959 INFO yapapi.executor] Golem is shutting down...
    [2021-06-08 12:48:26,960 INFO yapapi.executor] All jobs have finished
    [2021-06-08 12:48:26,960 INFO yapapi.executor] Waiting for Golem services to finish...
    [2021-06-08 12:48:27,167 INFO yapapi.summary] Accepted invoice from 'changeable-statement', amount: 0.004118599954461111
    [2021-06-08 12:48:28,423 INFO yapapi.summary] Accepted invoice from 'likely-potato', amount: 0.00000199040624975
    [2021-06-08 12:48:28,433 INFO yapapi.summary] Total cost: 0.004485379801521972
    [2021-06-08 12:48:28,433 INFO yapapi.summary] Executor has shut downPaste short logs here

maaktweluit commented 3 years ago

@azawlocki is this issue correctly put in yapapi repo?

azawlocki commented 3 years ago

@maaktweluit yes, that's where it belongs, thanks!

mateuszsrebrny commented 3 years ago

@azawlocki @shadeofblue is this a release blocker? Could you please have a look and label with timeframe label? Should we worry about this?

azawlocki commented 3 years ago

@mateuszsrebrny This issue is with us since beginning of yapapi and it didn't block any release yet, so I guess it's not a blocker :)

azawlocki commented 3 years ago

Note to SDK Team: in cases like this we could consider:

terminating agreements with all providers that are idle for, say, 1 minute; it wouldn't guarantee any progress with the task that got stuck but at least we wouldn't pay for idle agreements;
sending the task to another provider (perhaps also after some time); if another provider computes the task, we could terminate the agreement with the defective provider.

The second possibility would be harder to implement (we'd have to track which providers compute a task, and if any of them finishes, terminate agreements with the other ones).

golemfactory / yapapi

A single task hanging for 10 min on one provider causes agreements with other providers to be extended until it's done #441