yapapi issue with closing a python script when provider takes a long time to finish work

There is an error that occurs when closing execution of an example that uses yapapi.

Scenario: Python example requests generation of an image by an AI framework that is accessible in the golem network. The script runs in a forever loop asking for a prompt and requesting the image based on that prompt. It can be stopped by pressing Ctrl-C.

Steps to reproduce:

run python example code, e.g. python ai_runtime.py --network goerli --driver erc20next

Warning: this only works with yagna that supports gamerhash targeted code and has access to a provider that offers access to requested AI framework.

wait until it asks for a prompt and then type it
wait until it finishes work and asks for a prompt once again
press Ctrl-C

Python example: ai_runtime.py

Output:

Done
^CShutting down gracefully, please wait a short while or press Ctrl+C to exit immediately...
[2024-02-07T10:36:17.433+0100 WARNING yapapi.summary] Execution interrupted by CancelledError
[2024-02-07T10:36:20.712+0100 WARNING yapapi.services.service_runner] Unhandled exception in service
Traceback (most recent call last):
  File "/home/skrotki/golem/gamerhash-facade/example/ai-requestor/ai_runtime.py", line 105, in run_golem_example
    loop.run_until_complete(task)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
    self._run_once()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1871, in _run_once
    event_list = self._selector.select(timeout)
  File "/usr/lib/python3.10/selectors.py", line 469, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/skrotki/golem/gamerhash-facade/example/ai-requestor/.venv/lib/python3.10/site-packages/yapapi/rest/activity.py", line 244, in _get_results
    results = await self._activity._api.get_exec_batch_results(
  File "/home/skrotki/golem/gamerhash-facade/example/ai-requestor/.venv/lib/python3.10/site-packages/ya_activity/api_client.py", line 205, in __call_api
    raise e
  File "/home/skrotki/golem/gamerhash-facade/example/ai-requestor/.venv/lib/python3.10/site-packages/ya_activity/api_client.py", line 193, in __call_api
    response_data = await self.request(
  File "/home/skrotki/golem/gamerhash-facade/example/ai-requestor/.venv/lib/python3.10/site-packages/ya_activity/rest.py", line 192, in GET
    return await self.request(
  File "/home/skrotki/golem/gamerhash-facade/example/ai-requestor/.venv/lib/python3.10/site-packages/ya_activity/rest.py", line 180, in request
    raise ApiException(http_resp=r)
ya_activity.exceptions.ApiException: (500)
Reason: Internal Server Error
HTTP response headers: <CIMultiDictProxy('Content-Length': '77', 'Vary': 'Origin, Access-Control-Request-Method, Access-Control-Request-Headers', 'Content-Type': 'application/json', 'Date': 'Wed, 07 Feb 2024 09:36:20 GMT')>
HTTP response body: {"message":"GSB error: Bad request: Bad request: endpoint address not found"}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/skrotki/golem/gamerhash-facade/example/ai-requestor/.venv/lib/python3.10/site-packages/yapapi/services/service_runner.py", line 398, in _run_instance
    batch = batch_task.result()
  File "/home/skrotki/golem/gamerhash-facade/example/ai-requestor/.venv/lib/python3.10/site-packages/yapapi/services/service.py", line 369, in shutdown
    yield s
  File "/home/skrotki/golem/gamerhash-facade/example/ai-requestor/.venv/lib/python3.10/site-packages/yapapi/services/service_runner.py", line 414, in _run_instance
    fut_result = yield batch
  File "/home/skrotki/golem/gamerhash-facade/example/ai-requestor/.venv/lib/python3.10/site-packages/yapapi/engine.py", line 781, in process_batches
    results = await get_batch_results()
  File "/home/skrotki/golem/gamerhash-facade/example/ai-requestor/.venv/lib/python3.10/site-packages/yapapi/engine.py", line 761, in get_batch_results
    async for event_class, event_kwargs in remote:
  File "/home/skrotki/golem/gamerhash-facade/example/ai-requestor/.venv/lib/python3.10/site-packages/yapapi/rest/activity.py", line 283, in __aiter__
    results = await self._get_results(timeout=min(timeout, 5))
  File "/home/skrotki/golem/gamerhash-facade/example/ai-requestor/.venv/lib/python3.10/site-packages/yapapi/rest/activity.py", line 254, in _get_results
    raise BatchError("Activity terminated by provider", reason, error_msg)
yapapi.rest.activity.BatchError: ('Activity terminated by provider', None, None)

Observation:

if the code that handles get_exec_batch_results in yagna has some added artificial delays (~10sec) then the exception is not thrown

golemfactory / ya-runtime-ai

yapapi issue with closing a python script when provider takes a long time to finish work #55

There is an error that occurs when closing execution of an example that uses yapapi.