search

oda-hub / dispatcher-plugin-integral

Other
1 stars 1 forks source link

when job fails on backend, dispatcher may time out API #51

Closed volodymyrss closed 3 years ago

volodymyrss commented 3 years ago

also https://github.com/oda-hub/dispatcher-app/issues/225

volodymyrss commented 3 years ago

example is needed @ferrigno !

volodymyrss commented 3 years ago

maybe here https://github.com/oda-hub/oda_api/blob/Release-Update/doc/source/user_guide/AsynchronousAPI.ipynb

ferrigno commented 3 years ago

It should be this commit 13d6faac2f62b237d0ef65ce09ea6a59543448d3

https://github.com/oda-hub/oda_api/blob/13d6faac2f62b237d0ef65ce09ea6a59543448d3/doc/source/user_guide/AsynchronousAPI.ipynb

volodymyrss commented 3 years ago 
ImpossibleScWSelection('pick_size 0 <= 0'))]
WARNING:ddaclient:worker >> 1629920692.5011725 top 8417/140296916608832 will raise ('TimeDirectionScWList', ImpossibleScWSelection('pick_size 0 <= 0'))
WARNING:ddaclient:worker >> Traceback (most recent call last):
WARNING:ddaclient:worker >>   File "/pyenv/versions/3.8.2/bin/rundda.py", line 8, in <module>
WARNING:ddaclient:worker >>     sys.exit(main())
WARNING:ddaclient:worker >>   File "/pyenv/versions/3.8.2/lib/python3.8/site-packages/dataanalysis/rundda.py", line 276, in main
WARNING:ddaclient:worker >>     A.raise_stored_exceptions()
WARNING:ddaclient:worker >>   File "/pyenv/versions/3.8.2/lib/python3.8/site-packages/dataanalysis/core.py", line 1544, in raise_stored_exceptions
WARNING:ddaclient:worker >>     raise to_raise
WARNING:ddaclient:worker >> dataanalysis.core.AnalysisException: ('TimeDirectionScWList', ImpossibleScWSelection('pick_size 0 <= 0'))
ERROR:ddaclient:something failed in query: no reasonable response!, 8 / 10 attempts left
Traceback (most recent call last):
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/ddaclient.py", line 496, in _query
    return DDAproduct(response_json, self.ddcache_root_local, self)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/ddaclient.py", line 168, in __init__
    self.interpret_dda_worker_response(dda_worker_response)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/ddaclient.py", line 212, in interpret_dda_worker_response
    raise WorkerException(
ddaclient.WorkerException: data is None, the analysis failed unexcplicably

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/ddaclient.py", line 433, in query
    return self._query(target, modules, assume, inject, prompt_delegate, callback)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/ddaclient.py", line 511, in _query
    raise WorkerException("no reasonable response!", content=response.content,
ddaclient.WorkerException: no reasonable response!
volodymyrss commented 3 years ago

specific, non-recognized failure occurred when query by range found no scws. I would now fail correctly.

Note that currently set default jemx values lead this this failure, so please feel free to look The values seem good but are still not fully set

ferrigno commented 3 years ago

As I did not set correct coordinates for the Crab, the query by range could indeed fail.

I do not know if what you posted is the new error message. It is not so clear, but better than timeout.

volodymyrss commented 3 years ago

As I did not set correct coordinates for the Crab, the query by range could indeed fail.

I do not know if what you posted is the new error message. It is not so clear, but better than timeout.

I did not add anything, I just recovered what is assumed to be normal behavior at this time. We might correct it next. There is an issue for that.