search

ITISFoundation / osparc-simcore

🐼 osparc-simcore simulation framework
https://osparc.io
MIT License
46 stars 27 forks source link

distributed.protocol.pickle Fails to deserialize ServiceRunError #2876

Closed pcrespov closed 1 year ago

pcrespov commented 2 years ago 
distributed.protocol.pickle - INFO - Failed to deserialize b'\x80\x05\x95I\x0c\x00\x00\x00\x00\x00\x00\x8c\x16tblib.pickling_support\x94\x8c\x12unpickle_exception\x94\x93\x94(\x8c9simcore_service_dask_sidecar.computational_sidecar.errors\x94\x8c\x0fServiceRunError\x94\x93\x94X\xb6\x05\x00\x00The service simcore/services/comp/isolve:2.1.5 running in container f4e0f470a5c204c92f1aedbf7c9819e9fd8add76efc407bed13a9a0e64a1c4f3 failed with exit code 137\nlast logs: ["Elapsed time for \'Computing number of non-zeros\' was 00:00:00 wall clock time.\\n", \'Assembling matrix\\n\', \'[PROGRESS]: 14% [ 3277471 / 21849828 ] Assembling matrix\\n\', \'[PROGRESS]: 19% [ 4369962 / 21849828 ] Assembling matrix\\n\', \'[PROGRESS]: 24% [ 5462453 / 21849828 ] Assembling matrix\\n\', \'[PROGRESS]: 28% [ 6336446 / 21849828 ] Assembling matrix\\n\', \'[PROGRESS]: 32% [ 7210439 / 21849828 ] Assembling matrix\\n\', \'[PROGRESS]: 36% [ 8084432 / 21849828 ] Assembling matrix\\n\', \'[PROGRESS]: 39% [ 8739927 / 21849828 ] Assembling matrix\\n\', \'[PROGRESS]: 42% [ 9395422 / 21849828 ] Assembling matrix\\n\', \'[PROGRESS]: 45% [ 10050917 / 21849828 ] Assembling matrix\\n\', \'[PROGRESS]: 48% [ 10706412 / 21849828 ] Assembling matrix\\n\', \'[PROGRESS]: 51% [ 11361907 / 21849828 ] Assembling matrix\\n\', \'[PROGRESS]: 54% [ 12017402 / 21849828 ] Assembling matrix\\n\', \'[PROGRESS]: 57% [ 12672897 / 21849828 ] Assembling matrix\\n\', \'[PROGRESS]: 60% [ 13328392 / 21849828 ] Assembling matrix\\n\', \'[PROGRESS]: 62% [ 13765389 / 21849828 ] Assembling matrix\\n\', \'[PROGRESS]: 64% [ 14202386 / 21849828 ] Assembling matrix\\n\', \'[PROGRESS]: 66% [ 14639383 / 21849828 ] Assembling matrix\\n\', \'  /home/smu/service.cli/execute.sh: line 61:    73 Killed                  /home/smu/services/iSolve/iSolve input.h5\\n\']\x94\x85\x94Nh\x00\x8c\x12unpickle_traceback\x94\x93\x94\x8c\x05tblib\x94\x8c\x05Frame\x94\x93\x94)\x81\x94}\x94(\x8c\x08f_locals\x94}\x94\x8c\tf_globals\x94}\x94(\x8c\x08__name__\x94\x8c\x12distributed.worker\x94\x8c\x08__file__\x94\x8cA/home/scu/.venv/lib/python3.8/site-packages/distributed/worker.py\x94u\x8c\x06f_code\x94h\n\x8c\x04Code\x94\x93\x94)\x81\x94}\x94(\x8c\x0bco_filename\x94h\x16\x8c\x07co_name\x94\x8c\x15apply_function_simple\x94\x8c\x0bco_argcount\x94K\x00\x8c\x11co_kwonlyargcount\x94K\x00\x8c\x0bco_varnames\x94)\x8c\nco_nlocals\x94K\x00\x8c\x0cco_stacksize\x94K\x00\x8c\x08co_flags\x94K@\x8c\x0eco_firstlineno\x94K\x00ub\x8c\x08f_lineno\x94MO\x11ubMM\x11h\n\x8c\tTraceback\x94\x93\x94)\x81\x94}\x94(\x8c\x08tb_frame\x94h\x0c)\x81\x94}\x94(h\x0f}\x94h\x11}\x94(\x8c\x08__name__\x94\x8c/simcore_service_director_v2.modules.dask_client\x94\x8c\x08__file__\x94\x8c^/home/scu/.venv/lib/python3.8/site-packages/simcore_service_director_v2/modules/dask_client.py\x94uh\x17h\x19)\x81\x94}\x94(h\x1ch3h\x1d\x8c\x11_comp_sidecar_fct\x94h\x1fK\x00h K\x00h!)h"K\x00h#K\x00h$K@h%K\x00ubh&M+\x01ub\x8c\ttb_lineno\x94M+\x01\x8c\x07tb_next\x94h()\x81\x94}\x94(h+h\x0c)\x81\x94}\x94(h\x0f}\x94h\x11}\x94(h\x13\x8c"simcore_service_dask_sidecar.tasks\x94h\x15\x8cQ/home/scu/.venv/lib/python3.8/site-packages/simcore_service_dask_sidecar/tasks.py\x94uh\x17h\x19)\x81\x94}\x94(h\x1ch@h\x1d\x8c\x19run_computational_sidecar\x94h\x1fK\x00h K\x00h!)h"K\x00h#K\x00h$K@h%K\x00ubh&K\xa0ubh7K\xa0h8h()\x81\x94}\x94(h+h\x0c)\x81\x94}\x94(h\x0f}\x94h\x11}\x94(h\x13\x8c\x13asyncio.base_events\x94h\x15\x8c//usr/local/lib/python3.8/asyncio/base_events.py\x94uh\x17h\x19)\x81\x94}\x94(h\x1chKh\x1d\x8c\x12run_until_complete\x94h\x1fK\x00h K\x00h!)h"K\x00h#K\x00h$K@h%K\x00ubh&Mh\x02ubh7Mh\x02h8h()\x81\x94}\x94(h+h\x0c)\x81\x94}\x94(h\x0f}\x94h\x11}\x94(h\x13h?h\x15h@uh\x17h\x19)\x81\x94}\x94(h\x1ch@h\x1d\x8c _run_computational_sidecar_async\x94h\x1fK\x00h K\x00h!)h"K\x00h#K\x00h$K@h%K\x00ubh&K\x89ubh7Kzh8h()\x81\x94}\x94(h+h\x0c)\x81\x94}\x94(h\x0f}\x94h\x11}\x94(h\x13\x8c7simcore_service_dask_sidecar.computational_sidecar.core\x94h\x15\x8cf/home/scu/.venv/lib/python3.8/site-packages/simcore_service_dask_sidecar/computational_sidecar/core.py\x94uh\x17h\x19)\x81\x94}\x94(h\x1ch_h\x1d\x8c\x03run\x94h\x1fK\x00h K\x00h!)h"K\x00h#K\x00h$K@h%K\x00ubh&K\xf1ubh7K\xe3ububububub\x87\x94R\x94t\x94R\x94.'
Traceback (most recent call last):
  File "/home/scu/.venv/lib/python3.8/site-packages/distributed/worker.py", line 4429, in apply_function_simple
    result = function(*args, **kwargs)
  File "/home/scu/.venv/lib/python3.8/site-packages/simcore_service_director_v2/modules/dask_client.py", line 299, in _comp_sidecar_fct
  File "/home/scu/.venv/lib/python3.8/site-packages/simcore_service_dask_sidecar/tasks.py", line 160, in run_computational_sidecar
    result = asyncio.get_event_loop().run_until_complete(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/scu/.venv/lib/python3.8/site-packages/simcore_service_dask_sidecar/tasks.py", line 122, in _run_computational_sidecar_async
    output_data = await sidecar.run(command=command)
  File "/home/scu/.venv/lib/python3.8/site-packages/simcore_service_dask_sidecar/computational_sidecar/core.py", line 227, in run
    raise ServiceRunError(
simcore_service_dask_sidecar.computational_sidecar.errors.ServiceRunError: The service simcore/services/comp/isolve:2.1.5 running in container f4e0f470a5c204c92f1aedbf7c9819e9fd8add76efc407bed13a9a0e64a1c4f3 failed with exit code 137
last logs: ["Elapsed time for 'Computing number of non-zeros' was 00:00:00 wall clock time.\n", 'Assembling matrix\n', '[PROGRESS]: 14% [ 3277471 / 21849828 ] Assembling matrix\n', '[PROGRESS]: 19% [ 4369962 / 21849828 ] Assembling matrix\n', '[PROGRESS]: 24% [ 5462453 / 21849828 ] Assembling matrix\n', '[PROGRESS]: 28% [ 6336446 / 21849828 ] Assembling matrix\n', '[PROGRESS]: 32% [ 7210439 / 21849828 ] Assembling matrix\n', '[PROGRESS]: 36% [ 8084432 / 21849828 ] Assembling matrix\n', '[PROGRESS]: 39% [ 8739927 / 21849828 ] Assembling matrix\n', '[PROGRESS]: 42% [ 9395422 / 21849828 ] Assembling matrix\n', '[PROGRESS]: 45% [ 10050917 / 21849828 ] Assembling matrix\n', '[PROGRESS]: 48% [ 10706412 / 21849828 ] Assembling matrix\n', '[PROGRESS]: 51% [ 11361907 / 21849828 ] Assembling matrix\n', '[PROGRESS]: 54% [ 12017402 / 21849828 ] Assembling matrix\n', '[PROGRESS]: 57% [ 12672897 / 21849828 ] Assembling matrix\n', '[PROGRESS]: 60% [ 13328392 / 21849828 ] Assembling matrix\n', '[PROGRESS]: 62% [ 13765389 / 21849828 ] Assembling matrix\n', '[PROGRESS]: 64% [ 14202386 / 21849828 ] Assembling matrix\n', '[PROGRESS]: 66% [ 14639383 / 21849828 ] Assembling matrix\n', '  /home/smu/service.cli/execute.sh: line 61:    73 Killed                  /home/smu/services/iSolve/iSolve input.h5\n']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/scu/.venv/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 75, in loads
    return pickle.loads(x)
  File "/home/scu/.venv/lib/python3.8/site-packages/tblib/pickling_support.py", line 26, in unpickle_exception
    inst = func(*args)
TypeError: __init__() missing 4 required positional arguments: 'service_version', 'container_id', 'exit_code', and 'service_logs'
distributed.worker - WARNING - Compute Failed
Function:  _comp_sidecar_fct
args:      ()
kwargs:    {'docker_auth': DockerBasicAuth(server_address='registry.osparc.speag.com', username='admin', password=SecretStr('**********')), 'service_key': 'simcore/services/comp/isolve', 'service_version': '2.1.5', 'input_data': TaskInputData(__root__={'input_1': FileUrl(url=AnyUrl('https://storage.osparc.speag.com/production-simcore/api/6e70f479-7452-3da0-9141-328ec778e0e5/3b4a6dad-e2d3-4ccb-888e-b42e8e4ddf2d_Input.h5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=gfhfgh765gjtyjtj/20220306/us-east-1/s3/aws4_request&X-Amz-Date=20220306T121657Z&X-Amz-Expires=259200&X-Amz-SignedHeaders=host&X-Amz-Signature=a422459ac53feab9f81b81e77b08a582714a42f850863f9655947c9cfe1b8373', scheme='https', host='storage.osparc.speag.com', tld='com', host_type='domain', path='/production-simcore/api/6e70f479-7452-3da0-9141-328ec778e0e5/3b4a6dad-e2d3-4ccb-888e-b42e8e4ddf2d_Input.h5', query='X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=gfhfgh765gjtyjtj/20220306/us-east-1/s3/aws4_request&X-Amz-Date=20220306T12165
Exception: 'ServiceRunError(\'The service simcore/services/comp/isolve:2.1.5 running in container f4e0f470a5c204c92f1aedbf7c9819e9fd8add76efc407bed13a9a0e64a1c4f3 failed with exit code 137\\nlast logs: ["Elapsed time for \\\'Computing number of non-zeros\\\' was 00:00:00 wall clock time.\\\\n", \\\'Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 14% [ 3277471 / 21849828 ] Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 19% [ 4369962 / 21849828 ] Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 24% [ 5462453 / 21849828 ] Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 28% [ 6336446 / 21849828 ] Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 32% [ 7210439 / 21849828 ] Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 36% [ 8084432 / 21849828 ] Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 39% [ 8739927 / 21849828 ] Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 42% [ 9395422 / 21849828 ] Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 45% [ 10050917 / 21849828 ] Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 48% [ 10706412 / 21849828 ] Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 51% [ 11361907 / 21849828 ] Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 54% [ 12017402 / 21849828 ] Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 57% [ 12672897 / 21849828 ] Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 60% [ 13328392 / 21849828 ] Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 62% [ 13765389 / 21849828 ] Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 64% [ 14202386 / 21849828 ] Assembling matrix\\\\n\\\', \\\'[PROGRESS]: 66% [ 14639383 / 21849828 ] Assembling matrix\\\\n\\\', \\\'  /home/smu/service.cli/execute.sh: line 61:    73 Killed                  /home/smu/services/iSolve/iSolve input.h5\\\\n\\\']\')'
sanderegg commented 1 year ago

this was fixed. ServiceRuntimeError is now accessible from dv-2