search

ITISFoundation / osparc-issues

🐼 issue-only repo for the osparc project
3 stars 5 forks source link

Duplicating studies where dynamic services with states are present fails #1049

Closed sanderegg closed 1 year ago

sanderegg commented 1 year ago

Long Story Short

As a user I duplicate studies containing dynamic services. I also sometimes want to publish templates.

At the moment this is impossible since the copy fails when copying the state files.

Expected Behavior

The study I have can be duplicated or published successfuly.

Actual behaviour

The copy fails with the webserver complaining such as in 

{
  "log_timestamp": "2023-08-04 12:59:38,285",
  "HOUR": "12",
  "gl2_remote_ip": "10.0.0.6",
  "gl2_remote_port": 41943,
  "source": "osparc-master-03",
  "gl2_source_input": "6493ffffff98c73e61cc1b2b",
  "image_name": "itisfoundation/webserver:master-github-latest@sha256:3ce1fecdd7b83aa99568241ecff56b61e9dc65a29eecf6111d571136a984615f",
  "YEAR": "2023",
  "log_service": "master-simcore_master_webserver",
  "gl2_source_node": "ec586e72-92e0-460d-b4b5-50328b13cf69",
  "MINUTE": "59",
  "tag": "49ea57faba56",
  "timestamp": "2023-08-04T12:59:38.287Z",
  "gl2_accounted_message_size": 16519,
  "MONTHNUM": "08",
  "level": 3,
  "log_msg": "Unexpected server error \"<class 'aiohttp.web_exceptions.HTTPInternalServerError'>\" from access: 10.193.35.46 \"GET /v0/tasks/POST%20%2Fv0%2Fprojects%3Ffrom_study%3De926966c-32c5-11ee-b22c-02420ac1231f%26as_template%3Dtrue%26copy_data%3Dtrue.58428c40-0ba8-4aed-8a6a-66946eaf1f25/result\" done in 0.00 secs. Responding with status 500\nTraceback (most recent call last):\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/rest_middlewares.py\", line 76, in _middleware_handler\n    return await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/rest_middlewares.py\", line 166, in _middleware_handler\n    resp = await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/products/_middlewares.py\", line 70, in discover_product_middleware\n    return await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/meta_modeling/_projects.py\", line 78, in projects_redirection_middleware\n    response = await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/long_running_tasks/_error_handlers.py\", line 14, in base_long_running_error_handler\n    return await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/login/decorators.py\", line 45, in _wrapper\n    return await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/long_running_tasks.py\", line 32, in _test_task_context_decorator\n    return await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/long_running_tasks/_routes.py\", line 67, in get_task_result\n    task_result = tasks_manager.get_task_result(\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/long_running_tasks/_task.py\", line 229, in get_task_result\n    return tracked_task.task.result()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/long_running_tasks/_task.py\", line 417, in _progress_task\n    return await handler(progress, **task_kwargs)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/projects/_crud_api_create.py\", line 266, in create_project\n    await copy_file_coro\n  File \"/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/projects/_crud_api_create.py\", line 174, in _copy_files_from_source_project\n    await long_running_task.result()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/long_running_tasks/client.py\", line 116, in result\n    return await self._result\n  File \"/home/scu/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py\", line 86, in async_wrapped\n    return await fn(*args, **kwargs)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py\", line 48, in __call__\n    do = self.iter(retry_state=retry_state)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/tenacity/__init__.py\", line 349, in iter\n    return fut.result()\n  File \"/usr/local/lib/python3.10/concurrent/futures/_base.py\", line 451, in result\n    return self.__get_result()\n  File \"/usr/local/lib/python3.10/concurrent/futures/_base.py\", line 403, in __get_result\n    raise self._exception\n  File \"/home/scu/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py\", line 51, in __call__\n    result = await fn(*args, **kwargs)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/long_running_tasks/client.py\", line 88, in _task_result\n    response.raise_for_status()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiohttp/client_reqrep.py\", line 1005, in raise_for_status\n    raise ClientResponseError(\naiohttp.client_exceptions.ClientResponseError: 404, message='The file e926966c-32c5-11ee-b22c-02420ac1231f/47793924-92ed-466e-abab-7207433632e9/workspace  in master-simcore was not found', url=URL('http://master_storage:8080/v0/futures/POST%2520%252Fv0%252Fsimcore-s3%252Ffolders%253Fuser_id%253D3.fe707a8d-ddf0-4a7d-a62d-6d04e289dde8/result')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/monitoring.py\", line 183, in middleware_handler\n    resp = await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiozipkin/aiohttp_helpers.py\", line 157, in aiozipkin_middleware\n    resp = await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiohttp_session/__init__.py\", line 215, in factory\n    raise cast(web.HTTPException, response)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiohttp_session/__init__.py\", line 199, in factory\n    response = await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/rest_middlewares.py\", line 135, in _middleware_handler\n    _process_and_raise_unexpected_error(request, err)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/rest_middlewares.py\", line 64, in _process_and_raise_unexpected_error\n    raise resp\naiohttp.web_exceptions.HTTPInternalServerError: Unexpected Server error\nStack (most recent call last):\n  File \"/home/scu/.venv/bin/gunicorn\", line 8, in <module>\n    sys.exit(run())\n  File \"/home/scu/.venv/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py\", line 67, in run\n    WSGIApplication(\"%(prog)s [OPTIONS] [APP_MODULE]\").run()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/gunicorn/app/base.py\", line 231, in run\n    super().run()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/gunicorn/app/base.py\", line 72, in run\n    Arbiter(self).run()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/gunicorn/arbiter.py\", line 202, in run\n    self.manage_workers()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/gunicorn/arbiter.py\", line 551, in manage_workers\n    self.spawn_workers()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/gunicorn/arbiter.py\", line 622, in spawn_workers\n    self.spawn_worker()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/gunicorn/arbiter.py\", line 589, in spawn_worker\n    worker.init_process()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiohttp/worker.py\", line 51, in init_process\n    super().init_process()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/gunicorn/workers/base.py\", line 142, in init_process\n    self.run()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiohttp/worker.py\", line 57, in run\n    self.loop.run_until_complete(self._task)\n  File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 636, in run_until_complete\n    self.run_forever()\n  File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 603, in run_forever\n    self._run_once()\n  File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 1909, in _run_once\n    handle._run()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/monitor_slow_callbacks.py\", line 31, in instrumented\n    retval = _run(self)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiodebug/log_slow_callbacks.py\", line 27, in instrumented\n    return_value = _run(self)\n  File \"/usr/local/lib/python3.10/asyncio/events.py\", line 80, in _run\n    self._context.run(self._callback, *self._args)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiohttp/web_protocol.py\", line 433, in _handle_request\n    resp = await request_handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiohttp/web_app.py\", line 504, in _handle\n    resp = await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiohttp/web_middlewares.py\", line 117, in impl\n    return await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/monitoring.py\", line 227, in middleware_handler\n    log.error(",
  "created": "2023-08-03T14:53:52.341140366Z",
  "streams": [
    "000000000000000000000001"
  ],
  "log_level": "ERROR",
  "gl2_message_id": "01H70A3MXE08QEHCDHQZX80TG8",
  "message": "log_level=ERROR | log_timestamp=2023-08-04 12:59:38,285 | log_source=servicelib.aiohttp.monitoring:middleware_handler(227) | log_uid=None | log_msg=Unexpected server error \"<class 'aiohttp.web_exceptions.HTTPInternalServerError'>\" from access: 10.193.35.46 \"GET /v0/tasks/POST%20%2Fv0%2Fprojects%3Ffrom_study%3De926966c-32c5-11ee-b22c-02420ac1231f%26as_template%3Dtrue%26copy_data%3Dtrue.58428c40-0ba8-4aed-8a6a-66946eaf1f25/result\" done in 0.00 secs. Responding with status 500\nTraceback (most recent call last):\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/rest_middlewares.py\", line 76, in _middleware_handler\n    return await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/rest_middlewares.py\", line 166, in _middleware_handler\n    resp = await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/products/_middlewares.py\", line 70, in discover_product_middleware\n    return await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/meta_modeling/_projects.py\", line 78, in projects_redirection_middleware\n    response = await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/long_running_tasks/_error_handlers.py\", line 14, in base_long_running_error_handler\n    return await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/login/decorators.py\", line 45, in _wrapper\n    return await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/long_running_tasks.py\", line 32, in _test_task_context_decorator\n    return await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/long_running_tasks/_routes.py\", line 67, in get_task_result\n    task_result = tasks_manager.get_task_result(\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/long_running_tasks/_task.py\", line 229, in get_task_result\n    return tracked_task.task.result()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/long_running_tasks/_task.py\", line 417, in _progress_task\n    return await handler(progress, **task_kwargs)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/projects/_crud_api_create.py\", line 266, in create_project\n    await copy_file_coro\n  File \"/home/scu/.venv/lib/python3.10/site-packages/simcore_service_webserver/projects/_crud_api_create.py\", line 174, in _copy_files_from_source_project\n    await long_running_task.result()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/long_running_tasks/client.py\", line 116, in result\n    return await self._result\n  File \"/home/scu/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py\", line 86, in async_wrapped\n    return await fn(*args, **kwargs)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py\", line 48, in __call__\n    do = self.iter(retry_state=retry_state)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/tenacity/__init__.py\", line 349, in iter\n    return fut.result()\n  File \"/usr/local/lib/python3.10/concurrent/futures/_base.py\", line 451, in result\n    return self.__get_result()\n  File \"/usr/local/lib/python3.10/concurrent/futures/_base.py\", line 403, in __get_result\n    raise self._exception\n  File \"/home/scu/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py\", line 51, in __call__\n    result = await fn(*args, **kwargs)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/long_running_tasks/client.py\", line 88, in _task_result\n    response.raise_for_status()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiohttp/client_reqrep.py\", line 1005, in raise_for_status\n    raise ClientResponseError(\naiohttp.client_exceptions.ClientResponseError: 404, message='The file e926966c-32c5-11ee-b22c-02420ac1231f/47793924-92ed-466e-abab-7207433632e9/workspace  in master-simcore was not found', url=URL('http://master_storage:8080/v0/futures/POST%2520%252Fv0%252Fsimcore-s3%252Ffolders%253Fuser_id%253D3.fe707a8d-ddf0-4a7d-a62d-6d04e289dde8/result')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/monitoring.py\", line 183, in middleware_handler\n    resp = await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiozipkin/aiohttp_helpers.py\", line 157, in aiozipkin_middleware\n    resp = await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiohttp_session/__init__.py\", line 215, in factory\n    raise cast(web.HTTPException, response)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiohttp_session/__init__.py\", line 199, in factory\n    response = await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/rest_middlewares.py\", line 135, in _middleware_handler\n    _process_and_raise_unexpected_error(request, err)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/rest_middlewares.py\", line 64, in _process_and_raise_unexpected_error\n    raise resp\naiohttp.web_exceptions.HTTPInternalServerError: Unexpected Server error\nStack (most recent call last):\n  File \"/home/scu/.venv/bin/gunicorn\", line 8, in <module>\n    sys.exit(run())\n  File \"/home/scu/.venv/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py\", line 67, in run\n    WSGIApplication(\"%(prog)s [OPTIONS] [APP_MODULE]\").run()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/gunicorn/app/base.py\", line 231, in run\n    super().run()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/gunicorn/app/base.py\", line 72, in run\n    Arbiter(self).run()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/gunicorn/arbiter.py\", line 202, in run\n    self.manage_workers()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/gunicorn/arbiter.py\", line 551, in manage_workers\n    self.spawn_workers()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/gunicorn/arbiter.py\", line 622, in spawn_workers\n    self.spawn_worker()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/gunicorn/arbiter.py\", line 589, in spawn_worker\n    worker.init_process()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiohttp/worker.py\", line 51, in init_process\n    super().init_process()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/gunicorn/workers/base.py\", line 142, in init_process\n    self.run()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiohttp/worker.py\", line 57, in run\n    self.loop.run_until_complete(self._task)\n  File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 636, in run_until_complete\n    self.run_forever()\n  File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 603, in run_forever\n    self._run_once()\n  File \"/usr/local/lib/python3.10/asyncio/base_events.py\", line 1909, in _run_once\n    handle._run()\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/monitor_slow_callbacks.py\", line 31, in instrumented\n    retval = _run(self)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiodebug/log_slow_callbacks.py\", line 27, in instrumented\n    return_value = _run(self)\n  File \"/usr/local/lib/python3.10/asyncio/events.py\", line 80, in _run\n    self._context.run(self._callback, *self._args)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiohttp/web_protocol.py\", line 433, in _handle_request\n    resp = await request_handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiohttp/web_app.py\", line 504, in _handle\n    resp = await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/aiohttp/web_middlewares.py\", line 117, in impl\n    return await handler(request)\n  File \"/home/scu/.venv/lib/python3.10/site-packages/servicelib/aiohttp/monitoring.py\", line 227, in middleware_handler\n    log.error(",
  "command": "services/web/server/docker/entrypoint.sh services/web/server/docker/boot.sh",
  "container_name": "master-simcore_master_webserver.3.mz68uoc15cy3o0ljvl6tto148",
  "log_uid": "None",
  "log_source": "servicelib.aiohttp.monitoring:middleware_handler(227)",
  "SECOND": "38,285",
  "_id": "c4ddd5d2-32c6-11ee-9a52-0242ac130010",
  "image_id": "sha256:a4d35dcefc3951d04967fcfce706bc6e17662064a40f0407536d7f2533c5efb3",
  "container_id": "49ea57faba56aa0393c1c8e89a5bb99907e3c432d342c280ccf505d87e68710e",
  "MONTHDAY": "04"
}

Steps to reproduce

Additional context

elisabettai commented 1 year ago

I've just tested on master and duplication and publishing a template seems to work. I guess ANE's PR fixes this. @sanderegg, feel free to reopen if you see other problems.