Instantiating a portal template anonymously fails with cryptic "copying failed" error

Is there an existing issue for this?

[X] I have searched the existing issues

Which deploy/s?

development (master)

Current Behavior

I open a portal link (such as /study/1d7b331c-43bd-11ed-807f-02420a1875ec on master on-prem). I open a second tab (or alternatively, in the same tab enter the full portal link again), I open the portal link again. The second tab fails with

Expected Behavior

It can still fail, but with a more descriptive error such as: "You can only open one template per hour as an anonymous user"

Steps To Reproduce

Anything else?

FYI: I can reproduce this on osparc-master.speag.com, as well in anonymous mode, but I didnt test any other deployments or products. In the containers, the following error is logged:

Failed while copying project '[portal] CC Rabbit' to 'vjcpajlrvc@guest-at-osparc.io' [OEC:140426840347776]
Traceback (most recent call last):
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_webserver/studies_dispatcher/_studies_access.py", line 157, in copy_study_to_account
    product_name = await db.get_project_product(template_project["uuid"])
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_webserver/projects/db.py", line 882, in get_project_product
    raise ProjectNotFoundError(project_uuid=project_uuid)
simcore_service_webserver.projects.exceptions.ProjectNotFoundError: Project with uuid '1d7b331c-43bd-11ed-807f-02420a1875ec' not found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_webserver/studies_dispatcher/_studies_access.py", line 350, in get_redirection_to_study_page
    copied_project_id = await copy_study_to_account(request, template_project, user)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_webserver/studies_dispatcher/_studies_access.py", line 190, in copy_study_to_account
    await db.insert_project(
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_webserver/projects/db.py", line 300, in insert_project
    inserted_project = await self._insert_project_in_db(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_webserver/projects/db.py", line 158, in _insert_project_in_db
    async for attempt in AsyncRetrying(retry=retry_if_exception_type(TryAgain)):
  File "/home/scu/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 166, in __anext__
    do = await self.iter(retry_state=self._retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_webserver/projects/db.py", line 183, in _insert_project_in_db
    _reraise_if_not_unique_uuid_error(err)
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_webserver/projects/db.py", line 154, in _reraise_if_not_unique_uuid_error
    raise err
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_webserver/projects/db.py", line 165, in _insert_project_in_db
    result: ResultProxy = await conn.execute(
                          ^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/aiopg/sa/connection.py", line 168, in _execute
    await cursor.execute(str(compiled), post_processed_params[0])
  File "/home/scu/.venv/lib/python3.11/site-packages/opentelemetry/instrumentation/aiopg/aiopg_integration.py", line 135, in execute
    result = await _traced_cursor.traced_execution(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/opentelemetry/instrumentation/aiopg/aiopg_integration.py", line 122, in traced_execution
    return await query_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/aiopg/connection.py", line 426, in execute
    await self._conn._poll(waiter, timeout)
  File "/home/scu/.venv/lib/python3.11/site-packages/aiopg/connection.py", line 881, in _poll
    await asyncio.wait_for(self._waiter, timeout)
  File "/usr/local/lib/python3.11/asyncio/tasks.py", line 489, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/aiopg/connection.py", line 788, in _ready
    state = self._conn.poll()
            ^^^^^^^^^^^^^^^^^
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "projects_uuid_key"
DETAIL:  Key (uuid)=(c24b7d46-77d4-5ce5-926f-0ea25d5525a5) already exists.

(Link: https://monitoring.osparc-master.speag.com/graylog/messages/graylog_375/e6de4632-7fcb-11ef-96d9-0242c0a80208 )

ITISFoundation / osparc-simcore