conda-incubator / conda-store

Data science environments, for collaboration. ✨
https://conda.store
BSD 3-Clause "New" or "Revised" License
143 stars 46 forks source link

[BUG] - IntegrityError upon startup #852

Open peytondmurray opened 2 months ago

peytondmurray commented 2 months ago

Describe the bug

Whenever I start the conda-store services with docker compose up --build, I get integrity errors:

... <more startup stuff here>

conda-store-worker-1  | [2024-07-19 00:02:40,346: WARNING/ForkPoolWorker-4] [CondaStoreWorker] building conda_prefix=/var/lib/conda-store/filesystem/7ff0f78a-1721347282-1-python-flask-env took 77.196 [s]
conda-store-worker-1  | [2024-07-19 00:02:41,850: INFO/MainProcess] Task task_build_conda_env_export[build-1-conda-env-export] received
conda-store-worker-1  | [2024-07-19 00:02:41,852: INFO/MainProcess] Task task_build_conda_pack[build-1-conda-pack] received
conda-store-worker-1  | [2024-07-19 00:02:41,853: INFO/MainProcess] Task task_build_conda_docker[build-1-docker] received
conda-store-worker-1  | [2024-07-19 00:02:41,855: INFO/MainProcess] Task task_build_constructor_installer[build-1-constructor-installer] received
conda-store-worker-1  | [2024-07-19 00:02:41,855: INFO/ForkPoolWorker-4] Task task_build_conda_environment[build-1-environment] succeeded in 78.7562527419941s: None
conda-store-worker-1  | [2024-07-19 00:02:41,867: WARNING/ForkPoolWorker-4] /opt/conda-store-server/conda_store_server/_internal/build.py:415: UserWarning: Generating Docker images is currently not supported, see https://github.com/conda-incubator/conda-store/issues/666
conda-store-worker-1  |   warnings.warn(
conda-store-worker-1  | 
conda-store-worker-1  | [2024-07-19 00:02:41,868: INFO/ForkPoolWorker-4] Task task_build_conda_docker[build-1-docker] succeeded in 0.012042938004015014s: None
conda-store-worker-1  | [2024-07-19 00:02:41,899: WARNING/ForkPoolWorker-4] /opt/conda-store-server/conda_store_server/_internal/action/generate_constructor_installer.py:48: UserWarning: Installer generation requires constructor: https://github.com/conda/constructor
conda-store-worker-1  |   warnings.warn(
conda-store-worker-1  | 
conda-store-worker-1  | [2024-07-19 00:02:41,900: WARNING/ForkPoolWorker-4] [CondaStoreWorker] creating installer for conda environment=/var/lib/conda-store/filesystem/7ff0f78a-1721347282-1-python-flask-env took 0.018 [s]
conda-store-worker-1  | [2024-07-19 00:02:41,901: INFO/ForkPoolWorker-4] Task task_build_constructor_installer[build-1-constructor-installer] succeeded in 0.03257362500880845s: None
conda-store-worker-1  | [2024-07-19 00:02:43,334: INFO/ForkPoolWorker-2] Task task_build_conda_env_export[build-1-conda-env-export] succeeded in 1.4822059069992974s: None
conda-store-worker-1  | [2024-07-19 00:02:47,599: WARNING/ForkPoolWorker-3] [CondaStoreWorker] packaging archive of conda environment=/var/lib/conda-store/filesystem/7ff0f78a-1721347282-1-python-flask-env took 5.731 [s]
conda-store-worker-1  | [2024-07-19 00:02:47,604: INFO/ForkPoolWorker-3] Task task_build_conda_pack[build-1-conda-pack] succeeded in 5.751906952005811s: None
conda-store-worker-1  | [2024-07-19 00:02:47,605: INFO/MainProcess] Task task_update_storage_metrics[build-1] received
conda-store-worker-1  | [2024-07-19 00:02:47,628: INFO/ForkPoolWorker-2] Task task_update_storage_metrics[build-1] succeeded in 0.02287509001325816s: None
conda-store-server-1  | INFO:     127.0.0.1:38788 - "GET /conda-store/api/v1/ HTTP/1.1" 200 OK
conda-store-server-1  | INFO:     127.0.0.1:58982 - "GET /conda-store/api/v1/ HTTP/1.1" 200 OK
conda-store-worker-1  | [2024-07-19 00:03:01,985: INFO/ForkPoolWorker-5] packages to insert : 64960 
postgres-1            | 2024-07-19 00:03:01.989 UTC [81] ERROR:  duplicate key value violates unique constraint "_conda_package_uc"
postgres-1            | 2024-07-19 00:03:01.989 UTC [81] DETAIL:  Key (channel_id, name, version)=(2, _libgcc_mutex, 0.1) already exists.
postgres-1            | 2024-07-19 00:03:01.989 UTC [81] STATEMENT:  INSERT INTO conda_package (channel_id, license, name, version, summary) VALUES (2, 'None', '_libgcc_mutex', '0.1', 'Mutex for libgcc and libgcc-ng') RETURNING conda_package.id
conda-store-worker-1  | [2024-07-19 00:03:01,991: WARNING/ForkPoolWorker-5] (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "_conda_package_uc"
conda-store-worker-1  | DETAIL:  Key (channel_id, name, version)=(2, _libgcc_mutex, 0.1) already exists.
conda-store-worker-1  | 
conda-store-worker-1  | [SQL: INSERT INTO conda_package (channel_id, license, name, version, summary) VALUES (%(channel_id)s, %(license)s, %(name)s, %(version)s, %(summary)s) RETURNING conda_package.id]
conda-store-worker-1  | [parameters: {'channel_id': 2, 'license': 'None', 'name': '_libgcc_mutex', 'version': '0.1', 'summary': 'Mutex for libgcc and libgcc-ng'}]
conda-store-worker-1  | (Background on this error at: https://sqlalche.me/e/14/gkpj)
conda-store-worker-1  | [2024-07-19 00:03:01,996: ERROR/ForkPoolWorker-5] Task task_update_conda_channel[b07f7b56-0f96-43fb-a27b-65afaf22b1c6] raised unexpected: IntegrityError('(psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "_conda_package_uc"\nDETAIL:  Key (channel_id, name, version)=(2, _libgcc_mutex, 0.1) already exists.\n')
conda-store-worker-1  | Traceback (most recent call last):
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
conda-store-worker-1  |     self.dialect.do_execute(
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
conda-store-worker-1  |     cursor.execute(statement, parameters)
conda-store-worker-1  | psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "_conda_package_uc"
conda-store-worker-1  | DETAIL:  Key (channel_id, name, version)=(2, _libgcc_mutex, 0.1) already exists.
conda-store-worker-1  | 
conda-store-worker-1  | 
conda-store-worker-1  | The above exception was the direct cause of the following exception:
conda-store-worker-1  | 
conda-store-worker-1  | Traceback (most recent call last):
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/celery/app/trace.py", line 453, in trace_task
conda-store-worker-1  |     R = retval = fun(*args, **kwargs)
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/celery/app/trace.py", line 736, in __protected_call__
conda-store-worker-1  |     return self.run(*args, **kwargs)
conda-store-worker-1  |   File "/opt/conda-store-server/conda_store_server/_internal/worker/tasks.py", line 188, in task_update_conda_channel
conda-store-worker-1  |     channel.update_packages(db, subdirs=settings.conda_platforms)
conda-store-worker-1  |   File "/opt/conda-store-server/conda_store_server/_internal/orm.py", line 569, in update_packages
conda-store-worker-1  |     raise e
conda-store-worker-1  |   File "/opt/conda-store-server/conda_store_server/_internal/orm.py", line 564, in update_packages
conda-store-worker-1  |     db.bulk_insert_mappings(CondaPackage, packages.values())
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3810, in bulk_insert_mappings
conda-store-worker-1  |     self._bulk_save_mappings(
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3912, in _bulk_save_mappings
conda-store-worker-1  |     with util.safe_reraise():
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
conda-store-worker-1  |     compat.raise_(
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
conda-store-worker-1  |     raise exception
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3901, in _bulk_save_mappings
conda-store-worker-1  |     persistence._bulk_insert(
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 107, in _bulk_insert
conda-store-worker-1  |     _emit_insert_statements(
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
conda-store-worker-1  |     c = connection._execute_20(
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
conda-store-worker-1  |     return meth(self, args_10style, kwargs_10style, execution_options)
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
conda-store-worker-1  |     return connection._execute_clauseelement(
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
conda-store-worker-1  |     ret = self._execute_context(
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
conda-store-worker-1  |     self._handle_dbapi_exception(
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
conda-store-worker-1  |     util.raise_(
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
conda-store-worker-1  |     raise exception
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
conda-store-worker-1  |     self.dialect.do_execute(
conda-store-worker-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
conda-store-worker-1  |     cursor.execute(statement, parameters)
conda-store-worker-1  | sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "_conda_package_uc"
conda-store-worker-1  | DETAIL:  Key (channel_id, name, version)=(2, _libgcc_mutex, 0.1) already exists.
conda-store-worker-1  | 
conda-store-worker-1  | [SQL: INSERT INTO conda_package (channel_id, license, name, version, summary) VALUES (%(channel_id)s, %(license)s, %(name)s, %(version)s, %(summary)s) RETURNING conda_package.id]
conda-store-worker-1  | [parameters: {'channel_id': 2, 'license': 'None', 'name': '_libgcc_mutex', 'version': '0.1', 'summary': 'Mutex for libgcc and libgcc-ng'}]
conda-store-worker-1  | (Background on this error at: https://sqlalche.me/e/14/gkpj)

Even with fresh containers, this error always appears. It doesn't seem to affect anything, but getting error messages immediately upon starting up isn't helpful.

Expected behavior

No error appears.

How to Reproduce the problem?

docker compose up --build. I'm running Linux 6.6.39-1-lts x86_64.

Here's the output of pip list ``` Package Version ----------------------------- -------- accessible-pygments 0.0.5 aiohttp 3.9.5 aiosignal 1.3.1 alabaster 0.7.16 alembic 1.13.2 amqp 5.2.0 anyio 4.4.0 appdirs 1.4.4 argon2-cffi 23.1.0 argon2-cffi-bindings 21.2.0 asttokens 2.4.1 async-timeout 4.0.3 attrs 23.2.0 Babel 2.14.0 backports.tarfile 1.0.0 bcrypt 4.1.3 beautifulsoup4 4.12.3 billiard 4.2.0 black 22.3.0 boltons 24.0.0 boto3 1.34.143 botocore 1.34.143 Brotli 1.1.0 CacheControl 0.14.0 cachy 0.3.0 cattrs 23.2.3 celery 5.3.6 certifi 2024.7.4 cffi 1.16.0 charset-normalizer 3.3.2 click 8.1.7 click-default-group 1.2.4 click-didyoumean 0.3.1 click-plugins 1.1.1 click-repl 0.3.0 clikit 0.6.2 colorama 0.4.6 conda 23.5.2 conda-docker 0.1.2 conda_lock 2.5.7 conda-pack 0.8.0 conda-package-handling 2.3.0 conda_package_streaming 0.10.0 constructor 3.8.0 crashtest 0.4.1 cryptography 42.0.8 dataclasses 0.8 decorator 5.1.1 distlib 0.3.8 dnspython 2.6.1 docker 6.1.3 docstring-to-markdown 0.15 docutils 0.17.1 editables 0.5 email_validator 2.2.0 ensureconda 1.4.4 exceptiongroup 1.2.0 executing 2.0.1 fancycompleter 0.9.1 fastapi 0.111.0 fastapi-cli 0.0.4 filelock 3.15.4 flake8 7.1.0 flower 2.0.1 frozenlist 1.4.1 gitdb 4.0.11 GitPython 3.1.43 greenlet 3.0.3 h11 0.14.0 h2 4.1.0 hatch 1.12.0 hatchling 1.25.0 hpack 4.0.0 html5lib 1.1 httpcore 1.0.5 httpx 0.27.0 humanize 4.10.0 hyperframe 6.0.1 hyperlink 21.0.0 idna 3.7 imagesize 1.4.1 importlib_metadata 8.0.0 importlib_resources 6.4.0 iniconfig 2.0.0 ipython 8.26.0 itsdangerous 2.2.0 jaraco.classes 3.4.0 jaraco.context 5.3.0 jaraco.functools 4.0.0 jedi 0.19.1 jeepney 0.8.0 Jinja2 3.1.4 jmespath 1.0.1 jsonpatch 1.33 jsonpointer 3.0.0 keyring 25.2.1 kombu 5.3.7 libmambapy 1.5.1 linkify-it-py 2.0.3 lsprotocol 2023.0.1 Mako 1.3.5 mamba 1.5.1 markdown-it-py 2.2.0 MarkupSafe 2.1.5 matplotlib-inline 0.1.7 mccabe 0.7.0 mdit-py-plugins 0.4.1 mdurl 0.1.2 memray 1.13.3 minio 7.2.7 more-itertools 10.3.0 msgpack 1.0.8 multidict 6.0.5 mypy 1.10.1 mypy-extensions 1.0.0 myst-parser 0.18.1 orjson 3.10.6 packaging 24.1 paramiko 3.4.0 parso 0.8.4 pastel 0.2.1 pathspec 0.12.1 pdbpp 0.10.3 pexpect 4.9.0 pillow 10.4.0 pip 24.0 pkginfo 1.11.1 platformdirs 4.2.2 playwright 1.45.0 pluggy 1.5.0 prometheus_client 0.20.0 prompt_toolkit 3.0.47 psycopg2 2.9.9 ptyprocess 0.7.0 pure-eval 0.2.2 pycodestyle 2.12.0 pycosat 0.6.6 pycparser 2.22 pycryptodome 3.20.0 pydantic 1.10.16 pydata-sphinx-theme 0.13.3 pyee 11.1.0 pyflakes 3.2.0 pygls 1.3.1 Pygments 2.18.0 PyJWT 2.8.0 pylev 1.4.0 PyMySQL 1.1.0 PyNaCl 1.5.0 pyOpenSSL 24.0.0 pyrepl 0.9.0 PySocks 1.7.1 pytest 8.2.2 pytest-base-url 2.1.0 pytest-celery 0.0.0a1 pytest-memray 1.6.0 pytest-mock 3.14.0 pytest-playwright 0.5.1 python-dateutil 2.9.0 python-docker 0.2.0 python-lsp-jsonrpc 1.1.2 python-lsp-server 1.11.0 python-multipart 0.0.9 python-slugify 8.0.4 pytz 2024.1 PyYAML 6.0.1 redis 5.0.7 requests 2.32.3 rich 13.7.1 ruamel.yaml 0.17.40 ruamel.yaml.clib 0.2.8 ruff 0.5.1 ruff-lsp 0.0.54 s3transfer 0.10.2 SecretStorage 3.3.3 setuptools 70.2.0 shellingham 1.5.4 six 1.16.0 smmap 5.0.0 sniffio 1.3.1 snowballstemmer 2.2.0 soupsieve 2.5 Sphinx 4.5.0 sphinx-copybutton 0.5.2 sphinx-panels 0.6.0 sphinxcontrib-applehelp 1.0.4 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 2.0.1 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.5 SQLAlchemy 1.4.46 stack-data 0.6.3 starlette 0.37.2 text-unidecode 1.3 textual 0.73.0 tomli 2.0.1 tomli_w 1.0.0 tomlkit 0.13.0 toolz 0.12.1 tornado 6.4.1 tqdm 4.66.4 traitlets 5.14.3 trove-classifiers 2024.7.2 typed-ast 1.5.5 typer 0.12.3 typer-slim 0.12.3 typing_extensions 4.12.2 tzdata 2024.1 uc-micro-py 1.0.3 ujson 5.10.0 urllib3 1.26.19 userpath 1.7.0 uvicorn 0.30.1 vine 5.1.0 virtualenv 20.26.3 wcwidth 0.2.13 webencodings 0.5.1 websocket-client 1.8.0 wheel 0.43.0 wmctrl 0.5 yarl 1.9.4 zipp 3.19.2 zstandard 0.22.0 ```

Here's the output of python -VV:

Python 3.10.0 | packaged by conda-forge | (default, Nov 20 2021, 02:24:10) [GCC 9.4.0]

trallard commented 1 month ago

Going by the trace it seems this originates within https://github.com/conda-incubator/conda-store/blob/24e15a1b8bcca2fdc536d0aa66d0a860b96cd6ca/conda-store-server/conda_store_server/_internal/worker/tasks.py#L103

but I am unsure why duplicated entries are being added (or try to) since this seems to be at configuration time