search

conda-incubator / conda-store

Data science environments, for collaboration. ✨
https://conda.store
BSD 3-Clause "New" or "Revised" License
145 stars 50 forks source link

`library` envs share a namespace with `filesystem` envs and cannot have the same name #72

Closed jaimergp closed 3 years ago

jaimergp commented 3 years ago

With a fresh local deployment (no previous env created), creating an empty env with just python results in a second copy.

Env:

name: env-name-1
channels:
  - conda-forge
dependencies:
  - python

Relevant log lines:

conda-store-server_1  | [2021-06-24 14:47:53,120] {/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/werkzeug/_internal.py:225} INFO - 172.26.0.1 - - [24/Jun/2021 14:47:53] "GET /create/ HTTP/1.1" 200 -
conda-store-server_1  | [2021-06-24 14:48:29,630] {/opt/conda-store-server/conda_store_server/app.py:96} INFO - registering specification name=env-name-1 sha256=44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8
conda-store-server_1  | [2021-06-24 14:48:29,653] {/opt/conda-store-server/conda_store_server/app.py:102} INFO - scheduling specification for build name=env-name-1 sha256=44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8
conda-store-server_1  | [2021-06-24 14:48:29,672] {/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/werkzeug/_internal.py:225} INFO - 172.26.0.1 - - [24/Jun/2021 14:48:29] "POST /create/ HTTP/1.1" 302 -
conda-store-server_1  | [2021-06-24 14:48:29,836] {/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/werkzeug/_internal.py:225} INFO - 172.26.0.1 - - [24/Jun/2021 14:48:29] "GET / HTTP/1.1" 200 -

conda-store-build_1   | [2021-06-24 14:48:54,514] {/opt/conda-store-server/conda_store_server/conda.py:38} INFO - downloading channeldata url=https://repo.anaconda.com/pkgs/main/channeldata.json
conda-store-build_1   | [2021-06-24 14:48:54,900] {/opt/conda-store-server/conda_store_server/conda.py:46} INFO - downloading repodata url=https://repo.anaconda.com/pkgs/main/linux-64/repodata.json
conda-store-build_1   | [2021-06-24 14:48:57,214] {/opt/conda-store-server/conda_store_server/conda.py:46} INFO - downloading repodata url=https://repo.anaconda.com/pkgs/main/noarch/repodata.json
conda-store-build_1   | [2021-06-24 14:49:21,735] {/opt/conda-store-server/conda_store_server/build.py:111} INFO - polling interval set to 10 seconds
conda-store-build_1   | [2021-06-24 14:49:21,793] {/opt/conda-store-server/conda_store_server/app.py:96} INFO - registering specification name=env-name-1 sha256=b713d98b31554b3bfff940ae81df67f92762c36e34017507ff40bdc7371ac56e
conda-store-build_1   | [2021-06-24 14:49:21,815] {/opt/conda-store-server/conda_store_server/app.py:102} INFO - scheduling specification for build name=env-name-1 sha256=b713d98b31554b3bfff940ae81df67f92762c36e34017507ff40bdc7371ac56e
conda-store-build_1   | [2021-06-24 14:49:21,858] {/opt/conda-store-server/conda_store_server/app.py:96} INFO - registering specification name=env-name-2 sha256=45d17df515991ab4b0d9176c96cabe2a641da43fb8a8958b054c07f9c431cc70
conda-store-build_1   | [2021-06-24 14:49:21,866] {/opt/conda-store-server/conda_store_server/app.py:102} INFO - scheduling specification for build name=env-name-2 sha256=45d17df515991ab4b0d9176c96cabe2a641da43fb8a8958b054c07f9c431cc70
conda-store-build_1   | [2021-06-24 14:49:21,896] {/opt/conda-store-server/conda_store_server/build.py:119} INFO - number of queued conda builds 3
conda-store-build_1   | [2021-06-24 14:49:21,921] {/opt/conda-store-server/conda_store_server/build.py:129} INFO - number of schedulable conda builds 3
conda-store-build_1   | [2021-06-24 14:49:21,959] {/opt/conda-store-server/conda_store_server/build.py:157} INFO - building /data/store/44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8-env-name-1 symlinked to /data/envs/env-name-1
conda-store-build_1   | [2021-06-24 14:49:21,960] {/opt/conda-store-server/conda_store_server/build.py:159} INFO - previously unfinished build of /data/store/44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8-env-name-1 cleaning directory

conda-store-build_1   | [2021-06-24 14:51:33,015] {/opt/conda-store-server/conda_store_server/utils.py:42} INFO - building /data/store/44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8-env-name-1 took 131.052 [s]
conda-store-build_1   | [2021-06-24 14:51:33,041] {/opt/conda-store-server/conda_store_server/build.py:189} INFO - modifying permissions of /data/store/44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8-env-name-1 to permissions=775
conda-store-build_1   | [2021-06-24 14:51:45,527] {/opt/conda-store-server/conda_store_server/utils.py:42} INFO - chmod of /data/store/44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8-env-name-1 took 12.486 [s]
conda-store-build_1   | [2021-06-24 14:51:45,528] {/opt/conda-store-server/conda_store_server/build.py:200} INFO - modifying permissions of /data/store/44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8-env-name-1 to uid=1000 and gid=100
conda-store-build_1   | [2021-06-24 14:52:00,788] {/opt/conda-store-server/conda_store_server/utils.py:42} INFO - chown of /data/store/44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8-env-name-1 took 15.260 [s]
conda-store-build_1   | [2021-06-24 14:52:02,518] {/opt/conda-store-server/conda_store_server/build.py:238} INFO - packaging archive of conda environment=/data/store/44b8b0062e3f7c4ccb93eb2d662256bcb1e44bed01ea955602fcc798103be2c8-env-name-1
conda-store-build_1   | [2021-06-24 14:52:04,864] {/opt/conda-store-server/conda_store_server/build.py:214} ERROR - 
conda-store-build_1   | Files managed by conda were found to have been deleted/overwritten in the
conda-store-build_1   | following packages:
conda-store-build_1   | 
conda-store-build_1   | - ncurses 6.2:
conda-store-build_1   |     share/terminfo/2/2621A
conda-store-build_1   |     share/terminfo/E/Eterm
conda-store-build_1   |     share/terminfo/E/Eterm-color
conda-store-build_1   |     + 1034 others
conda-store-build_1   | 
conda-store-build_1   | This is usually due to `pip` uninstalling or clobbering conda managed files,
conda-store-build_1   | resulting in an inconsistent environment. Please check your environment for
conda-store-build_1   | conda/pip conflicts using `conda list`, and fix the environment by ensuring
conda-store-build_1   | only one version of each package is installed (conda preferred).
conda-store-build_1   | Traceback (most recent call last):
conda-store-build_1   |   File "/opt/conda-store-server/conda_store_server/build.py", line 209, in conda_build
conda-store-build_1   |     build_conda_pack(conda_store, build_path, build)
conda-store-build_1   |   File "/opt/conda-store-server/conda_store_server/build.py", line 241, in build_conda_pack
conda-store-build_1   |     conda.conda_pack(prefix=conda_prefix, output=output_filename)
conda-store-build_1   |   File "/opt/conda-store-server/conda_store_server/conda.py", line 25, in conda_pack
conda-store-build_1   |     conda_pack.pack(prefix=str(prefix), output=str(output))
conda-store-build_1   |   File "/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/conda_pack/core.py", line 514, in pack
conda-store-build_1   |     env = CondaEnv.from_prefix(prefix,
conda-store-build_1   |   File "/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/conda_pack/core.py", line 163, in from_prefix
conda-store-build_1   |     files = load_environment(prefix, **kwargs)
conda-store-build_1   |   File "/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/conda_pack/core.py", line 856, in load_environment
conda-store-build_1   |     raise CondaPackException(_missing_files_error.format(packages))
conda-store-build_1   | conda_pack.core.CondaPackException: 
conda-store-build_1   | Files managed by conda were found to have been deleted/overwritten in the
conda-store-build_1   | following packages:
conda-store-build_1   | 
conda-store-build_1   | - ncurses 6.2:
conda-store-build_1   |     share/terminfo/2/2621A
conda-store-build_1   |     share/terminfo/E/Eterm
conda-store-build_1   |     share/terminfo/E/Eterm-color
conda-store-build_1   |     + 1034 others
conda-store-build_1   | 
conda-store-build_1   | This is usually due to `pip` uninstalling or clobbering conda managed files,
conda-store-build_1   | resulting in an inconsistent environment. Please check your environment for
conda-store-build_1   | conda/pip conflicts using `conda list`, and fix the environment by ensuring
conda-store-build_1   | only one version of each package is installed (conda preferred).
conda-store-build_1   | [2021-06-24 14:52:04,992] {/opt/conda-store-server/conda_store_server/build.py:57} INFO - specification name=env-name-1 build has failed=1 times
conda-store-build_1   | [2021-06-24 14:52:04,994] {/opt/conda-store-server/conda_store_server/build.py:68} INFO - rescheduling specification name=env-name-1 on 2021-06-24 14:52:24.992881
conda-store-build_1   | [2021-06-24 14:52:06,074] {/opt/conda-store-server/conda_store_server/build.py:119} INFO - number of queued conda builds 3
conda-store-build_1   | [2021-06-24 14:52:06,093] {/opt/conda-store-server/conda_store_server/build.py:129} INFO - number of schedulable conda builds 2
conda-store-build_1   | [2021-06-24 14:52:06,123] {/opt/conda-store-server/conda_store_server/build.py:157} INFO - building /data/store/b713d98b31554b3bfff940ae81df67f92762c36e34017507ff40bdc7371ac56e-env-name-1 symlinked to /data/envs/env-name-1
conda-store-build_1   | [2021-06-24 14:52:06,124] {/opt/conda-store-server/conda_store_server/build.py:159} INFO - previously unfinished build of /data/store/b713d98b31554b3bfff940ae81df67f92762c36e34017507ff40bdc7371ac56e-env-name-1 cleaning directory

Screenshot 2021-06-24 at 16 34 29

I am guessing the other errors I have seen might be due to mutual overwrites between the copies.

Any idea why this is happening and how can I prevent this?

jaimergp commented 3 years ago

This error

conda-store-build_1   | Traceback (most recent call last):
conda-store-build_1   |   File "/opt/conda-store-server/conda_store_server/build.py", line 209, in conda_build
conda-store-build_1   |     build_conda_pack(conda_store, build_path, build)
conda-store-build_1   |   File "/opt/conda-store-server/conda_store_server/build.py", line 241, in build_conda_pack
conda-store-build_1   |     conda.conda_pack(prefix=conda_prefix, output=output_filename)
conda-store-build_1   |   File "/opt/conda-store-server/conda_store_server/conda.py", line 25, in conda_pack
conda-store-build_1   |     conda_pack.pack(prefix=str(prefix), output=str(output))
conda-store-build_1   |   File "/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/conda_pack/core.py", line 514, in pack
conda-store-build_1   |     env = CondaEnv.from_prefix(prefix,
conda-store-build_1   |   File "/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/conda_pack/core.py", line 163, in from_prefix
conda-store-build_1   |     files = load_environment(prefix, **kwargs)
conda-store-build_1   |   File "/opt/conda/envs/conda-store-server/lib/python3.9/site-packages/conda_pack/core.py", line 856, in load_environment
conda-store-build_1   |     raise CondaPackException(_missing_files_error.format(packages))
conda-store-build_1   | conda_pack.core.CondaPackException: 
conda-store-build_1   | Files managed by conda were found to have been deleted/overwritten in the
conda-store-build_1   | following packages:
conda-store-build_1   | 
conda-store-build_1   | - ncurses 6.2:
conda-store-build_1   |     share/terminfo/2/2621A
conda-store-build_1   |     share/terminfo/E/Eterm
conda-store-build_1   |     share/terminfo/E/Eterm-color
conda-store-build_1   |     + 1034 others

can be fixed by adding this kwarg to conda_pack:

def conda_pack(prefix, output):
    import conda_pack

    conda_pack.pack(prefix=str(prefix), output=str(output), ignore_missing_files=True)
#                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^

... but I don't think we want to do that.

jaimergp commented 3 years ago

It then enters into a never ending loop of failed attempts to build the environment, one attempt per copy each, like a table tennis game... :)

jaimergp commented 3 years ago

Chris and I talked about this. One of the issues discussed here is that library and filesystem triggers share the same namespace, so if you happen to add a library env with the same name as one of the filesystem envs (which was my case, given that I was using one of the example environments in tests/assets), the filesystem one is ignored.

This is is just a small issue. The missing files issue is a different one and has nothing to do with this. I'll open more issues to address them individually.

costrouc commented 3 years ago

This must be an error with conda-store not properly checking that the namespace + environment name is unique and instead is only checking environment name. The login in the environment name check need to be improved.

costrouc commented 3 years ago

This is now fixed with #96