Closed itutu-tienday closed 1 month ago
Issues currently being checked
Some error in 2 errors in make test_run
Traceback (most recent call last): File "/app/.snakemake/scripts/tmp_dgp9k68.func.py", line 30, in <module> rule_config = RuleConfigReader.read(snakemake.params.name) File "/app/studio/app/common/core/snakemake/snakemake_reader.py", line 15, in read matPath=rule["matPath"], KeyError: 'matPath'
/usr/bin/bash: line 1: conda: command not found
Pytest is determining success even though an error occurs in make test_run.
Confirmation of differences between make test_run and pytest test contents
Sporadic error (only sometimes): There appears to be some issues caused when some packages have not finished being installed, but the code proceeds. The error occurs when the subsequent package depends on the prior package.
Error:
- Installing gax-google-logging-v2 (0.8.3)
- Installing gax-google-pubsub-v1 (0.8.3)
- Installing gitdb (4.0.11)
- Installing google-auth-httplib2 (0.1.1)
- Installing google-cloud-core (2.3.3)
- Installing google-resumable-media (2.6.0)
- Installing greenlet (3.0.1)
- Installing h11 (0.14.0)
- Installing h5py (3.10.0)
- Installing iniconfig (2.0.0)
- Installing jsonschema (4.19.2)
- Installing jupyter-core (5.5.0)
- Installing jwcrypto (1.5.0): Failed
ChefInstallError
Failed to install setuptools >= 40.8.0.
Output:
Updating dependencies
Resolving dependencies...
Package operations: 1 install, 0 updates, 0 removals
- Installing setuptools (70.2.0)
CalledProcessError
Command '['/tmp/tmpuxta3mb1/.venv/bin/python', '-I', '-W', 'ignore', '-c', '\nimport importlib.util\nimport json\nimport sys\n\nfrom pathlib import Path\n\nspec = importlib.util.spec_from_file_location(\n "packaging", Path(r"/usr/local/lib/python3.8/site-packages/packaging/__init__.py")\n)\npackaging = importlib.util.module_from_spec(spec)\nsys.modules[spec.name] = packaging\n\nspec = importlib.util.spec_from_file_location(\n "packaging.tags", Path(r"/usr/local/lib/python3.8/site-packages/packaging/tags.py")\n)\npackaging_tags = importlib.util.module_from_spec(spec)\nspec.loader.exec_module(packaging_tags)\n\nprint(\n json.dumps([(t.interpreter, t.abi, t.platform) for t in packaging_tags.sys_tags()])\n)\n']' returned non-zero exit status 1.
at /usr/local/lib/python3.8/subprocess.py:516 in run
512│ # We don't call process.wait() as .__exit__ does that for us.
513│ raise
514│ retcode = process.poll()
515│ if check and retcode:
→ 516│ raise CalledProcessError(retcode, process.args,
517│ output=stdout, stderr=stderr)
518│ return CompletedProcess(process.args, retcode, stdout, stderr)
519│
520│
The following error occurred when trying to handle this error:
EnvCommandError
Command ['/tmp/tmpuxta3mb1/.venv/bin/python', '-I', '-W', 'ignore', '-c', '\nimport importlib.util\nimport json\nimport sys\n\nfrom pathlib import Path\n\nspec = importlib.util.spec_from_file_location(\n "packaging", Path(r"/usr/local/lib/python3.8/site-packages/packaging/__init__.py")\n)\npackaging = importlib.util.module_from_spec(spec)\nsys.modules[spec.name] = packaging\n\nspec = importlib.util.spec_from_file_location(\n "packaging.tags", Path(r"/usr/local/lib/python3.8/site-packages/packaging/tags.py")\n)\npackaging_tags = importlib.util.module_from_spec(spec)\nspec.loader.exec_module(packaging_tags)\n\nprint(\n json.dumps([(t.interpreter, t.abi, t.platform) for t in packaging_tags.sys_tags()])\n)\n'] errored with the following return code 1
Error output:
Traceback (most recent call last):
File "<string>", line 18, in <module>
File "<frozen importlib._bootstrap_external>", line 839, in exec_module
File "<frozen importlib._bootstrap_external>", line 975, in get_code
File "<frozen importlib._bootstrap_external>", line 1032, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/site-packages/packaging/tags.py'
at /usr/local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:342 in _run
338│ output = subprocess.check_output(
339│ cmd, stderr=stderr, env=env, text=True, **kwargs
340│ )
341│ except CalledProcessError as e:
→ 342│ raise EnvCommandError(e)
343│
344│ return output
345│
346│ def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
Cannot install setuptools.
Error:
at /usr/local/lib/python3.8/site-packages/poetry/installation/chef.py:102 in install
98│ InstalledRepository.load(self._env),
99│ )
100│ installer.update(True)
101│ if installer.run() != 0:
→ 102│ raise ChefInstallError(requirements, io.fetch_output(), io.fetch_error())
103│
104│
105│ class Chef:
106│ def __init__(
Cannot install build-system.requires for jwcrypto.
- Installing markupsafe (2.1.3)
- Downgrading msgpack (1.0.8 -> 1.0.7)
- Downgrading packaging (24.1 -> 23.2)
- Installing pandas (1.5.3)
- Installing plac (1.4.1)
- Installing pluggy (1.3.0)
- Installing proto-plus (1.22.3)
- Installing pydantic (1.10.13)
- Installing pyyaml (6.0.1)
- Installing ruamel-yaml (0.18.5)
- Installing scipy (1.9.3)
- Installing starlette (0.27.0)
- Installing uritemplate (4.1.1)
make: *** [test_run] Error 1
Also seen without errors with certain packages. They don't seem fully installed however the code proceeds to the tests
Looking at the Makefile history, With this commit
2d0f251592a296d689b7b6f1199cfe911366a2e4
Running make test_run
, the matPath not found error occurs. However it can be fixed by changing these files:
studio/app/common/core/snakemake/smk_builder.py
RuleBuilder
studio/app/common/core/snakemake/smk.py Rule
studio/app/common/core/snakemake/snakemake_reader.py RuleConfigReader
studio/tests/app/common/core/snakemake/test_snakemake_reader.py rule_config
studio/tests/app/common/core/snakemake/test_snakemake_setfile.py Node
studio/test_data/output_test/0123/snakemake.yaml studio/test_data/output_test/0123/workflow.yaml studio/test_data/output_test/default/snakemake/snakemake.yaml Same location as hdf5Path: null
/bin/bash: line 1: conda: command not found
In develop-main branch, fixing the matPath issue results in a new error:
rule suite2p_file_convert: input: studio/test_data/output/default/snakemake/input_0/data_endoscope.pkl output: studio/test_data/output/default/snakemake/suite2p_file_convert/suite2p_file_convert.pkl jobid: 2 reason: Forced execution resources: tmpdir=/tmp 2024-07-04 12:00:23,691 INFO: [snakemake.logging] text_handler():503 - rule suite2p_file_convert: input: studio/test_data/output/default/snakemake/input_0/data_endoscope.pkl output: studio/test_data/output/default/snakemake/suite2p_file_convert/suite2p_file_convert.pkl jobid: 2 reason: Forced execution resources: tmpdir=/tmp
2024-07-04 12:00:23,691 INFO: [snakemake.logging] text_handler():510 - 2024-07-04 12:00:28,304 INFO: [optinist] run():32 - start rule runner 2024-07-04 12:00:28,309 ERROR: [optinist] run():83 - Traceback (most recent call last):
File "/app/studio/app/common/core/rules/runner.py", line 34, in run input_info = cls.read_input_info(__rule.input)
File "/app/studio/app/common/core/rules/runner.py", line 152, in read_input_info load_data = PickleReader.read(filepath)
File "/app/studio/app/common/core/utils/pickle_handler.py", line 13, in read return pickle.load(f)
ModuleNotFoundError: No module named 'optinist.api.dataclass.image'
studio/test_data/output_test/0123/snakemake.yaml studio/test_data/output_test/default/snakemake/snakemake.yaml
hdf5Path: null
Add: matPath: null
/usr/bin/bash: line 1: conda: command not found ... subprocess.CalledProcessError: Command 'conda info --json' returned non-zero exit status 127.
[Thu Jul 4 12:00:41 2024] Finished job 1. 2 of 3 steps (67%) done Select jobs to execute...
[Thu Jul 4 12:00:41 2024] localrule all: input: studio/test_data/output/default/snakemake/suite2p_roi/suite2p_roi.pkl jobid: 0 reason: Input files updated by another job: studio/test_data/output/default/snakemake/suite2p_roi/suite2p_roi.pkl resources: tmpdir=/tmp
[Thu Jul 4 12:00:41 2024] Finished job 0. 3 of 3 steps (100%) done Complete log: .snakemake/log/2024-07-04T120032.806394.snakemake.log .Building DAG of jobs... /usr/bin/bash: line 1: conda: command not found Traceback (most recent call last): File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/init.py", line 771, in snakemake success = workflow.execute( File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/workflow.py", line 924, in execute dag.create_conda_envs( File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/dag.py", line 349, in create_conda_envs env.create(dryrun) File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/deployment/conda.py", line 393, in create pin_file = self.pin_file File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/common/init.py", line 217, in get value = self.method(instance) File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/deployment/conda.py", line 103, in pin_file f".{self.conda.platform}.pin.txt" File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/common/init.py", line 217, in get value = self.method(instance) File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/deployment/conda.py", line 96, in conda return Conda( File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/deployment/conda.py", line 653, in init shell.check_output(self._get_cmd("conda info --json"), text=True) File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/shell.py", line 61, in check_output return sp.check_output(cmd, shell=True, executable=executable, *kwargs) File "/usr/local/lib/python3.8/subprocess.py", line 415, in check_output return run(popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/local/lib/python3.8/subprocess.py", line 516, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command 'conda info --json' returned non-zero exit status 127.
Commit Previous code:
services:
test_studio:
image: test_studio
container_name: test_studio
build:
context: .
dockerfile: studio/config/docker/Dockerfile.test
environment:
PYTHONPATH: .
TZ: Asia/Tokyo
volumes:
- ./studio:/app/studio
- ./conda/yaml:/app/conda/yaml
- ./studio/test_data:/tmp/studio
- ./frontend/build:/app/frontend/build
No errors with yarn test
(node:29942) [DEP0040] DeprecationWarning: The
punycode
module is deprecated. Please use a userland alternative instead. (Usenode --trace-deprecation ...
to show where the warning was created) PASS src/store/slice/AlgorithmList/AlgorithmList.test.ts (node:29940) [DEP0040] DeprecationWarning: Thepunycode
module is deprecated. Please use a userland alternative instead. (Usenode --trace-deprecation ...
to show where the warning was created) (node:29941) [DEP0040] DeprecationWarning: Thepunycode
module is deprecated. Please use a userland alternative instead. (Usenode --trace-deprecation ...
to show where the warning was created) PASS src/store/test/record/RecordReproduce.test.ts PASS src/store/test/RunSelectors.test.ts (node:29939) [DEP0040] DeprecationWarning: Thepunycode
module is deprecated. Please use a userland alternative instead. (Usenode --trace-deprecation ...
to show where the warning was created) PASS src/store/test/visualize/Visualize.test.ts PASS src/store/test/workflow/AlgorithmNode/AlgorithmNode.test.ts PASS src/store/slice/Experiments/Experiments.test.ts PASS src/store/slice/FilesTree/FilesTree.test.ts PASS src/store/test/workflow/InputNode/InputNode.test.tsTest Suites: 8 passed, 8 total Tests: 22 passed, 22 total Snapshots: 0 total Time: 0.831 s, estimated 1 s Ran all test suites.
_ ERROR collecting studio/tests/app/common/core/snakemake/test_snakemake_executor.py __ studio/tests/app/common/core/snakemake/test_snakemake_executor.py:25: in
shutil.copytree( ../../miniconda3/envs/optinist_dev/lib/python3.8/shutil.py:555: in copytree with os.scandir(src) as itr: E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/studio/outputtest/default/snakemake' ____ ERROR collecting studio/tests/app/common/routers/testexperiment.py ____ studio/tests/app/common/routers/test_experiment.py:12: inshutil.copytree( ../../miniconda3/envs/optinist_dev/lib/python3.8/shutil.py:555: in copytree with os.scandir(src) as itr: E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/studio/output_test/0123' shutil.copytree( f"{DIRPATH.DATA_DIR}/output_test/{workspace_id}/{unique_id}", f"{DIRPATH.DATA_DIR}/output/{workspace_id}/{unique_id}", dirs_exist_ok=True, )
The cause is likely that within /output_test/ there is no folder {workspace_id}/{unique_id}
ls studio/test_data/output_test
0123 default result_test smk_test test
This issue persists https://github.com/arayabrain/barebone-studio/issues/372#issuecomment-2204907885
✔ Network barebone-studio_default Created 0.0s Skipping virtualenv creation, as specified in config file. Installing dependencies from lock file
Package operations: 122 installs, 17 updates, 0 removals
- Installing pyasn1 (0.5.0)
- Installing pyparsing (3.1.1)
- Installing attrs (23.1.0)
- Installing cachetools (5.3.2)
- Downgrading certifi (2024.7.4 -> 2023.7.22)
- Installing httplib2 (0.22.0)
- Downgrading idna (3.7 -> 3.4)
- Installing protobuf (4.25.0)
- Installing pyasn1-modules (0.3.0)
- Downgrading pycparser (2.22 -> 2.21)
- Installing rpds-py (0.12.0)
- Installing rsa (4.9)
- Installing six (1.16.0)
- Downgrading urllib3 (2.2.2 -> 1.26.18)
- Downgrading zipp (3.19.2 -> 3.17.0)
- Installing exceptiongroup (1.1.3)
- Installing future (0.18.3)
- Installing google-auth (2.23.4)
- Installing googleapis-common-protos (1.61.0)
- Installing grpcio (1.59.2)
- Downgrading importlib-resources (6.4.0 -> 6.1.1)
- Installing oauth2client (4.1.3)
- Installing ply (3.8)
- Installing referencing (0.30.2)
- Downgrading requests (2.32.3 -> 2.29.0)
- Installing sniffio (1.3.0)
- Installing wrapt (1.16.0)
- Installing anyio (3.7.1)
- Downgrading cryptography (43.0.0 -> 41.0.6)
- Installing deprecated (1.2.14)
- Installing google-api-core (2.14.0)
- Installing google-crc32c (1.5.0)
- Installing google-gax (0.12.5)
- Installing grpc-google-logging-v2 (0.8.1)
- Installing grpc-google-pubsub-v1 (0.8.1)
- Installing grpcio-status (1.59.2)
- Installing jsonschema-specifications (2023.7.1)
- Installing numpy (1.23.5)
- Installing pkgutil-resolve-name (1.3.10)
- Downgrading platformdirs (4.2.2 -> 3.11.0)
- Installing python-dateutil (2.8.2)
- Installing pytz (2023.3.post1)
- Installing ruamel-yaml-clib (0.2.8)
- Installing smmap (5.0.1)
- Installing traitlets (5.13.0)
- Installing typing-extensions (4.8.0)
- Installing dnspython (2.4.2)
- Installing dpath (2.1.6)
- Downgrading fastjsonschema (2.20.0 -> 2.18.1)
- Installing gax-google-logging-v2 (0.8.3): Preparing...
- Installing gax-google-pubsub-v1 (0.8.3): Preparing...
- Installing gitdb (4.0.11)
- Installing google-auth-httplib2 (0.1.1)
- Installing google-cloud-core (2.3.3)
- Installing google-resumable-media (2.6.0)
- Installing greenlet (3.0.1)
- Installing h11 (0.14.0)
- Installing h5py (3.10.0): Installing...
- Installing iniconfig (2.0.0)
- Installing jsonschema (4.19.2)
- Installing jupyter-core (5.5.0)
Installing jwcrypto (1.5.0): Failed
ChefInstallError
Failed to install setuptools >= 40.8.0.
Output: Updating dependencies Resolving dependencies...
Package operations: 1 install, 0 updates, 0 removals
- Installing setuptools (71.1.0)
CalledProcessError
Command '['/tmp/tmpbwmjyabf/.venv/bin/python', '-I', '-W', 'ignore', '-c', '\nimport importlib.util\nimport json\nimport sys\n\nfrom pathlib import Path\n\nspec = importlib.util.spec_from_file_location(\n "packaging", Path(r"/usr/local/lib/python3.8/site-packages/packaging/init.py")\n)\npackaging = importlib.util.module_from_spec(spec)\nsys.modules[spec.name] = packaging\n\nspec = importlib.util.spec_from_file_location(\n "packaging.tags", Path(r"/usr/local/lib/python3.8/site-packages/packaging/tags.py")\n)\npackaging_tags = importlib.util.module_from_spec(spec)\nspec.loader.exec_module(packaging_tags)\n\nprint(\n json.dumps([(t.interpreter, t.abi, t.platform) for t in packaging_tags.sys_tags()])\n)\n']' returned non-zero exit status 1.
at /usr/local/lib/python3.8/subprocess.py:516 in run 512│ # We don't call process.wait() as .exit does that for us. 513│ raise 514│ retcode = process.poll() 515│ if check and retcode: → 516│ raise CalledProcessError(retcode, process.args, 517│ output=stdout, stderr=stderr) 518│ return CompletedProcess(process.args, retcode, stdout, stderr) 519│ 520│
The following error occurred when trying to handle this error:
EnvCommandError
Command ['/tmp/tmpbwmjyabf/.venv/bin/python', '-I', '-W', 'ignore', '-c', '\nimport importlib.util\nimport json\nimport sys\n\nfrom pathlib import Path\n\nspec = importlib.util.spec_from_file_location(\n "packaging", Path(r"/usr/local/lib/python3.8/site-packages/packaging/init.py")\n)\npackaging = importlib.util.module_from_spec(spec)\nsys.modules[spec.name] = packaging\n\nspec = importlib.util.spec_from_file_location(\n "packaging.tags", Path(r"/usr/local/lib/python3.8/site-packages/packaging/tags.py")\n)\npackaging_tags = importlib.util.module_from_spec(spec)\nspec.loader.exec_module(packaging_tags)\n\nprint(\n json.dumps([(t.interpreter, t.abi, t.platform) for t in packaging_tags.sys_tags()])\n)\n'] errored with the following return code 1
Error output: Traceback (most recent call last): File "
", line 18, in File " ", line 839, in exec_module File " ", line 975, in get_code File " ", line 1032, in get_data FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/site-packages/packaging/tags.py' at /usr/local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:342 in _run 338│ output = subprocess.check_output(
- Installing iniconfig (2.0.0)
- Installing jsonschema (4.19.2)
- Installing jupyter-core (5.5.0)
Installing jwcrypto (1.5.0): Failed
ChefInstallError
Failed to install setuptools >= 40.8.0.
Output: Updating dependencies Resolving dependencies...
Package operations: 1 install, 0 updates, 0 removals
- Installing setuptools (71.1.0)
CalledProcessError
Command '['/tmp/tmpbwmjyabf/.venv/bin/python', '-I', '-W', 'ignore', '-c', '\nimport importlib.util\nimport json\nimport sys\n\nfrom pathlib import Path\n\nspec = importlib.util.spec_from_file_location(\n "packaging", Path(r"/usr/local/lib/python3.8/site-packages/packaging/init.py")\n)\npackaging = importlib.util.module_from_spec(spec)\nsys.modules[spec.name] = packaging\n\nspec = importlib.util.spec_from_file_location(\n "packaging.tags", Path(r"/usr/local/lib/python3.8/site-packages/packaging/tags.py")\n)\npackaging_tags = importlib.util.module_from_spec(spec)\nspec.loader.exec_module(packaging_tags)\n\nprint(\n json.dumps([(t.interpreter, t.abi, t.platform) for t in packaging_tags.sys_tags()])\n)\n']' returned non-zero exit status 1.
at /usr/local/lib/python3.8/subprocess.py:516 in run 512│ # We don't call process.wait() as .exit does that for us. 513│ raise 514│ retcode = process.poll() 515│ if check and retcode: → 516│ raise CalledProcessError(retcode, process.args, 517│ output=stdout, stderr=stderr) 518│ return CompletedProcess(process.args, retcode, stdout, stderr) 519│ 520│
The following error occurred when trying to handle this error:
EnvCommandError
Command ['/tmp/tmpbwmjyabf/.venv/bin/python', '-I', '-W', 'ignore', '-c', '\nimport importlib.util\nimport json\nimport sys\n\nfrom pathlib import Path\n\nspec = importlib.util.spec_from_file_location(\n "packaging", Path(r"/usr/local/lib/python3.8/site-packages/packaging/init.py")\n)\npackaging = importlib.util.module_from_spec(spec)\nsys.modules[spec.name] = packaging\n\nspec = importlib.util.spec_from_file_location(\n "packaging.tags", Path(r"/usr/local/lib/python3.8/site-packages/packaging/tags.py")\n)\npackaging_tags = importlib.util.module_from_spec(spec)\nspec.loader.exec_module(packaging_tags)\n\nprint(\n json.dumps([(t.interpreter, t.abi, t.platform) for t in packaging_tags.sys_tags()])\n)\n'] errored with the following return code 1
Error output: Traceback (most recent call last): File "
", line 18, in File " ", line 839, in exec_module File " ", line 975, in get_code File " ", line 1032, in get_data FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/site-packages/packaging/tags.py' at /usr/local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:342 in _run 338│ output = subprocess.check_output(
- Installing h5py (3.10.0)
- Installing iniconfig (2.0.0)
- Installing jsonschema (4.19.2)
- Installing jupyter-core (5.5.0)
- Installing jwcrypto (1.5.0): Failed
OBJECTIVE
Work contents
Supplemental information
Deliverables
Confirmed Issues
make test_run
matPath key errormake test_run
command conda not foundmake test_run
Loading packages before dependencies have finishedpytest
output_test/{workspace_id}/{unique_id}