arayabrain / barebone-studio

pipeline tool
GNU General Public License v3.0
1 stars 3 forks source link

[Onboarding]Test code specification confirmation #372

Closed itutu-tienday closed 1 month ago

itutu-tienday commented 3 months ago

Confirmed Issues

  1. Need to catch snakemae errors
  2. make test_run matPath key error
  3. make test_run command conda not found
  4. make test_run Loading packages before dependencies have finished
  5. pytest output_test/{workspace_id}/{unique_id}
  6. others (refactoring, etc)
itutu-tienday commented 3 months ago

Issues currently being checked

milesAraya commented 3 months ago

Sporadic error (only sometimes): There appears to be some issues caused when some packages have not finished being installed, but the code proceeds. The error occurs when the subsequent package depends on the prior package.

  Error:

  - Installing gax-google-logging-v2 (0.8.3)
  - Installing gax-google-pubsub-v1 (0.8.3)
  - Installing gitdb (4.0.11)
  - Installing google-auth-httplib2 (0.1.1)
  - Installing google-cloud-core (2.3.3)
  - Installing google-resumable-media (2.6.0)
  - Installing greenlet (3.0.1)
  - Installing h11 (0.14.0)
  - Installing h5py (3.10.0)
  - Installing iniconfig (2.0.0)
  - Installing jsonschema (4.19.2)
  - Installing jupyter-core (5.5.0)
  - Installing jwcrypto (1.5.0): Failed

  ChefInstallError

  Failed to install setuptools >= 40.8.0.

  Output:
  Updating dependencies
  Resolving dependencies...

  Package operations: 1 install, 0 updates, 0 removals

    - Installing setuptools (70.2.0)

    CalledProcessError

    Command '['/tmp/tmpuxta3mb1/.venv/bin/python', '-I', '-W', 'ignore', '-c', '\nimport importlib.util\nimport json\nimport sys\n\nfrom pathlib import Path\n\nspec = importlib.util.spec_from_file_location(\n    "packaging", Path(r"/usr/local/lib/python3.8/site-packages/packaging/__init__.py")\n)\npackaging = importlib.util.module_from_spec(spec)\nsys.modules[spec.name] = packaging\n\nspec = importlib.util.spec_from_file_location(\n    "packaging.tags", Path(r"/usr/local/lib/python3.8/site-packages/packaging/tags.py")\n)\npackaging_tags = importlib.util.module_from_spec(spec)\nspec.loader.exec_module(packaging_tags)\n\nprint(\n    json.dumps([(t.interpreter, t.abi, t.platform) for t in packaging_tags.sys_tags()])\n)\n']' returned non-zero exit status 1.

    at /usr/local/lib/python3.8/subprocess.py:516 in run
         512│             # We don't call process.wait() as .__exit__ does that for us.
         513│             raise
         514│         retcode = process.poll()
         515│         if check and retcode:
      →  516│             raise CalledProcessError(retcode, process.args,
         517│                                      output=stdout, stderr=stderr)
         518│     return CompletedProcess(process.args, retcode, stdout, stderr)
         519│ 
         520│ 

  The following error occurred when trying to handle this error:

    EnvCommandError

    Command ['/tmp/tmpuxta3mb1/.venv/bin/python', '-I', '-W', 'ignore', '-c', '\nimport importlib.util\nimport json\nimport sys\n\nfrom pathlib import Path\n\nspec = importlib.util.spec_from_file_location(\n    "packaging", Path(r"/usr/local/lib/python3.8/site-packages/packaging/__init__.py")\n)\npackaging = importlib.util.module_from_spec(spec)\nsys.modules[spec.name] = packaging\n\nspec = importlib.util.spec_from_file_location(\n    "packaging.tags", Path(r"/usr/local/lib/python3.8/site-packages/packaging/tags.py")\n)\npackaging_tags = importlib.util.module_from_spec(spec)\nspec.loader.exec_module(packaging_tags)\n\nprint(\n    json.dumps([(t.interpreter, t.abi, t.platform) for t in packaging_tags.sys_tags()])\n)\n'] errored with the following return code 1

    Error output:
    Traceback (most recent call last):
      File "<string>", line 18, in <module>
      File "<frozen importlib._bootstrap_external>", line 839, in exec_module
      File "<frozen importlib._bootstrap_external>", line 975, in get_code
      File "<frozen importlib._bootstrap_external>", line 1032, in get_data
    FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/site-packages/packaging/tags.py'

    at /usr/local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:342 in _run
        338│                 output = subprocess.check_output(
        339│                     cmd, stderr=stderr, env=env, text=True, **kwargs
        340│                 )
        341│         except CalledProcessError as e:
      → 342│             raise EnvCommandError(e)
        343│ 
        344│         return output
        345│ 
        346│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

  Cannot install setuptools.

  Error:

  at /usr/local/lib/python3.8/site-packages/poetry/installation/chef.py:102 in install
       98│             InstalledRepository.load(self._env),
       99│         )
      100│         installer.update(True)
      101│         if installer.run() != 0:
    → 102│             raise ChefInstallError(requirements, io.fetch_output(), io.fetch_error())
      103│ 
      104│ 
      105│ class Chef:
      106│     def __init__(

Cannot install build-system.requires for jwcrypto.

  - Installing markupsafe (2.1.3)
  - Downgrading msgpack (1.0.8 -> 1.0.7)
  - Downgrading packaging (24.1 -> 23.2)
  - Installing pandas (1.5.3)
  - Installing plac (1.4.1)
  - Installing pluggy (1.3.0)
  - Installing proto-plus (1.22.3)
  - Installing pydantic (1.10.13)
  - Installing pyyaml (6.0.1)
  - Installing ruamel-yaml (0.18.5)
  - Installing scipy (1.9.3)
  - Installing starlette (0.27.0)
  - Installing uritemplate (4.1.1)
make: *** [test_run] Error 1

Also seen without errors with certain packages. They don't seem fully installed however the code proceeds to the tests

Screenshot 2024-07-03 at 10 30 10
milesAraya commented 3 months ago

Looking at the Makefile history, With this commit

2d0f251592a296d689b7b6f1199cfe911366a2e4

Running make test_run, the matPath not found error occurs. However it can be fixed by changing these files: studio/app/common/core/snakemake/smk_builder.py RuleBuilder

studio/app/common/core/snakemake/smk.py Rule

studio/app/common/core/snakemake/snakemake_reader.py RuleConfigReader

studio/tests/app/common/core/snakemake/test_snakemake_reader.py rule_config

studio/tests/app/common/core/snakemake/test_snakemake_setfile.py Node

studio/test_data/output_test/0123/snakemake.yaml studio/test_data/output_test/0123/workflow.yaml studio/test_data/output_test/default/snakemake/snakemake.yaml Same location as hdf5Path: null

Remaining error:

/bin/bash: line 1: conda: command not found

milesAraya commented 3 months ago

In develop-main branch, fixing the matPath issue results in a new error:

rule suite2p_file_convert: input: studio/test_data/output/default/snakemake/input_0/data_endoscope.pkl output: studio/test_data/output/default/snakemake/suite2p_file_convert/suite2p_file_convert.pkl jobid: 2 reason: Forced execution resources: tmpdir=/tmp 2024-07-04 12:00:23,691 INFO: [snakemake.logging] text_handler():503 - rule suite2p_file_convert: input: studio/test_data/output/default/snakemake/input_0/data_endoscope.pkl output: studio/test_data/output/default/snakemake/suite2p_file_convert/suite2p_file_convert.pkl jobid: 2 reason: Forced execution resources: tmpdir=/tmp

2024-07-04 12:00:23,691 INFO: [snakemake.logging] text_handler():510 - 2024-07-04 12:00:28,304 INFO: [optinist] run():32 - start rule runner 2024-07-04 12:00:28,309 ERROR: [optinist] run():83 - Traceback (most recent call last):

File "/app/studio/app/common/core/rules/runner.py", line 34, in run input_info = cls.read_input_info(__rule.input)

File "/app/studio/app/common/core/rules/runner.py", line 152, in read_input_info load_data = PickleReader.read(filepath)

File "/app/studio/app/common/core/utils/pickle_handler.py", line 13, in read return pickle.load(f)

ModuleNotFoundError: No module named 'optinist.api.dataclass.image'

Files changed:

studio/test_data/output_test/0123/snakemake.yaml studio/test_data/output_test/default/snakemake/snakemake.yaml

milesAraya commented 3 months ago

Error with conda persists

/usr/bin/bash: line 1: conda: command not found ... subprocess.CalledProcessError: Command 'conda info --json' returned non-zero exit status 127.

[Thu Jul 4 12:00:41 2024] Finished job 1. 2 of 3 steps (67%) done Select jobs to execute...

[Thu Jul 4 12:00:41 2024] localrule all: input: studio/test_data/output/default/snakemake/suite2p_roi/suite2p_roi.pkl jobid: 0 reason: Input files updated by another job: studio/test_data/output/default/snakemake/suite2p_roi/suite2p_roi.pkl resources: tmpdir=/tmp

[Thu Jul 4 12:00:41 2024] Finished job 0. 3 of 3 steps (100%) done Complete log: .snakemake/log/2024-07-04T120032.806394.snakemake.log .Building DAG of jobs... /usr/bin/bash: line 1: conda: command not found Traceback (most recent call last): File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/init.py", line 771, in snakemake success = workflow.execute( File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/workflow.py", line 924, in execute dag.create_conda_envs( File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/dag.py", line 349, in create_conda_envs env.create(dryrun) File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/deployment/conda.py", line 393, in create pin_file = self.pin_file File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/common/init.py", line 217, in get value = self.method(instance) File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/deployment/conda.py", line 103, in pin_file f".{self.conda.platform}.pin.txt" File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/common/init.py", line 217, in get value = self.method(instance) File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/deployment/conda.py", line 96, in conda return Conda( File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/deployment/conda.py", line 653, in init shell.check_output(self._get_cmd("conda info --json"), text=True) File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/shell.py", line 61, in check_output return sp.check_output(cmd, shell=True, executable=executable, *kwargs) File "/usr/local/lib/python3.8/subprocess.py", line 415, in check_output return run(popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/local/lib/python3.8/subprocess.py", line 516, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command 'conda info --json' returned non-zero exit status 127.

conda.yaml deleted from previous version of test. Replaced with Poetry

Commit Previous code:

services:
  test_studio:
    image: test_studio
    container_name: test_studio
    build:
      context: .
      dockerfile: studio/config/docker/Dockerfile.test
    environment:
      PYTHONPATH: .
      TZ: Asia/Tokyo
    volumes:
      - ./studio:/app/studio
      - ./conda/yaml:/app/conda/yaml
      - ./studio/test_data:/tmp/studio
      - ./frontend/build:/app/frontend/build
milesAraya commented 3 months ago

No errors with yarn test

(node:29942) [DEP0040] DeprecationWarning: The punycode module is deprecated. Please use a userland alternative instead. (Use node --trace-deprecation ... to show where the warning was created) PASS src/store/slice/AlgorithmList/AlgorithmList.test.ts (node:29940) [DEP0040] DeprecationWarning: The punycode module is deprecated. Please use a userland alternative instead. (Use node --trace-deprecation ... to show where the warning was created) (node:29941) [DEP0040] DeprecationWarning: The punycode module is deprecated. Please use a userland alternative instead. (Use node --trace-deprecation ... to show where the warning was created) PASS src/store/test/record/RecordReproduce.test.ts PASS src/store/test/RunSelectors.test.ts (node:29939) [DEP0040] DeprecationWarning: The punycode module is deprecated. Please use a userland alternative instead. (Use node --trace-deprecation ... to show where the warning was created) PASS src/store/test/visualize/Visualize.test.ts PASS src/store/test/workflow/AlgorithmNode/AlgorithmNode.test.ts PASS src/store/slice/Experiments/Experiments.test.ts PASS src/store/slice/FilesTree/FilesTree.test.ts PASS src/store/test/workflow/InputNode/InputNode.test.ts

Test Suites: 8 passed, 8 total Tests: 22 passed, 22 total Snapshots: 0 total Time: 0.831 s, estimated 1 s Ran all test suites.

milesAraya commented 3 months ago

pytest errors

_ ERROR collecting studio/tests/app/common/core/snakemake/test_snakemake_executor.py __ studio/tests/app/common/core/snakemake/test_snakemake_executor.py:25: in shutil.copytree( ../../miniconda3/envs/optinist_dev/lib/python3.8/shutil.py:555: in copytree with os.scandir(src) as itr: E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/studio/outputtest/default/snakemake' ____ ERROR collecting studio/tests/app/common/routers/testexperiment.py ____ studio/tests/app/common/routers/test_experiment.py:12: in shutil.copytree( ../../miniconda3/envs/optinist_dev/lib/python3.8/shutil.py:555: in copytree with os.scandir(src) as itr: E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/studio/output_test/0123'

shutil.copytree( f"{DIRPATH.DATA_DIR}/output_test/{workspace_id}/{unique_id}", f"{DIRPATH.DATA_DIR}/output/{workspace_id}/{unique_id}", dirs_exist_ok=True, )

The cause is likely that within /output_test/ there is no folder {workspace_id}/{unique_id} ls studio/test_data/output_test

0123 default result_test smk_test test

itutu-tienday commented 2 months ago
milesAraya commented 2 months ago

This issue persists https://github.com/arayabrain/barebone-studio/issues/372#issuecomment-2204907885

✔ Network barebone-studio_default Created 0.0s Skipping virtualenv creation, as specified in config file. Installing dependencies from lock file

Package operations: 122 installs, 17 updates, 0 removals

  • Installing pyasn1 (0.5.0)
  • Installing pyparsing (3.1.1)
  • Installing attrs (23.1.0)
  • Installing cachetools (5.3.2)
  • Downgrading certifi (2024.7.4 -> 2023.7.22)
  • Installing httplib2 (0.22.0)
  • Downgrading idna (3.7 -> 3.4)
  • Installing protobuf (4.25.0)
  • Installing pyasn1-modules (0.3.0)
  • Downgrading pycparser (2.22 -> 2.21)
  • Installing rpds-py (0.12.0)
  • Installing rsa (4.9)
  • Installing six (1.16.0)
  • Downgrading urllib3 (2.2.2 -> 1.26.18)
  • Downgrading zipp (3.19.2 -> 3.17.0)
  • Installing exceptiongroup (1.1.3)
  • Installing future (0.18.3)
  • Installing google-auth (2.23.4)
  • Installing googleapis-common-protos (1.61.0)
  • Installing grpcio (1.59.2)
  • Downgrading importlib-resources (6.4.0 -> 6.1.1)
  • Installing oauth2client (4.1.3)
  • Installing ply (3.8)
  • Installing referencing (0.30.2)
  • Downgrading requests (2.32.3 -> 2.29.0)
  • Installing sniffio (1.3.0)
  • Installing wrapt (1.16.0)
  • Installing anyio (3.7.1)
  • Downgrading cryptography (43.0.0 -> 41.0.6)
  • Installing deprecated (1.2.14)
  • Installing google-api-core (2.14.0)
  • Installing google-crc32c (1.5.0)
  • Installing google-gax (0.12.5)
  • Installing grpc-google-logging-v2 (0.8.1)
  • Installing grpc-google-pubsub-v1 (0.8.1)
  • Installing grpcio-status (1.59.2)
  • Installing jsonschema-specifications (2023.7.1)
  • Installing numpy (1.23.5)
  • Installing pkgutil-resolve-name (1.3.10)
  • Downgrading platformdirs (4.2.2 -> 3.11.0)
  • Installing python-dateutil (2.8.2)
  • Installing pytz (2023.3.post1)
  • Installing ruamel-yaml-clib (0.2.8)
  • Installing smmap (5.0.1)
  • Installing traitlets (5.13.0)
  • Installing typing-extensions (4.8.0)
  • Installing dnspython (2.4.2)
  • Installing dpath (2.1.6)
  • Downgrading fastjsonschema (2.20.0 -> 2.18.1)
  • Installing gax-google-logging-v2 (0.8.3): Preparing...
  • Installing gax-google-pubsub-v1 (0.8.3): Preparing...
  • Installing gitdb (4.0.11)
  • Installing google-auth-httplib2 (0.1.1)
  • Installing google-cloud-core (2.3.3)
  • Installing google-resumable-media (2.6.0)
  • Installing greenlet (3.0.1)
  • Installing h11 (0.14.0)
  • Installing h5py (3.10.0): Installing...
  • Installing iniconfig (2.0.0)
  • Installing jsonschema (4.19.2)
  • Installing jupyter-core (5.5.0)
  • Installing jwcrypto (1.5.0): Failed

    ChefInstallError

    Failed to install setuptools >= 40.8.0.

    Output: Updating dependencies Resolving dependencies...

    Package operations: 1 install, 0 updates, 0 removals

    • Installing setuptools (71.1.0)

    CalledProcessError

    Command '['/tmp/tmpbwmjyabf/.venv/bin/python', '-I', '-W', 'ignore', '-c', '\nimport importlib.util\nimport json\nimport sys\n\nfrom pathlib import Path\n\nspec = importlib.util.spec_from_file_location(\n "packaging", Path(r"/usr/local/lib/python3.8/site-packages/packaging/init.py")\n)\npackaging = importlib.util.module_from_spec(spec)\nsys.modules[spec.name] = packaging\n\nspec = importlib.util.spec_from_file_location(\n "packaging.tags", Path(r"/usr/local/lib/python3.8/site-packages/packaging/tags.py")\n)\npackaging_tags = importlib.util.module_from_spec(spec)\nspec.loader.exec_module(packaging_tags)\n\nprint(\n json.dumps([(t.interpreter, t.abi, t.platform) for t in packaging_tags.sys_tags()])\n)\n']' returned non-zero exit status 1.

    at /usr/local/lib/python3.8/subprocess.py:516 in run 512│ # We don't call process.wait() as .exit does that for us. 513│ raise 514│ retcode = process.poll() 515│ if check and retcode: → 516│ raise CalledProcessError(retcode, process.args, 517│ output=stdout, stderr=stderr) 518│ return CompletedProcess(process.args, retcode, stdout, stderr) 519│ 520│

    The following error occurred when trying to handle this error:

    EnvCommandError

    Command ['/tmp/tmpbwmjyabf/.venv/bin/python', '-I', '-W', 'ignore', '-c', '\nimport importlib.util\nimport json\nimport sys\n\nfrom pathlib import Path\n\nspec = importlib.util.spec_from_file_location(\n "packaging", Path(r"/usr/local/lib/python3.8/site-packages/packaging/init.py")\n)\npackaging = importlib.util.module_from_spec(spec)\nsys.modules[spec.name] = packaging\n\nspec = importlib.util.spec_from_file_location(\n "packaging.tags", Path(r"/usr/local/lib/python3.8/site-packages/packaging/tags.py")\n)\npackaging_tags = importlib.util.module_from_spec(spec)\nspec.loader.exec_module(packaging_tags)\n\nprint(\n json.dumps([(t.interpreter, t.abi, t.platform) for t in packaging_tags.sys_tags()])\n)\n'] errored with the following return code 1

    Error output: Traceback (most recent call last): File "", line 18, in File "", line 839, in exec_module File "", line 975, in get_code File "", line 1032, in get_data FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/site-packages/packaging/tags.py'

    at /usr/local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:342 in _run 338│ output = subprocess.check_output(

  • Installing iniconfig (2.0.0)
  • Installing jsonschema (4.19.2)
  • Installing jupyter-core (5.5.0)
  • Installing jwcrypto (1.5.0): Failed

    ChefInstallError

    Failed to install setuptools >= 40.8.0.

    Output: Updating dependencies Resolving dependencies...

    Package operations: 1 install, 0 updates, 0 removals

    • Installing setuptools (71.1.0)

    CalledProcessError

    Command '['/tmp/tmpbwmjyabf/.venv/bin/python', '-I', '-W', 'ignore', '-c', '\nimport importlib.util\nimport json\nimport sys\n\nfrom pathlib import Path\n\nspec = importlib.util.spec_from_file_location(\n "packaging", Path(r"/usr/local/lib/python3.8/site-packages/packaging/init.py")\n)\npackaging = importlib.util.module_from_spec(spec)\nsys.modules[spec.name] = packaging\n\nspec = importlib.util.spec_from_file_location(\n "packaging.tags", Path(r"/usr/local/lib/python3.8/site-packages/packaging/tags.py")\n)\npackaging_tags = importlib.util.module_from_spec(spec)\nspec.loader.exec_module(packaging_tags)\n\nprint(\n json.dumps([(t.interpreter, t.abi, t.platform) for t in packaging_tags.sys_tags()])\n)\n']' returned non-zero exit status 1.

    at /usr/local/lib/python3.8/subprocess.py:516 in run 512│ # We don't call process.wait() as .exit does that for us. 513│ raise 514│ retcode = process.poll() 515│ if check and retcode: → 516│ raise CalledProcessError(retcode, process.args, 517│ output=stdout, stderr=stderr) 518│ return CompletedProcess(process.args, retcode, stdout, stderr) 519│ 520│

    The following error occurred when trying to handle this error:

    EnvCommandError

    Command ['/tmp/tmpbwmjyabf/.venv/bin/python', '-I', '-W', 'ignore', '-c', '\nimport importlib.util\nimport json\nimport sys\n\nfrom pathlib import Path\n\nspec = importlib.util.spec_from_file_location(\n "packaging", Path(r"/usr/local/lib/python3.8/site-packages/packaging/init.py")\n)\npackaging = importlib.util.module_from_spec(spec)\nsys.modules[spec.name] = packaging\n\nspec = importlib.util.spec_from_file_location(\n "packaging.tags", Path(r"/usr/local/lib/python3.8/site-packages/packaging/tags.py")\n)\npackaging_tags = importlib.util.module_from_spec(spec)\nspec.loader.exec_module(packaging_tags)\n\nprint(\n json.dumps([(t.interpreter, t.abi, t.platform) for t in packaging_tags.sys_tags()])\n)\n'] errored with the following return code 1

    Error output: Traceback (most recent call last): File "", line 18, in File "", line 839, in exec_module File "", line 975, in get_code File "", line 1032, in get_data FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/site-packages/packaging/tags.py'

    at /usr/local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:342 in _run 338│ output = subprocess.check_output(

  • Installing h5py (3.10.0)
  • Installing iniconfig (2.0.0)
  • Installing jsonschema (4.19.2)
  • Installing jupyter-core (5.5.0)
  • Installing jwcrypto (1.5.0): Failed