search

arayabrain / barebone-studio

pipeline tool
GNU General Public License v3.0
1 stars 3 forks source link

[Cloud Support] Resolving CaImAn errors on AWS ECS #440

Closed itutu-tienday closed 3 weeks ago

itutu-tienday commented 1 month ago

The following error occurs when running CaImAn in AWS ECS environment. Countermeasures are needed.

ERROR - smk_status_logger.py -
{'level': 'debug', 'msg': 'Full Traceback (most recent call last):\n
  File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/deployment/conda.py", line 576, in create\n
    out = create_env(env_file, filetype="yaml")\n
  File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/deployment/conda.py", line 538, in create_env\n
    out = shell.check_output(\n
  File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/shell.py", line 61, in check_output\n
    return sp.check_output(cmd, shell=True, executable=executable, **kwargs)\n
  File "/usr/local/lib/python3.8/subprocess.py", line 415, in check_output\n
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,\n
  File "/usr/local/lib/python3.8/subprocess.py", line 516, in run\n
    raise CalledProcessError(retcode, process.args,\n
subprocess.CalledProcessError: Command \'conda env create --quiet --file "/app/.snakemake/conda/23a0fcc13df369566a15cf65d2b25641_.yaml" --prefix "/app/.snakemake/conda/23a0fcc13df369566a15cf65d2b25641_"\' died with <Signals.SIGKILL: 9>.\n
\n
During handling of the above exception, another exception occurred:\n
\n
Traceback (most recent call last):\n
  File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/__init__.py", line 771, in snakemake\n
    success = workflow.execute(\n
  File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/workflow.py", line 924, in execute\n
    dag.create_conda_envs(\n
  File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/dag.py", line 349, in create_conda_envs\n
    env.create(dryrun)\n
  File "/root/.cache/pypoetry/virtualenvs/optinist-9TtSrW0h-py3.8/lib/python3.8/site-packages/snakemake/deployment/conda.py", line 595, in create\n
    raise CreateCondaEnvironmentException(\n
snakemake.exceptions.CreateCondaEnvironmentException: Could not create conda environment from /app/studio/app/optinist/wrappers/caiman/conda/caiman.yaml:\n
Command:\n
conda env create --quiet --file "/app/.snakemake/conda/23a0fcc13df369566a15cf65d2b25641_.yaml" --prefix "/app/.snakemake/conda/23a0fcc13df369566a15cf65d2b25641_"\n
Output:\n
Collecting package metadata (repodata.json): ...working... \n
', 'timestamp': 1714061661.5794506}
itutu-tienday commented 1 month ago

Probably due to lack of memory in the execution environment (ECS container). Increased memory (from 3GB to 8GB) and confirmed that the error was suppressed.