[Bug]: Directory Issue?

[CESTechSupport]

What happened?

The run keeps stopping due to some sort of directory error. After 1-2 runs, the following error shows up:

/bin/bash: .command.run: No such file or directory

To have this working, we had to delete the workflow and restart wsl (wsl --shutdown). Then, we had to restart the computer and re-install the workflow to get it back to work.

After 1-2 runs, we have to repeat those steps again.

Operating System

Windows 11

Workflow Execution

EPI2ME Labs desktop application

Workflow Execution - EPI2ME Labs Versions

v5.0.2

Workflow Execution - CLI Execution Profile

None

Workflow Version

v0.3.1

Relevant log output

This is epi2me-labs/wf-clone-validation v0.3.1.
--------------------------------------------------------------------------------
Checking fastq input.
WARN: Overriding Medaka model with r1041_e82_400bps_sup_g615.
[7a/73a50b] Submitted process > fastcat (13)
[7d/7ffa63] Submitted process > fastcat (2)
ERROR ~ Error executing process > 'fastcat (13)'
Caused by:
  Process `fastcat (13)` terminated with an error exit status (127)
Command executed:
  mkdir fastcat_stats
  fastcat             -s barcode14             -r fastcat_stats/per-read-stats.tsv             -f fastcat_stats/per-file-stats.tsv                          barcode14             | bgzip -@ 4 > seqs.fastq.gz
Command exit status:
  127
Command output:
  (empty)
Command error:
  /bin/bash: .command.run: No such file or directory
Work dir:
  /mnt/wsl/docker-desktop-bind-mounts/Ubuntu/fdeace0c97c37e435605c4867be6a665446a517252e505ab6a2b1207e75ae9a7/wsl/docker-desktop-bind-mounts/Ubuntu/b05e93a5e61410f1e9713345a81d648218bf4e2f993a03f8a26ba471543ad953/instances/wf-clone-validation_1c51e7e5-3f24-4ba1-a895-dee7d4209646/work/7a/73a50b0ace95631d20768dd0c60b79
Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
 -- Check '/mnt/wsl/docker-desktop-bind-mounts/Ubuntu/fdeace0c97c37e435605c4867be6a665446a517252e505ab6a2b1207e75ae9a7/wsl/docker-desktop-bind-mounts/Ubuntu/b05e93a5e61410f1e9713345a81d648218bf4e2f993a03f8a26ba471543ad953/instances/wf-clone-validation_1c51e7e5-3f24-4ba1-a895-dee7d4209646/nextflow.log' file for details
WARN: Killing running tasks (1)

[sarahjeeeze]

Hi, thanks for this feedback. We will try to recreate this error and get back to you soon.

[sarahjeeeze]

Hi, we have updated the application and it should now be more stable on windows - so it may be worth trying and seeing if the error still occurs. https://labs.epi2me.io/downloads/

[CESTechSupport]

We will try the new version, thanks for the update!

[CESTechSupport]

Hi, we installed the new version. Unfortunately, there are other issues.

The analysis is currently stuck at 'assmebleCore' and it looks like the analysis still running?

So our team tried to re-install the workflow and EPI2ME and this keeps showing up. Is there any way to fix this?

[sarahjeeeze]

Hi, did the workflow exit? Assemble core will retry a few times if it struggles to assemble. If you go to the overview page and the report issue button could you send the issue export file to support@nanoporetech.com or click that copy issue and it should provide a json which you could send.

[CESTechSupport]

I believe that my team already sent the error file to that email already, and they suggest we should follow-up with the Devs.

I think the workflow is still shown as running (see previous image), even though we deleted EPI2ME and re-installed it. But it is not allowing us to re-install the workflow due to that Error running nextflow pull.

Maybe we need to completely remove everything and try again? We installed the prerequisites via EPI2ME labs (WSL, JAVA and Nextflow), only Docker separately.

[sarahjeeeze]

I will investigate where your email went as it would be useful to see the files. We only just released this version of the platform so would be good to be aware of any issues. I would try going to the epi2melabs directory in your user area eg. C://username/epi2melabs workflows and if there are any older versions of the workflow present then just delete those directories and relaunch the application and try installing the workflow again.

[sarahjeeeze]

In the meantime you could also attach the logs to this issue

[CESTechSupport]

Yes, there was an old workflow folder and re-installation was done successfully after deleting the older version.

Everything went well for the first run, but there was an error when generating the report.

ERROR ~ Error executing process > 'pipeline:report (1)'
Caused by:
  Process `pipeline:report (1)` terminated with an error exit status (1)
Command executed:
  workflow-glue report     --downsampled_stats downsampled_stats/*     --revision null     --commit null     --status final_status.csv     --per_barcode_stats per_barcode_stats/*     --host_filter_stats host_filter_stats/*     --params params.json     --versions versions     --report_name wf-clone-validation-report.html     --plannotate_json plannotate_report.json     --lengths plannotate.json     --inserts_json insert_data.json     --qc_inserts qc_inserts/*     --assembly_quality assembly_quality/*
Command exit status:
  1
Command output:
  (empty)
Command error:
  /home/epi2melabs/conda/lib/python3.8/site-packages/icon_font_to_png/icon_font.py:39: DeprecationWarning: invalid escape sequence \.
    is_icon = re.compile("\.(.*):before,?")
  /home/epi2melabs/conda/lib/python3.8/site-packages/tinycss/token_data.py:216: DeprecationWarning: invalid escape sequence \`
    """A single atomic token.
  /home/epi2melabs/conda/lib/python3.8/site-packages/si_prefix/__init__.py:44: DeprecationWarning: invalid escape sequence \s
    u'(?P<si_unit>[%s])?\s*' % SI_PREFIX_UNITS)
  /home/epi2melabs/conda/lib/python3.8/site-packages/si_prefix/__init__.py:249: DeprecationWarning: invalid escape sequence \s
    u'(?P<si_unit>[%s])?\s*$' % SI_PREFIX_UNITS)
  /home/epi2melabs/conda/lib/python3.8/site-packages/plannotate/annotate.py:218: DeprecationWarning: invalid escape sequence \|
    problem_name = "pdb\|(.*)\|"
  [13:18:44 - workflow_glue] Starting entrypoint.
  2023-08-02 13:18:44.680 INFO    workflow_glue: Starting entrypoint.
  Traceback (most recent call last):
    File "/mnt/c/Users/CES/epi2melabs/workflows/epi2me-labs/wf-clone-validation/bin/workflow-glue", line 7, in <module>
      cli()
    File "/mnt/c/Users/CES/epi2melabs/workflows/epi2me-labs/wf-clone-validation/bin/workflow_glue/__init__.py", line 72, in cli
      args.func(args)
    File "/mnt/c/Users/CES/epi2melabs/workflows/epi2me-labs/wf-clone-validation/bin/workflow_glue/report.py", line 118, in main
      pd.DataFrame(seq_summary['sample_name'].value_counts())
    File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/util/_decorators.py", line 311, in wrapper
      return func(*args, **kwargs)
    File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/frame.py", line 6393, in sort_index
      return super().sort_index(
    File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/generic.py", line 4544, in sort_index
      indexer = get_indexer_indexer(
    File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/sorting.py", line 91, in get_indexer_indexer
      indexer = nargsort(
    File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/sorting.py", line 391, in nargsort
      return items.argsort(ascending=ascending, kind=kind, na_position=na_position)
    File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/arrays/base.py", line 628, in argsort
      return nargsort(
    File "/home/epi2melabs/conda/lib/python3.8/site-packages/pandas/core/sorting.py", line 403, in nargsort
      indexer = non_nan_idx[non_nans.argsort(kind=kind)]
  TypeError: '<' not supported between instances of 'int' and 'str'
Work dir:
  /mnt/c/Users/CES/epi2melabs/instances/wf-clone-validation_ebd5baba-f59c-4487-8e8f-b806f3b622fb/work/91/c48b09b950606bcf678cb61e66e06a
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
 -- Check '/mnt/c/Users/CES/epi2melabs/instances/wf-clone-validation_ebd5baba-f59c-4487-8e8f-b806f3b622fb/nextflow.log' file for details

When we ran the workflow for the second time, we got this directory issue again.

Command output:
  (empty)
Command error:
  /bin/bash: .command.run: No such file or directory
Work dir:
  /mnt/wsl/docker-desktop-bind-mounts/epi2me/fdeace0c97c37e435605c4867be6a665446a517252e505ab6a2b1207e75ae9a7/wsl/docker-desktop-bind-mounts/epi2me/c8f415b33a428a04f4f6bb3949b0c426eb50ce509c9d66fd83637107ce6ae7f4/instances/wf-clone-validation_866ff30b-d4ee-4411-b25a-a484e3a1ba2c/work/c7/4bf80d0f0eeec7df63cea3423ed8a9
Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
 -- Check '/mnt/wsl/docker-desktop-bind-mounts/epi2me/fdeace0c97c37e435605c4867be6a665446a517252e505ab6a2b1207e75ae9a7/wsl/docker-desktop-bind-mounts/epi2me/c8f415b33a428a04f4f6bb3949b0c426eb50ce509c9d66fd83637107ce6ae7f4/instances/wf-clone-validation_866ff30b-d4ee-4411-b25a-a484e3a1ba2c/nextflow.log' file for details
WARN: Killing running tasks (1)

So we had to do the same thing again; delete the workflow, shutdown wsl and reinstall the workflow. The third run was successful without any issues. I think the directory issue is still remaining.

[sarahjeeeze]

ok thanks for the feedback, we will investigate soon. So did you get a report after all that?

[CESTechSupport]

We tried a few more times, we could not get a report for that specific batch. But other batches worked perfectly fine after deleting+reinstalling process.

[CESTechSupport]

Hi Sarah, just a quick update on our issue. Everything is working properly now, no issues and no errors. We don't know what we did, but it looks like that directory issue is gone for now.

[sarahjeeeze]

ok let us know if it returns

[sarahjeeeze]

Closing for now as this was resolved, feel free to open another issue if any other problems.