Sensible handling of empty FASTQs

[bede]

Should these fail validation or be skipped (with an apt warning to stderr) at upload time? Plus message for EC

% gpas upload --token token.json fastq.csv
INFO: Validation successful
INFO: Using 12 processes
INFO: Finished decontaminating 2 sample(s)                                                                                            
Traceback (most recent call last):
  File "/Users/bede/miniconda3/envs/gpas-cli-dev/bin/gpas", line 33, in <module>
    sys.exit(load_entry_point('gpas', 'console_scripts', 'gpas')())
  File "/Users/bede/Research/Git/gpas-cli/src/gpas/cli.py", line 266, in main
    defopt.run(
  File "/Users/bede/miniconda3/envs/gpas-cli-dev/lib/python3.10/site-packages/defopt.py", line 275, in run
    return func(*ba.args, **ba.kwargs)
  File "/Users/bede/Research/Git/gpas-cli/src/gpas/cli.py", line 129, in upload_wrapper
    jsonify_exceptions(
  File "/Users/bede/Research/Git/gpas-cli/src/gpas/misc.py", line 122, in jsonify_exceptions
    return function(**kwargs)
  File "/Users/bede/Research/Git/gpas-cli/src/gpas/cli.py", line 98, in upload
    batch.upload(dry_run=dry_run)
  File "/Users/bede/Research/Git/gpas-cli/src/gpas/lib.py", line 857, in upload
    self._decontaminate()
  File "/Users/bede/Research/Git/gpas-cli/src/gpas/lib.py", line 559, in _decontaminate
    self._parse_decontamination_stats(samples_runs)
  File "/Users/bede/Research/Git/gpas-cli/src/gpas/lib.py", line 564, in _parse_decontamination_stats
    samples_decontamination_stats = {
  File "/Users/bede/Research/Git/gpas-cli/src/gpas/lib.py", line 565, in <dictcomp>
    s: parse_decontamination_stats(r.stdout) for s, r in samples_runs.items()
  File "/Users/bede/Research/Git/gpas-cli/src/gpas/lib.py", line 899, in parse_decontamination_stats
    "fraction": round(delta / count_in, 4),

[bede]

handle empty before and after RIAK

[bede]

ZeroDivisionError

[bede]

Difficulties

• RIAK generates *.fasta.gz when processing empty *.fastq.gz (easy workaround)
• Viridian 0.3.7 throws KeyError (tricky – leaves sample in Uploaded state)

[iqbal-lab]

a) Obviously we should fix viridian to handle this better, but also b) if it's spotted after RIAK we presumably dont even send to viridian?

[bede]

Hi! Currently I'm keen on having ability to run e.g. empty negative control samples through the system so that they at least make it to the GPAS platform where they can be viewed in the portal, queried using APIs. It would be helpful if Viridian could treat them like any other rubbish sample with insufficient coverage – Martin is aware.

[iqbal-lab]

that makes sense, ok!

[bede]

Thanks. I realise informatic negative controls is OTT, but as a starting point for ensuring that all neg controls auditably make it to the platform I think it's sensible, and can be optimised later.

[iqbal-lab]

(I need to check but) Viridian will quite possibly classify this as a fail, for too many Ns. Am right that that's fine with you so long as it completes gracefully and outputs the json?

[bede]

Yes exactly : )