isolates (7 off) won't process repeatedly

[sharbie88]

I've run Nullarbor successfully a few times, including all of these isolates except one, but these 7 isolates (of a total 402) don't seem to be processing and appear to be stopping the program this time. Their sizes range from 66.3kb to 121.1kb. I've attached the log file. nullarbor.log

[tseemann]

• You are using an older version of nullarbor, you can update with

  
  conda update -n nullarbor_env --all 

* in the log i see this error:

% grep Error nullarbor.log make: [Makefile:130: PN0247-S0386/contigs.fa] Error 137 make: [Makefile:130: PC0073a-S0073/contigs.fa] Error 137 make: [Makefile:130: PN0231-S0373/contigs.fa] Error 137 make: [Makefile:130: PN0247-S0386/contigs.fa] Error 137 make: *** [Makefile:130: PC0073a-S0073/contigs.fa] Error 137

Maybe remove those 5 isolates from the `--input` file and go into nullarbor folder and type `make again` then `make -j 2` to continue.
(I suspect these isolates are too low yield and won't assemble.)

* It seems you have some abberant spaces in some IDs

[23:44:18] Renamed isolate: PC0012-S0012 => PC0012-S0012 [23:44:18] Renamed isolate: PC0063-S0063 => PC0063-S0063

[tseemann]

The way to avoid this and QC your data quickly before the big long run is to use the Preview mode:

See https://github.com/tseemann/nullarbor#quick-preview-mode

For best results, make sure you conda update to the latest version.

[sharbie88]

Thanks. I've done the updates as you suggested.

The log error files you mentioned aren't any of the 7 that I noticed 'come up' at the start of each repeat - they are 41, 51, 122, 141, 191, 394, 402. Am I making a connection here that isn't relevant?

Found and fixed up the abberant spaces.

I ran the quick-preview suggestions and checked out the makefile - was I just looking for errors? the 5 isolates above looked the same as every other isolate listed, so I went ahead and ran it.

Thanks again.

[tseemann]

When you upgrade nullarbor you will need to rm -f */{resistome,virulome}.tab and make again The read QC table in the preview mode should suggest isolates with low depth to remove. Things are repeatedly failing for a reason, but sometimes hard to pin down. It's standard that 5% of isolates in a large study are dodgy and need to be removed.

[sharbie88]

Sorry, I need a bit of help here please. I created the Makefile, no 'errors' included, and ran it (outdir: 402_NZAK3_191023). But errors resulted. From the log file of that run (attached), I get

make: *** [Makefile:118: roary/pan.svg] Error 2
make: *** Deleting file 'roary/pan.svg'

nullarbor.zip

Then I did rm -f */{resistome,virulome}.tab in nullarbor_env, and also within the outdir (402_NZAK3_191023) - because I wasn't sure where to do it and there's no feedback after the command to say it's happened or not.

So going back to your Preview mode suggestion... the rough tree looked good except for S0080, which I've removed. But no QC table in the html file. Where is the read QC table in preview mode?

[tseemann]

The rm -f is meant to be run from the --outdir folder you specified. So go into that folder and do this:

rm -fv */resistome.tab
rm -fv */virulome.tab
rm -f *.gff
rm -fr roary
find . -type f -size 0 -delete -print
make

[sharbie88]

Thanks. I did the above 6 lines. The first two lines removed a heap of files. Did make - still the same result. I also tried the nullarbor.pl --force --run --name NAME --ref REF.fasta --input INPUT.tab --outdir OUTDIR method - same result. Log file attached. Thanks.

nullarbor.zip

[tseemann]

@sharbie88 the final line of your log file shows an error:

Can't locate File/Slurp.pm in @INC (you may need to install the File::Slurp module) (@INC contains: /home/sharb/anaconda3/envs/nullarbor_env/lib/site_perl/5.26.2/x86_64-linux-thread-multi /home/sharb/anaconda3/envs/nullarbor_env/lib/site_perl/5.26.2 /home/sharb/anaconda3/envs/nullarbor_env/lib/5.26.2/x86_64-linux-thread-multi /home/sharb/anaconda3/envs/nullarbor_env/lib/5.26.2 .) at /home/sharb/anaconda3/envs/nullarbor_env/bin/roary2svg.pl line 8.
BEGIN failed--compilation aborted at /home/sharb/anaconda3/envs/nullarbor_env/bin/roary2svg.pl line 8.
make: *** [Makefile:118: roary/pan.svg] Error 2
make: *** Deleting file 'roary/pan.svg'
make: Leaving directory '/home/sharb/Documents/Nullarbor/402_NZAK3_191023'

You haven't installed nullarbor correctly somehow. The perl File::Slurp module is missing. (this is not needed on the current version). Please upgrade your nullarbor with the command i gave you in a previous response. conda update -n nullarbor_env --all or start a new env with conda create -n nullarbor2_env nullarbor=2.0.20191013. I am unable to support old versions.

[sharbie88]

Thanks. I'd used conda update -n nullarbor_env --all repeatedly, but obviously didn't work. I started a new env as suggested and ran make again. Getting closer.

nw_display -S -s -w 1024 -l 'font-size:12;font-family:sans-serif;' -i 'opacity:0' -b 'opacity:0' -v 16 roary/accessory_binary_genes.fa.newick > roary/acc.svg
nullarbor-report.pl --name 402_NZAK3_191023 --indir . --outdir report
[20:05:22] Hello sharb
[20:05:22] This is nullarbor-report.pl 2.0.20191013
[20:05:22] Send complaints to Torsten Seemann
[20:05:22] Making folder --outdir /home/sharb/Documents/Nullarbor/402_NZAK3_191023/report
[20:05:22] Identified 401 isolates.
[20:05:22] Generating: jobinfo
[20:05:22] Generating: seqdata
[20:05:22] Generating: identification
[20:05:23] Generating: mlst
[20:05:23] Generating: serotype
[20:05:23] WARNING: no analysis available for serotype
[20:05:23] Generating: resistome
[20:05:23] ERROR: Can't open /home/sharb/Documents/Nullarbor/402_NZAK3_191023/resistome.tab in Nullarbor::Tabular::load
make: *** [Makefile:72: report/index.html] Error 1

nullarbor.zip

How do I uninstall the first nullarbor_env? Do I just drag it into the bin?

[sharbie88]

Torsten, I went back to use preview mode after upgrading to Nullarbor 2.0.20191013 and no Report folder is created. I could view the tree though using the .nwk file. But which is the read QC file? Using the information on the terminal window, e.g.

Writing preview.msh...
cat isolates.txt | wc -l > preview.mat
mash dist -p 4 -t preview.msh preview.msh | grep -v '^#' | sed 's/\/R1.fq.gz//' >> preview.mat
quicktree -in m -out t preview.mat | nw_order -c n - > preview.nwk
nw_display -S -s -w 1024 -l 'font-size:12;font-family:sans-serif;' -i 'opacity:0' -b 'opacity:0' -v 16 preview.nwk > preview.svg
fq --ref ref.fa PN0164-S0331/R1.fq.gz PN0164-S0331/R2.fq.gz > PN0164-S0331/yield.tab
[fq] running command: cat PN0164-S0331/R1.fq.gz PN0164-S0331/R2.fq.gz | seqtk fqchk -q0 -
[fq] processed 4260648 reads.
[fq] calculating depth, using size  (via --ref ref.fa)
Illegal division by zero at /home/sharb/anaconda3/envs/nullarbor2_env/bin/fq line 67, <$IN> line 154.
make: *** [Makefile:136: PN0164-S0331/yield.tab] Error 2
make: *** Deleting file 'PN0164-S0331/yield.tab'

Is this telling me that there isn't enough read depth to PN0164-S0331 and that I shouldn't use it? (even though I've used it before and it worked fine)

I also noticed that other users had the same issue as mine above with the report/index.html error, but no solutions were available. Any help would be greatly appreciated please.

[tseemann]

No report will be create dif there are errors along the way.

It means your reference file has no sequence in it, or the file is 0 bytes long.

What is your reference file? Please attach the nullarbor.log file.

[sharbie88]

Reference file is NCBI RefSeq NZ_LT009690.1. The (renamed) NZAK3.fasta file is 2.9MB and it's the same one I've been using all along for these isolates.

This is the nullarbor.log file from the preview attempt nullarbor.log

This is the nullarbor.log file from the run after updating to nullarbor=2.0.191013 nullarbor.zip

Thanks.

[tseemann]

The error is still there:

Can't locate File/Slurp.pm in @INC (you may need to install the File::Slurp module) (@INC contains: /home/sharb/anaconda3/envs/nullarbor_env/lib/site_perl/5.26.2/x86_64-linux-thread-multi /home/sharb/anaconda3/envs/nullarbor_env/lib/site_perl/5.26.2 /home/sharb/anaconda3/envs/nullarbor_env/lib/5.26.2/x86_64-linux-thread-multi /home/sharb/anaconda3/envs/nullarbor_env/lib/5.26.2 .) at /home/sharb/anaconda3/envs/nullarbor_env/bin/roary2svg.pl line 8.
BEGIN failed--compilation aborted at /home/sharb/anaconda3/envs/nullarbor_env/bin/roary2svg.pl line 8.

You should not be getting this error, because roary2svg.pl does not use File::Slurp in recent versions. See https://github.com/tseemann/nullarbor/blob/master/bin/roary2svg.pl

Your log file says you aren't using the current version:

[12:08:21] This is nullarbor.pl 2.0.20181010

Do this:

# start a BRAND NEW LOGIN SHELL, THEN:
rm -fr /home/sharb/anaconda3/envs/nullarbor_env
conda update --all

cat ~/.condarc
# IT SHOULD LOOK LIKE THIS:
#channels:
#  - conda-forge
#  - bioconda
#- defaults

conda create -n nullarbor_env nullarbor=2.0.20191013
conda activate nullarbor_env
cd /your/nulla/folder
make

[sharbie88]

I removed the nullarbor_env and conda update --all produced the following (I've attached a copy of the whole terminal window message):

RemoveError: This operation will remove conda without replacing it with
another version of conda.

conda_update_all_191030.txt

The channel order was correct:

channels:
  - conda-forge
  - bioconda
  - defaults

I haven't updated nullarbor yet.

[tseemann]

I can't explain that error sorry. I use miniconda3 not anaconda, as do most of the users i encounter. Instructions here https://bioconda.github.io/user/install.html

[ajmerritt1]

It might be redundant given that a change to miniconda3 is best advice but you can try:

conda update -c bioconda perl Or

conda remove -c bioconda perl Then conda install -c bioconda perl

Source is somewhat similar issue. https://stackoverflow.com/questions/54836416/cant-locate-file-slurp-pm-in-inc-in-a-conda-installed-program

conda remove -c bioconda perl was reported to resolve the problem but it was not stated which environment to run this in so YMMV.

Best of luck

[sharbie88]

Thanks Torsten.

1. From your response, my first preference would be to change to miniconda if that's the recommended way (for preventing future issues), so I'll follow the instructions in your link. Regarding what's already in anaconda3, what's your advice on removal or conflict with a miniconda installation?

2. In the meantime, I removed and reinstalled bioconda perl in my base environment and in anaconda3. . which perl gave me: /home/sharb/anaconda3/bin/perl

   perl -V gave me:

Built under linux
  Compiled at Apr 18 2019 16:12:45
  @INC:
    /home/sharb/anaconda3/lib/site_perl/5.26.2/x86_64-linux-thread-multi
    /home/sharb/anaconda3/lib/site_perl/5.26.2
    /home/sharb/anaconda3/lib/5.26.2/x86_64-linux-thread-multi
    /home/sharb/anaconda3/lib/5.26.2

and env -i perl -V gave the last few lines as:

Built under linux
  Compiled at Nov 19 2018 15:54:44
  @INC:
    /etc/perl
    /usr/local/lib/x86_64-linux-gnu/perl/5.26.1
    /usr/local/share/perl/5.26.1
    /usr/lib/x86_64-linux-gnu/perl5/5.26
    /usr/share/perl5
    /usr/lib/x86_64-linux-gnu/perl/5.26
    /usr/share/perl/5.26
    /usr/local/lib/site_perl
    /usr/lib/x86_64-linux-gnu/perl-base

could lines 7 and 8 perl5 be causing any issues?? I'm just guessing here.

And conda update --all still gives me

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: failed

RemoveError: This operation will remove conda without replacing it with
another version of conda

BTW I'm using Ubuntu.

Thanks.

[tseemann]

Sorry, I can not fix your software installations. If you have your computer and conda correctly setup then nullarbor will install fine.I would rm -fr $HOME/anaconda3 and delete all the lines from ~/.bashrc or ~/.bash_profile related to anaconda3. Then install everything in miniconda3 from scratch.

If you encounter a new/different problem feel free to file a new issue.