table_collapsed_relfreq_level7.qza' does not exist

[Jawen-D]

Hello,

I have produced successful run before using 2 samples. However when I use more than 2 samples the analysis stops when collapsed feature tables folder is created with empty tsv files inside. nohup file has this error message:

'table_collapsed_relfreq_level7.qza' does not exist. mv: cannot stat 'feature-table.biom': No such file or directory Usage: biom convert [OPTIONS] Try 'biom convert -h' for help.

Error: Invalid value for '-i' / '--input-fp': File 'feature-table_relfreq_level7.biom' does not exist. cat: feature-table_absfreq_level7.tsv: No such file or directory cat: feature-table_absfreq_level7.tsv: No such file or directory mv: cannot stat 'tablecollapsed*': No such file or directory mv: cannot stat 'feature-tablefreq_level.biom': No such file or directory

Here's the script I used: nohup /home/ihopelab/Microbiome/0809/data/MetONTIIME/MetONTIIME.sh -w /home/ihopelab/Microbiome/0809/data/MetONTIIME_test/00_all_group/1122_control_V_patient/ -f /home/ihopelab/Microbiome/0809/data/MetONTIIME_test/00_all_group/1122_control_V_patient/sample-metadata.tsv -s /home/ihopelab/Microbiome/0809/data/MetONTIIME_test/00_all_group/1122_control_V_patient/silva_132_99_16S_sequence.qza -t /home/ihopelab/Microbiome/0809/data/MetONTIIME_test/00_all_group/1122_control_V_patient/silva_132_99_16S_taxonomy.qza -n 8 -c Vsearch -m 3 -q 0.8 -i 0.85 &

I need help.

Thank you.

[MaestSi]

Hi @Jawen-D , were feature tables at levels < 7 produced? My guess is that none of your reads was classified at species level. Could you please upload to QIIME Viewer file taxa-bar-plots.qzv and check this out? Were you also analysing the previous 2 samples you wrote about in this run, together with the new samples? I guess you were not. Moreover, could you please run the following lines of code one after the other, and tell me which is the first one failing?

cd /home/ihopelab/Microbiome/0809/data/MetONTIIME_test/00_all_group/1122_control_V_patient/
lev=7
qiime taxa collapse \
  --i-table table.qza --i-taxonomy taxonomy.qza \
  --p-level $lev \
  --o-collapsed-table table_collapsed_absfreq_level$lev.qza

qiime metadata tabulate  \
  --m-input-file table_collapsed_absfreq_level$lev.qza  \
  --o-visualization table_collapsed_absfreq_level$lev.qzv

  qiime tools export \
  --input-path table_collapsed_absfreq_level$lev.qza \
  --output-path .

  mv feature-table.biom feature-table_absfreq_level$lev.biom

  biom convert \
  -i feature-table_absfreq_level$lev.biom \
  -o feature-table_absfreq_level$lev.tsv \
  --to-tsv \
  --table-type 'Taxon table'

  qiime feature-table relative-frequency \
  --i-table table_collapsed_absfreq_level$lev.qza \
  --o-relative-frequency-table table_collapsed_relfreq_level$lev.qza

  qiime metadata tabulate  \
  --m-input-file table_collapsed_relfreq_level$lev.qza  \
  --o-visualization table_collapsed_relfreq_level$lev.qzv

  qiime tools export \
  --input-path table_collapsed_relfreq_level$lev.qza \
  --output-path .

  mv feature-table.biom feature-table_relfreq_level$lev.biom

  biom convert \
  -i feature-table_relfreq_level$lev.biom \
  -o feature-table_relfreq_level$lev.tsv \
  --to-tsv \
  --table-type 'Taxon table'

Best, Simone

[Jawen-D]

< 7 produced?

It produced a folder "collapsed_feature_tables' and it contains feature-table_absfreq_level1_stringent.tsv to level 7 inside but they are all empty.

Also, no taxa-bar-plots.qzv were produced. Only sample-metada.tsv, demux_summary.qzv, manifest.txt and sequences.qza were produced before the analysis stopped due to the error.

Thanks

[MaestSi]

So, if those files were not produced, my guess is that either the manifest.txt or the sample-metadata.tsv file contain some wrong information. Could you check those files only contain the samples you are supposed to analyse? SM

[Jawen-D]

Could you check those files only contain the samples you are supposed to analyse?

Yes, they contain all the samples I'm supposed to analyse.

[MaestSi]

Then there should probably be another error in the nohup.out file before the one you reported. Could you please check or share the file with me? Thanks, SM

[Jawen-D]

Here's the nohup file.

nohup.txt

[MaestSi]

Here is the relevant error:

Plugin error from vsearch:

  Command '['vsearch', '--cluster_size', '/tmp/tmpne8kbykx', '--id', '1', '--centroids', '/tmp/q2-DNAFASTAFormat-cvdo2ntf', '--uc', '/tmp/tmpiof2oc94', '--qmask', 'none', '--xsize', '--threads', '8', '--minseqlength', '1', '--fasta_width', '0']' died with <Signals.SIGKILL: 9>.

The process qiime vsearch dereplicate-sequences was killed. This may be due, for example, to insufficient RAM memory available. SM

[Jawen-D]

Thanks for the swift response. May I know how should I proceed with this?

I'm thinking of restarting our server and rerunning the analysis again. is there any changes that I should make with the script or my process?

[MaestSi]

No, I don't think you should make any changes to the script. I would just try again with the same options and then, if it fails again, try reducing the number of threads. Remember to delete all files (a part from fastq.gz files) before restarting the analysis. SM

[Jawen-D]

Thank you for your advise. I will rerun my analysis for now after restarting our server and then I will give you an update once the analysis is done.

Thanks

[MaestSi]

Hi @Jawen-D , are there any updates on this? Best, Simone

[Jawen-D]

Hello MaestSi/MetONTIIME,

I tried lowering the core number and it worked. But on the other server with multiple users it still show the same error message.

Thank you for your advise.

On Wed, Dec 21, 2022 at 17:16 Simone Maestri @.***> wrote:

Hi @Jawen-D https://github.com/Jawen-D , are there any updates on this? Best, Simone

— Reply to this email directly, view it on GitHub https://github.com/MaestSi/MetONTIIME/issues/54#issuecomment-1360988781, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUXMW5RHMRNHFCWOMQNVHXTWOK4D3ANCNFSM6AAAAAASNAE55I . You are receiving this because you were mentioned.Message ID: @.***>

[MaestSi]

Thanks for the update. Best, Simone

[Jawen-D]

Hello MaestSi,

I keep having this error:

error:

'table_collapsed_relfreq_level7.qza' does not exist.

mv: cannot stat 'feature-table.biom': No such file or directory

Usage: biom convert [OPTIONS]

Try 'biom convert -h' for help.

Error: Invalid value for '-i' / '--input-fp': File 'feature-table_relfreq_level7.biom' does not exist.

cat: feature-table_absfreq_level7.tsv: No such file or directory

cat: feature-table_absfreq_level7.tsv: No such file or directory

mv: cannot stat 'table_collapsed*': No such file or directory

mv: cannot stat 'feature-table_freq_level.biom': No such file or directory

This is the script that I run:

nohup ~/installers/MetONTIIME/MetONTIIME/MetONTIIME.sh -w ~/ihopelab_files/01_raw_samples_29/29samples/1226_2samples/ -f ~/ihopelab_files/01_raw_samples_29/29samples/1226_2samples/sample-metadata.tsv -s ~/ihopelab_files/01_raw_samples_29/29samples/1226_2samples/silva_132_99_16S_sequence.qza -t ~/ihopelab_files/01_raw_samples_29/29samples/1226_2samples/silva_132_99_16S_taxonomy.qza -n 12 -c Vsearch -m 3 -q 0.8 -i 0.85 &

On Mon, Nov 28, 2022 at 17:20 Simone Maestri @.***> wrote:

Hi @Jawen-D https://github.com/Jawen-D , were feature tables at levels < 7 produced? My guess is that none of your reads was classified at species level. Could you please upload to QIIME Viewer https://view.qiime2.org file taxa-bar-plots.qzv and check this out? Were you also analysing the previous 2 samples you wrote about in this run, together with the new samples? I guess you were not. Moreover, could you please run the following lines of code one after the other, and tell me which is the first one failing?

cd /home/ihopelab/Microbiome/0809/data/MetONTIIME_test/00_all_group/1122_control_V_patient/ lev=7 qiime taxa collapse \ --i-table table.qza --i-taxonomy taxonomy.qza \ --p-level $lev \ --o-collapsed-table table_collapsed_absfreq_level$lev.qza

qiime metadata tabulate \ --m-input-file table_collapsed_absfreq_level$lev.qza \ --o-visualization table_collapsed_absfreq_level$lev.qzv

qiime tools export \ --input-path table_collapsed_absfreq_level$lev.qza \ --output-path .

mv feature-table.biom feature-table_absfreq_level$lev.biom

biom convert \ -i feature-table_absfreq_level$lev.biom \ -o feature-table_absfreq_level$lev.tsv \ --to-tsv \ --table-type 'Taxon table'

qiime feature-table relative-frequency \ --i-table table_collapsed_absfreq_level$lev.qza \ --o-relative-frequency-table table_collapsed_relfreq_level$lev.qza

qiime metadata tabulate \ --m-input-file table_collapsed_relfreq_level$lev.qza \ --o-visualization table_collapsed_relfreq_level$lev.qzv

qiime tools export \ --input-path table_collapsed_relfreq_level$lev.qza \ --output-path .

mv feature-table.biom feature-table_relfreq_level$lev.biom

biom convert \ -i feature-table_relfreq_level$lev.biom \ -o feature-table_relfreq_level$lev.tsv \ --to-tsv \ --table-type 'Taxon table'

Best, Simone

— Reply to this email directly, view it on GitHub https://github.com/MaestSi/MetONTIIME/issues/54#issuecomment-1328702293, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUXMW5TV7ZYRBUIYNLRNRY3WKRTOHANCNFSM6AAAAAASNAE55I . You are receiving this because you were mentioned.Message ID: @.***>

[MaestSi]

Hi @Jawen-D , are you talking about the server with a single user, right? Is this the first error in the nohup.out file? Did the script produce the taxa-bar-plots.qzv file? If yes, could you please upload it to QIIME2 View and check if any of the reads was assigned to level = 7? Best, SM

[Jawen-D]

Hello

Thanks for the swift response. I checked the nohup file it says it cant find the file path for the metadatafile: taxonomy.qza

Yes its the single user server.

On Wed, Jan 11, 2023 at 16:21 Simone Maestri @.***> wrote:

Hi @Jawen-D https://github.com/Jawen-D , are you talking about the server with a single user, right? Is this the first error in the nohup.out file? Did the script produce the taxa-bar-plots.qzv file? If yes, could you please upload it to QIIME2 View https://view.qiime2.org and check if any of the reads was assigned to level = 7? Best, SM

— Reply to this email directly, view it on GitHub https://github.com/MaestSi/MetONTIIME/issues/54#issuecomment-1378331783, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUXMW5X5X76P5NYTEURERNTWRZNOXANCNFSM6AAAAAASNAE55I . You are receiving this because you were mentioned.Message ID: @.***>

[MaestSi]

This means it probably crashed during VSearch taxonomy assignment. The issue reminds me of this one, where Vsearch crashed, probably due to RAM memory issues. Please try reducing the number of threads (-n parameter) or increasing the memory, if you are using a job scheduler. Best, Simone

[Jawen-D]

Thanks for the advice. Is there a way to salvage the results I had before it crashed and just rerun starting on that specific steps with a script?

So far the last output I had is the rep-seq_tmp.qza and table_tmp.qza

On Wed, Jan 11, 2023 at 16:57 Simone Maestri @.***> wrote:

This means it probably crashed during VSearch taxonomy assignment. The issue reminds me of this one https://github.com/MaestSi/MetONTIIME/issues/56#issuecomment-1365283219, where Vsearch crashed, probably due to RAM memory issues. Please try reducing the number of threads (-n parameter) or increasing the memory, if you are using a job scheduler. Best, Simone

— Reply to this email directly, view it on GitHub https://github.com/MaestSi/MetONTIIME/issues/54#issuecomment-1378362347, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUXMW5VCARHNAWGR62KBROTWRZRXTANCNFSM6AAAAAASNAE55I . You are receiving this because you were mentioned.Message ID: @.***>