analysis starting from fastq

[asogg]

Hello Simone,

i started directly from fastq using this command: nohup ./MetONTIIME.sh /home/as/MetONTIIME/fastq_pass/barcode01/ /home/as/MetONTIIME/sample-metadata.tsv /home/as/MetONTIIME/silva_132_99_16S_sequence.qza /home/as/MetONTIIME/silva_132_99_16S_taxonomy.qza 24 Blast 1 0.9 0.9

the process run indefinitely and i have to stop , this is the nohup.out content:

Imported /home/as/MetONTIIME/fastq_pass/barcode01//manifest.txt as SingleEndFastqManifestPhred33V2 to sequences.qza Saved FeatureTable[Frequency] to: table_tmp.qza Saved FeatureData[Sequence] to: rep-seqs_tmp.qza Saved FeatureTable[Frequency] to: table.qza Saved FeatureData[Sequence] to: rep-seqs.qza Saved Visualization to: demux_summary.qzv Plugin error from feature-table:

The following IDs are not present in the metadata: 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_0', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_1', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_10', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_100', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_101', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_102', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_103', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_104', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_105', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_106', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_107', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_108', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_109', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_11', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_110', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_111', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_12', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_13', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_14', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_15', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_16', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_17', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_18', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_19', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_2', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_20', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_21', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_22', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_23', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_24', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_25', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_26', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_27', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_28', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_29', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_3', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_30', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_31', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_32', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_33', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_34', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_35', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_36', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_37', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_38', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_39', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_4', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_40', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_41', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_42', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_43', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_44', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_45', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_46', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_47', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_48', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_49', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_5', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_50', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_51', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_52', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_53', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_54', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_55', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_56', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_57', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_58', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_59', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_6', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_60', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_61', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_62', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_63', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_64', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_65', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_66', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_67', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_68', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_69', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_7', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_70', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_71', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_72', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_73', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_74', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_75', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_76', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_77', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_78', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_79', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_8', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_80', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_81', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_82', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_83', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_84', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_85', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_86', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_87', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_88', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_89', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_9', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_90', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_91', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_92', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_93', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_94', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_95', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_96', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_97', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_98', 'FAQ57575_pass_barcode01_1d1a0042_8d1a0ba4_99'

Debug info has been saved to /tmp/qiime2-q2cli-err-g0s1lng_.log Saved Visualization to: rep-seqs.qzv

what's wrong for you? grazie mille Alessio

[MaestSi]

Ciao, are you using the latest version of MetONTIIME? If yes, the syntax you used is not correct, please look at the README for the correct command. SM

[asogg]

Ciao Simone. I have to check, there is a command like --version?

[MaestSi]

Unfortunately no, but you can look at version.txt file in the MetONTIIME repository you downloaded. In any case, I would suggest updating to the latest version, if you are not already using it. SM

[asogg]

Hello Simone, the version is 1.14.0, i'll check the update and i'll came back grazie A

[asogg]

Hi Simone, I use git clone https://github.com/MaestSi/MetONTIIME.git cd MetONTIIME chmod 755 * ./install.sh

On the old installation o I have to make a new env deleting the old one

[MaestSi]

Hi, if you don't have to compare your analyses with some analyses you already did, I'd suggest deleting the MetONTIIME_env environment and creating a new one. Otherwise, it would be enough to clone the repository, and change file permissions with chmod. In the latter case, you would be ready to go with the analysis, just remember to use the new syntax described in the README. SM

[asogg]

thank you for the suggestion Simone , i'll try to analyze the data with the new version and correct syntax. grazie A

[asogg]

Hi, i used this command: nohup ./MetONTIIME.sh -w ./fastq_pass/barcode01 -f /home/as/sample-metadata.tsv -s ./silva_132_99_16S_sequence.qza -t ./silva_132_99_16S_taxonomy.qza -n 24 -c Vsearch -m 1 -q 0.9 -i 0.9

the analysis stopped without the taxa bar plot file and with some errors , see attached files nohup.txt

[MaestSi]

Ciao, the error is: no such option: --o-search-results This is due to the fact that you probably did not reinstall MetONTIIME_env from scratch, and you are therefore using an older QIIME2 version, where that parameter was not available yet. If you want to keep working with that QIIME2 version, you should also delete: --o-search-results search_results.qza at lines 148 and 163 of MetONTIIME.sh script. Best, SM

[asogg]

Thanks! I'll run the pipeline following your suggestions and I' ll came back with the results. Best Alessio

[asogg]

Hello Simone, i deleted the lines 148 and 163 but without success. i'll try to reinstall from scratch MetONTIIME_env. i have to delete only the MetONTIIME_env right? thanks for the support Alessio

[MaestSi]

Yes, but what does "without success" mean? Would you like to share with me the new error? SM

[asogg]

Sorry Simone, but erroneously i have eliminated the file. in the meantime i have deleted the MetONTIIME_env and reinstalled as indicated in your page . Now i am testing the pipeline on barcode 01. about this, is it possible to analyze all the twelve barcodes simultaneously only indicating the folder or i have to specify other parameters. best Alessio

[MaestSi]

Ciao Alessio, it's enough you put all fastq.gz files in the directory you specify with -w parameter. In case you do not have a sample-metadata.tsv file with additional metadata, be sure not to provide with -f parameter the one that was generated with barcode01 only, before running the pipeline with all the samples. SM

[asogg]

Ciao Simone, unfortunately the computer crashed last night but in the nohup file there are no error messages (i attach it). i also have the metadata file that i attach and the command line that i use: nohup ./MetONTIIME.sh -w ./fastq -f /home/as/sample-metadata.tsv -s ./silva_132_99_16S_sequence.qza -t ./silva_132_99_16S_taxonomy.qza -n 24 -c Blast -m 1 -q 0.9 -i 0.9

Regarding the speed, is blast or vsearch faster, because it seems to me very slow despite having skipped the basecalling part. sample-metadata.xlsx there is no gpu based solution or an alternative to blast like diamond? nohup.txt grazie mille A

p.s each fastq is about 1GB

[MaestSi]

Hi, based on the nohup.out file, the software did not crash and is still running. Why do you think it crashed? Vsearch is much faster than Blast, since Blast implementation in QIIME2 is single-threaded. SM

[asogg]

Hello, the pc freezed (also the mouse, maybe not for the pipeline) and i had to reset. Ok i restart the pipeline with Vsearch. see you later . thanks A

[asogg]

Ciao Simone, the samples are running for 10 days with MetONTIIME starting from fastq files ( so excluding basecalling) and detected are 5800000 reads about. The workstation is running with 24 threads and 128GB of ram on ssd nvme, I don't know if this computation time is normal or abnormal. For that, do you have statistics on average computational times using Vsearch and in case if there are faster classifiers. best Alessio

[MaestSi]

Hi, unfortunately VSEARCH classifier is quite slow (but QIIME2 implementation of Blast is even slower), and 5.8M reads is definitely a huge number. I don't have reliable time estimates but, usually, I tend to work with up to 100k reads per sample and obtain results in a few days. You may downsample your reads following this command, after editing the number of downsampled reads per sample to the appropriate value, and restart the analysis (or run it in parallel, if you can). Best, SM

[asogg]

Hello, thanks for the suggestion. I'll try to downsample the reads to reduce the computation time. I think that it is possible to close the issue (eventually i'll open a new one if i'll have issues). all the best Alessio

[MaestSi]

Perfect! Best, Simone