new error in sqm2tables.py

[RafaelLopez-Sanchez]

Hi Fernando and Javier:

I've updated SQM again it works great,except for the fact there is a new error when I call the sqm2tables.py. The error says this: File "/home/install/SqueezeMeta-1.0.0/utils//sqm2tables.py", line 174, in main(parse_args()) File "/home/install/SqueezeMeta-1.0.0/utils//sqm2tables.py", line 63, in main sampleNames, orfs, kegg, cog, pfam, custom = parse_orf_table(perlVars['$mergedfile'], nokegg, nocog, nopfam, args.trusted_functions, args.ignore_unclassified, customMethods) ValueError: not enough values to unpack (expected 6, got 5) Thanks and have a nice day! Rafa

[fpusan]

Hi! Please update also the SqueezeMeta/lib/utils.py file

El vie., 17 ene. 2020 2:15, obiwanlopez1986 notifications@github.com escribió:

Hi Fernando and Javier:

I've updated SQM again it works great,except for the fact there is a new error when I call the sqm2tables.py. The error says this: File "/home/install/SqueezeMeta-1.0.0/utils//sqm2tables.py", line 174, in main(parse_args()) File "/home/install/SqueezeMeta-1.0.0/utils//sqm2tables.py", line 63, in main sampleNames, orfs, kegg, cog, pfam, custom = parse_orf_table(perlVars['$mergedfile'], nokegg, nocog, nopfam, args.trusted_functions, args.ignore_unclassified, customMethods) ValueError: not enough values to unpack (expected 6, got 5) Thanks and have a nice day! Rafa

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jtamames/SqueezeMeta/issues/57?email_source=notifications&email_token=ACAHO7SJZQ3G6GYKPQ6P6Z3Q6EBDNA5CNFSM4KH7JDJ2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IGZZN6A, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACAHO7WIGXI4NFSOO4FVI5LQ6EBDNANCNFSM4KH7JDJQ .

[RafaelLopez-Sanchez]

Hi Fernando I've already done that and now the error I get is this

File "/home/install/SqueezeMeta-1.0.0/utils//sqm2tables.py", line 174, in main(parse_args()) File "/home/install/SqueezeMeta-1.0.0/utils//sqm2tables.py", line 63, in main sampleNames, orfs, kegg, cog, pfam, custom = parse_orf_table(perlVars['$mergedfile'], nokegg, nocog, nopfam, args.trusted_functions, args.ignore_unclassifie d, customMethods) File "/home/install/SqueezeMeta-1.0.0/utils/../lib/utils.py", line 122, in parse_orf_table abundances = array([int(line[idx[sample]]) for sample in samples]) File "/home/install/SqueezeMeta-1.0.0/utils/../lib/utils.py", line 122, in abundances = array([int(line[idx[sample]]) for sample in samples]) IndexError: list index out of range

Thanks for your help

Rafa

[fpusan]

Hi! Can you send me the first lines of the /path/to/project/results/13.*.orftable file?

[RafaelLopez-Sanchez]

Hi Fernando, here is what it says:

--Created by /home/install/SqueezeMeta-1.0.0/scripts/13.mergeannot2.pl, Mon Jan 13 18:09:12 2020

ORF ID Contig ID Molecule Method Length NT Length AA GC perc Gene name Tax KEGG ID KEGGFUN KEGGPATH COG ID COGFUN COGPATH PFAM Antismash Antismash NAME CAZy CAZy NAME TPM 0hrs TPM 9hrs TPM 24hrs TPM 48hrs Coverage 0hrs Coverage 9hrs Coverage 24hrs Coverage 48hrs Raw read count 0hrs Raw read count 9hrs Raw read count 24hrs Raw read count 48hrs RAW base count 0hrs RAW base count 9hrs RAW base count 24hrs RAW base count 48hrs Hits megahit_1_1-156 megahit_1 CDS Prodigal 157 53 39.10 pphA k_Bacteria;p_Proteobacteria;c_Gammaproteobacteria;o_Pseudomonadales;f_Moraxellaceae;g_Acinetobacter K07313 serine/threonine protein phosphatase 1 [EC:3.1.3.16] Not Included in Pathway or Brite; Unclassified: metabolism; Enzymes with EC numbers COG0639 Diadenosine tetraphosphatase and related serine/threonine protein phosphatases Signal transduction mechanisms WP_049594749.1* 0 1.017 1.001 0.347 0 1.506 1.154 0.481 0 4 4 1 0 235 180 75 25 megahit_2_2-250 megahit_2 CDS Prodigal 250 84 44.18 yqeH

Rafa

[fpusan]

That's a tough one, specially since you are mixing results from different versions. It seems that a line in that file might be malformed. Please edit the SqueezeMeta/lib/utils.py file, adding the following to line 121:

print(len(line), line)

Make sure than the indentation matches the next line, using spaces, not tabs. It should look like this

            print(len(line), line)  
            abundances = array([int(line[idx[sample]]) for sample in samples])

Let's see if this gives us any new info

[RafaelLopez-Sanchez]

I've got this error message : Traceback (most recent call last): File "/home/install/SqueezeMeta-1.0.0/utils//sqm2tables.py", line 174, in main(parse_args()) File "/home/install/SqueezeMeta-1.0.0/utils//sqm2tables.py", line 63, in main sampleNames, orfs, kegg, cog, pfam, custom = parse_orf_table(perlVars['$mergedfile'], nokegg, nocog, nopfam, args.trusted_functions, args.ignore_unclassifie d, customMethods) File "/home/install/SqueezeMeta-1.0.0/utils/../lib/utils.py", line 122, in parse_orf_table abundances = array([int(line[idx[sample]]) for sample in samples]) File "/home/install/SqueezeMeta-1.0.0/utils/../lib/utils.py", line 122, in abundances = array([int(line[idx[sample]]) for sample in samples]) IndexError: list index out of range

[fpusan]

But before you should have a lot of printed lines, right?

[RafaelLopez-Sanchez]

Yes i've got this:

37 ['megahit_1_1-156', 'megahit_1', 'CDS', 'Prodigal', '157', '53', '39.10', 'pphA', 'k_Bacteria;p_Proteobacteria;c_Gammaproteobacteria;o_Pseudomonadales;f_Mora xellaceae;g_Acinetobacter', 'K07313', 'serine/threonine protein phosphatase 1 [EC:3.1.3.16]', 'Not Included in Pathway or Brite; Unclassified: metabolism; Enzy mes with EC numbers', 'COG0639', 'Diadenosine tetraphosphatase and related serine/threonine protein phosphatases', 'Signal transduction mechanisms', '', 'WP_04 9594749.1', '', '', '', '0', '1.017', '1.001', '0.347', '0', '1.506', '1.154', '0.481', '0', '4', '4', '1', '0', '235', '180', '75', '25'] 37 ['megahit_2_2-250', 'megahit_2', 'CDS', 'Prodigal', '250', '84', '44.18', 'yqeH', 'k_Bacteria;n_Terrabacteria group;p_Firmicutes;c_Bacilli;o_Lactobacillales; f_Leuconostocaceae', 'K06948', '30S ribosome assembly GTPase', 'Brite Hierarchies; Protein families: genetic information processing; Ribosome biogenesis', 'COG 1161', 'Predicted GTPases', 'General function prediction only', '', 'WP_002288652.1', '', '', '', '0', '1.115', '0.784', '0.217', '0', '2.060', '1.506', '0.301 ', '0', '7', '5', '1', '0', '513', '375', '75', '25'] 37 ['megahit_3_3-251', 'megahit_3', 'CDS', 'Prodigal', '250', '84', '48.59', '', 'k_Eukaryota;n_Opisthokonta;n_Fungi;n_Dikarya;p_Ascomycota;n_saccharomyceta;n_P ezizomycotina;n_leotiomyceta;n_sordariomyceta;c_Sordariomycetes;n_Sordariomycetidae;o_Sordariales;f_Sordariaceae;g_Neurospora;s_Neurospora tetrasperma', '', '', '', '', '', '', '', '', '', '', '', '0', '0', '0', '2.605', '0', '0', '0', '3.514', '0', '0', '0', '12', '0', '0', '0', '875', '3'] 37 ['megahit_4_3-281', 'megahit_4', 'CDS', 'Prodigal', '280', '94', '45.88', '', 'k_Bacteria;n_Terrabacteria group;p_Firmicutes;c_Bacilli;o_Lactobacillales;f_Ae rococcaceae;g_Aerococcus', '', '', '', 'ENOG4111Q3F', 'conserved domain protein', 'Function unknown', '', '', '', '', '', '0', '0.284', '0.280', '0.388', '0', '0.538', '0.538', '0.538', '0', '2', '2', '2', '0', '150', '150', '150', '25'] 37 ['megahit_5_3-242', 'megahit_5', 'CDS', 'Prodigal', '241', '81', '54.58', 'pit', 'k_Bacteria;p_Proteobacteria;c_Gammaproteobacteria;o_Enterobacterales;f_Ente robacteriaceae;g_Enterobacter', 'K16322', 'low-affinity inorganic phosphate transporter', 'Brite Hierarchies; Protein families: signaling and cellular processe s; Transporters', 'COG0306', 'Phosphate/sulphate permeases', 'Inorganic ion transport and metabolism', '', 'WP_017693819.1', '', '', '', '0', '1.487', '0.488', '0.225', '0', '2.737', '0.938', '0.312', '0', '9', '3', '1', '0', '657', '225', '75', '25'] 37 ['megahit_6_3-80', 'megahit_6', 'CDS', 'Prodigal', '79', '27', '44.87', '', 'k_Bacteria;p_Proteobacteria;c_Gammaproteobacteria;o_Pseudomonadales;f_Moraxellac eae;g_Acinetobacter', '', '', '', '', '', '', '', 'WP_076753138.1', '', '', '', '0', '7.628', '3.003', '0.693', '0', '6.692', '2.077', '0.295', '0', '15', '6', '1', '0', '522', '162', '23', '25'] 37 ['megahit_7_1-300', 'megahit_7', 'CDS', 'Prodigal', '301', '101', '59.00', '', 'k_Bacteria;p_Proteobacteria;c_Gammaproteobacteria;o_Enterobacterales;f_Erwini aceae;g_Pantoea;s_Pantoea ananatis', '', '', '', 'COG0277', 'FAD/FMN-containing dehydrogenases', 'Energy production and conversion', 'PF01565 [FAD binding doma in ]', 'WP_009776090.1', '', '', '', '0', '1.587', '0', '0', '0', '2.993', '0', '0', '0', '12', '0', '0', '0', '898', '0', '0', '25'] 37 ['megahit_8_77-235', 'megahit_8', 'CDS', 'Prodigal', '160', '54', '39.62', 'wecB', 'k_Bacteria;n_Terrabacteria group;p_Firmicutes;c_Bacilli;o_Lactobacillales ;f_Streptococcaceae;g_Streptococcus', 'K01791', 'UDP-N-acetylglucosamine 2-epimerase (non-hydrolysing) [EC:5.1.3.14]', 'Metabolism; Carbohydrate metabolism; Am ino sugar and nucleotide sugar metabolism | Cellular Processes; Cellular community - prokaryotes; Biofilm formation - Vibrio cholerae', 'COG0381', 'UDP-N-acety lglucosamine 2-epimerase', 'Cell envelope biogenesis, outer membrane', '', 'WP_012211575.1', '', 'GT0', ' Glycosyltransferases not yet assigned to a family', '0', '7.235', '4.911', '7.480', '0', '9.604', '7.465', '7.547', '0', '29', '20', '22', '0', '1527', '1187', '1200', '25'] 37 ['megahit_9_3-296', 'megahit_9', 'CDS', 'Prodigal', '295', '99', '29.93', '', 'k_Bacteria;n_FCB group;n_Bacteroidetes/Chlorobi group;p_Bacteroidetes;c_Flavob acteriia;o_Flavobacteriales;f_Flavobacteriaceae;g_Chryseobacterium;s_Chryseobacterium zeae', '', '', '', '', '', '', '', '', '', '', '', '1.920', '0.540', '0.13 3', '0.552', '0.255', '1.014', '0.255', '0.755', '1', '4', '1', '3', '75', '298', '75', '222', '1'] 37 ['megahit_10_1-297', 'megahit_10', 'CDS', 'Prodigal', '298', '100', '55.56', '', 'k_Eukaryota;n_Opisthokonta;n_Fungi;n_Dikarya;p_Ascomycota;n_saccharomyceta; n_Pezizomycotina;n_leotiomyceta;n_sordariomyceta;c_Sordariomycetes;n_Sordariomycetidae;o_Sordariales;f_Sordariaceae;g_Neurospora', '', '', '', 'ENOG410ZHQS', ' ', 'Function unknown', '', '', '', '', '', '0', '0', '0', '3.094', '0', '0', '0', '4.276', '0', '0', '0', '17', '0', '0', '0', '1270', '25'] 37 ['megahit_11_1-267', 'megahit_11', 'CDS', 'Prodigal', '268', '90', '52.06', 'ABCC-BAC', 'k_Bacteria;n_Terrabacteria group;p_Actinobacteria;c_Actinobacteria;o _Corynebacteriales;f_Mycobacteriaceae;g_Mycobacteroides;s_Mycobacteroides abscessus', 'K06148', 'ATP-binding cassette, subfamily C, bacterial', 'Brite Hierarchi es; Protein families: signaling and cellular processes; Transporters', 'COG1132', 'ABC-type multidrug transport system, ATPase and permease components', '', '' , 'WP_089942140.1', '', '', '', '2.115', '0.891', '1.024', '0.607', '0.281', '1.685', '1.966', '0.843', '1', '6', '7', '3', '75', '450', '525', '225', '25'] 37 ['megahit_12_2-85', 'megahit_12', 'CDS', 'Prodigal', '85', '29', '55.95', '', '', '', '', '', '', '', '', '', '', '', '', '', '0', '0', '0', '4.505', '0', '0 ', '0', '2.893', '0', '0', '0', '7', '0', '0', '0', '243', '0'] 37 ['megahit_13_1-78', 'megahit_13', 'CDS', 'Prodigal', '79', '27', '35.90', '', '', '', '', '', '', '', '', '', '', '', '', '', '0', '0.509', '1.001', '1.386', '0', '0.846', '1.154', '1.513', '0', '1', '2', '2', '0', '66', '90', '118', '0'] 37 ['megahit_15_68-382', 'megahit_15', 'CDS', 'Prodigal', '316', '106', '59.37', 'K06911', 'k_Bacteria;p_Proteobacteria;c_Gammaproteobacteria;o_Enterobacterales ;f_Enterobacteriaceae;g_Enterobacter', 'K06911', 'uncharacterized protein', 'Not Included in Pathway or Brite; Poorly characterized; Function unknown', 'COG174 1', 'Pirin-related protein', 'General function prediction only', 'PF05726 [Pirin C-terminal cupin domain]', 'WP_029881941.1*', '', '', '', '0', '2.896', '0.620 ', '0.343', '0', '4.616', '1.190', '0.263', '0', '23', '5', '2', '0', '1454', '375', '83', '25']

[fpusan]

Are those the last lines you got before the error? I would need to see those

[RafaelLopez-Sanchez]

The last lines before the error are this:

37 ['megahit_13786_1-66', 'megahit_13786', 'CDS', 'Prodigal', '67', '23', '51.52', '', '', '', '', '', '', '', '', '', '', '', '', '', '0', '2.404', '2.366', '0', '0', '1.848', '2.303', '0', '0', '4', '4', '0', '0', '122', '152', '0', '0'] 37 ['megahit_13786_244-423', 'megahit_13786', 'CDS', 'Prodigal', '181', '61', '53.89', '', 'k_Bacteria;p_Proteobacteria;c_Gammaproteobacteria;o_Enterobacterales', '', '', '', 'COG5475', 'Uncharacterized small protein', 'Function unknown', '', 'WP_015583987.1', '', '', '', '0', '2.424', '3.036', '0', '0', '3.378', '4.989', '0', '0', '11', '14', '0', '0', '608', '898', '0', '25'] 20 ['megahit_13786_459-1490', 'megahit_13786', 'CDS', 'Prodigal', '1033', '345', '58.14', 'gpgS', 'k_Bacteria;p_Proteobacteria;c_Gammaproteobacteria;o_Enterobacterales;f_Erwiniaceae;g_Pantoea', 'K13693', 'glucosyl-3-phosphoglycerate synthase [EC:2.4.1.266]', 'Brite Hierarchies; Protein families: metabolism; Glycosyltransferases', 'COG0463', 'Glycosyltransferases involved in cell wall biogenesis', 'Cell envelope biogenesis, outer membrane', '', 'WP_034935549.1', '', 'GT81*', ' NDP-Glc: glucosyl-3-phosphoglycerate synthase (EC 2.4.1.-); NDP-Man: mannosyl-3-phosphoglycerate synthase (EC 2.4.1.-);'] Traceback (most recent call last): File "/home/install/SqueezeMeta-1.0.0/utils/sqm2tables.py", line 174, in main(parse_args()) File "/home/install/SqueezeMeta-1.0.0/utils/sqm2tables.py", line 63, in main sampleNames, orfs, kegg, cog, pfam, custom = parse_orf_table(perlVars['$mergedfile'], nokegg, nocog, nopfam, args.trusted_functions, args.ignore_unclassified, customMethods) File "/home/install/SqueezeMeta-1.0.0/utils/../lib/utils.py", line 122, in parse_orf_table abundances = array([int(line[idx[sample]]) for sample in samples]) File "/home/install/SqueezeMeta-1.0.0/utils/../lib/utils.py", line 122, in abundances = array([int(line[idx[sample]]) for sample in samples]) IndexError: list index out of range

[fpusan]

Ok. The last line has 20 fields instead of 37, that's what breaking the parser. That happened a while ago for some corner cases, but the bug should have been corrected. It's hard to tell whether this is because you started the project using an older version, or because of an existing bug.

Would you mind cloning the latest version of SqueezeMeta from master and using that to repeat the project? In order to link the updated version with the databases you already have, just download the new version and run:

/path/to/new/SqueezeMeta/scripts/preparing_databases/configure_nodb.pl /path/to/db

Where db is the database folder containing the nr.dmnd file.

[RafaelLopez-Sanchez]

Not at all, I'll do that right away and as soon as the project is finished I will let you know

[RafaelLopez-Sanchez]

Hi Fernando the same error occurred in the new project with the updated version :

Error :

File "/home/install/SqueezeMeta/utils/sqm2tables.py", line 174, in main(parse_args()) File "/home/install/SqueezeMeta/utils/sqm2tables.py", line 63, in main sampleNames, orfs, kegg, cog, pfam, custom = parse_orf_table(perlVars['$mergedfile'], nokegg, nocog, nopfam, args.trusted_functions, args.ignore_unclassified, customMethods) File "/home/install/SqueezeMeta/utils/../lib/utils.py", line 122, in parse_orf_table abundances = array([int(line[idx[sample]]) for sample in samples]) File "/home/install/SqueezeMeta/utils/../lib/utils.py", line 122, in abundances = array([int(line[idx[sample]]) for sample in samples]) IndexError: list index out of range

After that I placed the command you told me before in the line 121 of the utils.py script and same error again:

37 ['megahit_20876_409-1266', 'megahit_20876', 'CDS', 'Prodigal', '859', '287', '69.81', 'folP', 'k_Bacteria;n_Terrabacteria group;p_Actinobacteria;c_Actinobacteria;o_Corynebacteriales;f_Corynebacteriaceae;g_Corynebacterium;s_Corynebacterium vitaeruminis', 'K00796', 'dihydropteroate synthase [EC:2.5.1.15]', 'Metabolism; Metabolism of cofactors and vitamins; Folate biosynthesis', 'COG0294', 'Dihydropteroate synthase and related enzymes', 'Coenzyme metabolism', 'PF00809 [Pterin binding enzyme]', 'WP_092532059.1', '', '', '', '0', '0.925', '1.547', '0.819', '0', '1.659', '2.770', '1.090', '0', '20', '34', '13', '0', '1423', '2377', '935', '25'] 20 ['megahit_20876_1263-1961', 'megahit_20876', 'CDS', 'Prodigal', '700', '234', '72.10', 'gpgS', 'k_Bacteria;n_Terrabacteria group;p_Actinobacteria;c_Actinobacteria;o_Corynebacteriales;f_Corynebacteriaceae;g_Corynebacterium;s_Corynebacterium vitaeruminis', 'K13693', 'glucosyl-3-phosphoglycerate synthase [EC:2.4.1.266]', 'Brite Hierarchies; Protein families: metabolism; Glycosyltransferases', 'COG0463', 'Glycosyltransferases involved in cell wall biogenesis', 'Cell envelope biogenesis, outer membrane', 'PF00535 [Glycosyl transferase family 2];PF13641 [Glycosyltransferase like family 2]', 'WP_075545579.1', '', 'GT81*', ' NDP-Glc: glucosyl-3-phosphoglycerate synthase (EC 2.4.1.-); NDP-Man: mannosyl-3-phosphoglycerate synthase (EC 2.4.1.-);'] Traceback (most recent call last): File "/home/install/SqueezeMeta/utils/sqm2tables.py", line 174, in main(parse_args()) File "/home/install/SqueezeMeta/utils/sqm2tables.py", line 63, in main sampleNames, orfs, kegg, cog, pfam, custom = parse_orf_table(perlVars['$mergedfile'], nokegg, nocog, nopfam, args.trusted_functions, args.ignore_unclassified, customMethods) File "/home/install/SqueezeMeta/utils/../lib/utils.py", line 122, in parse_orf_table abundances = array([int(line[idx[sample]]) for sample in samples]) File "/home/install/SqueezeMeta/utils/../lib/utils.py", line 122, in abundances = array([int(line[idx[sample]]) for sample in samples]) IndexError: list index out of range

[fpusan]

The contigs are different (they should be, since its a different run) but the annotation is the same. Considering that you are using custom databases, there might be something weird in some of the entries that is breaking our parser.

Could you send us the full /path/to/Project/results/13.*.orftable file?

[RafaelLopez-Sanchez]

Sure here it is:

13.pozol.orftable.zip

[fpusan]

I see it now. Did you use Windows to generate the files for the external databases?

[RafaelLopez-Sanchez]

I used the databases from dbCAN2 and ANTISMASH formatted for diamond use in the Linux server version Ubuntu 18.04.2 LTS with your script : /path/to/SqueezeMeta/bin/diamond makedb -d /path/to/ext/db/dbname.dmnd --in /path/to/my/ext/dbname.fasta

[fpusan]

Which command did you use to call SqueezeMeta? Did you provide an additional file with functional annotations for the external databases?

[RafaelLopez-Sanchez]

I've used this command: SqueezeMeta.pl -m coassembly -p pozol -s sample.file -f /home/samples/pozol/shotgun/no_mais_sequences/ --extdb mydb.list -miniden 60 -t 40

And the file mydb.list says this:

CAZy /tres/DB/CAZyDB/fasta/CAZyDB.fasta.dmnd /tres/DB/CAZyDB/fasta/CAZyDB.07312019.fam-activities.txt Antismash /tres/DB/antismashDB/antismashSM1/proteins.fasta.dmnd /tres/DB/antismashDB/antismashSM1/clusters.txt

[fpusan]

Could you send me the CAZyDB.07312019.fam-activities.txt and antismashDB/antismashSM1/clusters.txt files?

Were any of them perhaps generated in Windows?

[RafaelLopez-Sanchez]

Don't know, but it's a possibility because they are downloadable files in the DBCAN2 and antismash DB CAZyDB.07312019.fam-activities.txt

clusters.txt.zip

[fpusan]

Indeed. There are two lines in the CAZyDB file with Windows-like line terminators, which do not play nice with our scripts. We'll fix it for v1.1.

For now, just run

dos2unix CAZyDB.07312019.fam-activities.txt

And to be safe also

dos2unix clusters.txt

After that you can use those files with SqueezeMeta. I think it should be fine if you restart the project from step 4.

Let me know if that works for you and sorry for the inconvenience!

[fpusan]

Actually, just try repeating step 13 after you run dos2unix in the mapping files, that might be enough!

[RafaelLopez-Sanchez]

Will do Fernando, and thank you very much for your time and patience with this issue. As soon as the step 13 is finished I'll let you know if it worked

[RafaelLopez-Sanchez]

It worked! Thanks a lot Fernando! Have a nice day!

[fpusan]

Glad to hear. Closing issue