metageni / SUPER-FOCUS

A tool for agile functional analysis of shotgun metagenomic data
GNU General Public License v3.0
21 stars 12 forks source link

_csv.Error: field larger than field limit (131072) #59

Closed s4251484 closed 4 years ago

s4251484 commented 4 years ago

Hi @metageni and the team

i encountered this issue where superfocus successfully generated the .m8 file but failed to generate the excel files. FYI, I am analysing the contigs with superfocus diamond 98.

Verbose output: ....... 478 queries aligned. diamond v0.9.32.133 (C) Max Planck Society for the Advancement of Science Documentation, support and updates available at http://www.diamondsearch.org

CPU threads: 24

Loading subject IDs... [2.603s] Scoring parameters: (Matrix=custom Lambda=0.267 K=0.041 Penalties=11/1) DB sequences = 5234971 DB sequences used = 9623 DB letters = 1880813436

Target sequences to report alignments for: 25

Generating output... [12.417s] [2020-05-20 11:25:15,035 - INFO] Parsing Alignments Traceback (most recent call last): File "/RDS/Q1034/conda/superfocus/bin/superfocus", line 12, in sys.exit(main()) File "/RDS/Q1034/conda/superfocus/lib/python3.8/site-packages/superfocus_app/superfocus.py", line 339, in main results, binning_reads = parse_alignments(alignment_name, results, normalise_output, len(query_files), File "/RDS/Q1034/conda/superfocus/lib/python3.8/site-packages/superfocus_app/do_alignment.py", line 150, in parse_alignments for row in alignment_reader: _csv.Error: field larger than field limit (131072)

understand that its a limit with the csv - how could I possibly fix it - seems like it is going to generate some huge excel files.

metageni commented 4 years ago

@s4251484 Thanks for reporting this.

One thing to be aware - unrelated to what you reported. Make sure you predict the ORFs for the contigs before running SUPER-FOCUS. Otherwise, the tool will assign the best hit which will not represent all the functions on the contig.

I will need to take a deeper look into your problem and I will get back to you.

metageni commented 4 years ago

@s4251484 The solution seems to be simple. I will push it into master and will you be able to re-run it?

Best

https://stackoverflow.com/questions/15063936/csv-error-field-larger-than-field-limit-131072

s4251484 commented 4 years ago

hi! yea i modified the do.alignment.py accordingly. yea it is a fix!

meanwhile I shall work on my ORF! legit advice! thanks!

@s4251484 The solution seems to be simple. I will push it into master and will you be able to re-run it?

Best

https://stackoverflow.com/questions/15063936/csv-error-field-larger-than-field-limit-131072