sqlite3.OperationalError: disk I/O error

[yangkl0909]

Hi Paul, I am running FindFungi for the demo ERR675624 fastq on cluster(CentOS Linux release 7.4.1708). I had remove bsub and got the following results. There something error I can't fix, could help have a check? Thank you.

• ouput files in ./ERR675624/FindFungi/Results

  128K Oct 21 09:34 BLAST_Processing
  24bytes Oct 21 11:32 ERR675624.WordCloud.R
  0bytes Oct 21 11:32 Final_Results_ERR675624-lca.sorted.csv
  21bytes Oct 21 11:29 Final_Results_ERR675624-taxids.txt
  91K Oct 21 05:48 Final_Results_ERR675624.tsv
  2.9K Oct 21 11:30 Final_Results_ERR675624.tsv_AllResults-taxids.txt
  11M Oct 21 05:48 Final_Results_ERR675624.tsv_AllResults.tsv
  0bytes Oct 21 11:32 Final_Results_ERR675624_AllResults-lca.sorted.csv

The CSV is always empty with the errors showing below

• source script code

  $ScriptPath/LowestCommonAncestor_V4.sh $Dir/Results/Final_Results_$z.tsv
  $ScriptPath/LowestCommonAncestor_V4.sh $Dir/Results/Final_Results_$z.tsv_AllResults.tsv

• error information (using ete3-3.1.1 )

  Uploading to /users/username/.etetoolkit/taxa.sqlite
  Traceback (most recent call last):
  File "/users/username/tools/FindFungi-v0.23.3/LowestCommonAncestor_V4.py", line 26, in <module>
  ncbi = NCBITaxa()
  File "/users/username/username/tools/conda3/lib/python2.7/site-packages/ete3/ncbi_taxonomy/ncbiquery.py", line 120, in __init__
  self.update_taxonomy_database(taxdump_file)
  File "/users/username/username/tools/conda3/lib/python2.7/site-packages/ete3/ncbi_taxonomy/ncbiquery.py", line 129, in update_taxonomy_database
  update_db(self.dbfile)
  File "/users/username/username/tools/conda3/lib/python2.7/site-packages/ete3/ncbi_taxonomy/ncbiquery.py", line 760, in update_db
  upload_data(dbfile)
  File "/users/username/username/tools/conda3/lib/python2.7/site-packages/ete3/ncbi_taxonomy/ncbiquery.py", line 791, in upload_data
  db.execute(cmd)
  sqlite3.OperationalError: disk I/O error
  Done

• error information(using older version ete3)

  /users/username/tools/FindFungi-v0.23.3//LowestCommonAncestor_V4.sh: line 14: 28870 Segmentation fault      python2.7 ~/tools/FindFungi-v0.23.3/LowestCommonAncestor_V4.py ${1} ${y}-taxids.txt ${y}-lca.csv
  Done
  /users/username/tools/FindFungi-v0.23.3//LowestCommonAncestor_V4.sh: line 14: 28875 Segmentation fault      python2.7 ~/tools/FindFungi-v0.23.3/LowestCommonAncestor_V4.py ${1} ${y}-taxids.txt ${y}-lca.csv
  Done

Best Keli

[GiantSpaceRobot]

Hi Keli,

I haven't seen this error before with FindFungi. For ete3-3.1.1, it looks like ete3 cannot write to file. Is your directory underneath Dropbox/Google Drive? If this is the case, try pausing Dropbox/Google Drive and run the LowestCommonAncestor_V4.sh programme, or run the pipeline again. This problem can occur because the syncing of information causes read/write inconsistencies.

If this is not the problem, please make sure you are using the newest release of ete3 (https://github.com/etetoolkit/ete/tree/master/ete3) as they have have fixed taxdump database bugs recently.

Lastly, have you been using conda to manage Python versions/installations etc? I have seen this cause dependency clashes before.

Best, Paul

[yangkl0909]

@GiantSpaceRobot
Thank you.

I fixed the qlite3.OperationalError: disk I/O error by moving the output back to my PC. We don't run Google Drive or any other sync tools in our cluster. It seems the error might caused by Lustre file system in our cluster showing like this https://github.com/TApplencourt/EMSL_Basis_Set_Exchange_Local/issues/6.

However, when moving on, another error occur in the LowestCommonAncestor_V4.py.

Traceback (most recent call last):
  File "/u2host/findfungi_test/FindFungi-v0.23.3/LowestCommonAncestor_V4.py", line 56, in <module>
    for i in descendants:
NameError: name 'descendants' is not defined
Done

I got the Final_Results_ERR675624-lca.csv with one line title while no other information.

Taxon name,Taxid,Reads mapping to taxid,Reads mapping to children taxids,Pearson skewness score,Percent of pseudo-chromosomes with read hits

Seems there is an indentation error in this python script?

---

• Attached the first 5 line in Final_Results_ERR675624.tsv

C	ERR675624.1001391_FC61KB9AAXX:4:14:2367:14327#GATCAG/1	76775	100	100	0.042346	100
C	ERR675624.1001391_FC61KB9AAXX:4:14:2367:14327#GATCAG/2	76775	100	100	0.042346	100
C	ERR675624.101282_FC61KB9AAXX:4:2:6436:3574#GATCAG/1	1759314	100	100	0.471951	100
C	ERR675624.101282_FC61KB9AAXX:4:2:6436:3574#GATCAG/2	294747	100	100	-0.42256	100
C	ERR675624.1029318_FC61KB9AAXX:4:14:9179:15769#GATCAG/1	1759314	100	100	0.471951	100

Is the Final_Results_ERR675624.tsv output right?

[GiantSpaceRobot]

Hi @yangkl0909

That output looks correct. And you are correct that there was an indentation error in the LowestCommonAncestor_V4.py script. I have fixed this now (https://github.com/GiantSpaceRobot/FindFungi/blob/master/FindFungi-v0.23.3/LowestCommonAncestor_V4.py).

Let me know if you have any trouble. Thank you.

Paul

[yangkl0909]

@GiantSpaceRobot Hi Paul.

Sorry, the same error occurs in the script LowestCommonAncestor_V4.py.

Thank you.

Best Keli

[GiantSpaceRobot]

Hi Keli,

I have re-formatted the LowestCommonAncestor_V4.py script. It is working for on a Unix server. Let me know how it goes.

Thanks, Paul

[yangkl0909]

Hi Paul,

Finally, I got the csv tables with your new script.

It was done in a new ubuntu 14.04 environment with python 2.7.13(without Conda) run in docker. While still running error in ubuntu 16.04 with python 2.7.15 (install by Conda). Maybe the script can't run in python 2.7.15 or a conda environment?

Thanks a lot.

Best Keli

[GiantSpaceRobot]

Glad to hear it, Paul

[mhyleung]

Dear all

I am running this on python 2.7.13 (without Conda), but even with the new LowestCommonAncestor_V4.py script I am encountering the same error:

  File "/disk/rdisk08/mhyleung/tools/FindFungi/FindFungi-v0.23.3/LowestCommonAncestor_V4.py", line 56, in <module>
    for i in descendants:
NameError: name 'descendants' is not defined
Done

My output also has only the top line and no other information.

Thanks

Marc

[GiantSpaceRobot]

Hi Marc,

This script is working on my system, I am unsure why this error is happening. Are you certain ete3 is up-to-date? If it is, can you try running the script with a small test set? (data attached)

Execute the script:

python LowestCommonAncestor_V4.py KrakenOutput.txt Taxids.txt MyOutput.tsv

Let me know if this works/fails.

Best, Paul

Taxids.txt KrakenOutput.txt

[mhyleung]

Hi Paul

The same descendants error persists. From my understanding on this thread there is no way that the script would work on a conda version of python right? It seems like I am encountering some trouble trying to get ete3 (v3.1.1) using a non-conda version of python 2.7.13, and ete3 is currently only working under conda for me. I will keep on trying to fix my python 2.7.13 issue, but in the meantime, I just want to know if there is a way the script would work even if I run python on conda.

Thanks again

Marc

[GiantSpaceRobot]

Hi Marc,

Unfortunately I have not tested the scripts/pipeline in Conda at all so I can't offer any advice. FYI, I am using Python 2.7.15rc1 and it works fine with ete3 v3.1.1.

Paul