search

shohei-kojima / MEGAnE

MEGAnE
MIT License
24 stars 2 forks source link

UnboundLocalError: local variable 'mdz' referenced before assignment #6

Closed YiweiNiu closed 1 year ago

YiweiNiu commented 1 year ago

Hi,

I am using the Github version of MEGAnE on a human WGS sample, and I encountered an error as follws:

2022-08-23 03:39:07,957:INFO:Initial check started.
2022-08-23 03:39:23,669:INFO:estimated read lenth = 103
2022-08-23 03:39:23,688:INFO:-cov 31.150000 was rounded to 31
2022-08-23 03:39:24,172:INFO:All 24 main chromosome(s) were found in /MEI2/gatkout/CN000440/CN000440.final.bam.
2022-08-23 03:39:24,173:INFO:"chrX" was found in /MEI2/gatkout/CN000440/CN000440.final.bam. "chrX" will be considered as a female sex chromosome.
2022-08-23 03:39:24,173:INFO:"chrY" was found in /MEI2/gatkout/CN000440/CN000440.final.bam. "chrY" will be considered as a male sex chromosome.
2022-08-23 03:39:24,437:INFO:Preprocessing started.
2022-08-23 03:39:40,264:INFO:N=1155 repeats found in /RefData/MEI/megane_repeat_lib/Dfam_custom.rep. N=1155 will be analyzed. N=0 will be excluded due to non-ME repeats.
2022-08-23 03:42:08,352:INFO:Discordant read search started.
2022-08-23 04:11:28,742:INFO:Screening results:nonXY_reads=675617148,X_reads=17382654,Y_reads=4772766,chimeric_reads=4581543,hybrid_reads=3575876,pA_reads=63208,absent_reads=84389
2022-08-23 04:11:28,965:INFO:Clipped read processing started.

2022-08-23 04:34:36,350:INFO:Unmapped read processing started.
2022-08-23 04:37:31,880:INFO:Hybrid read processing started.
2022-08-23 04:42:00,515:INFO:Integration junction search (outside of TEs) started.
2022-08-23 04:43:12,690:INFO:Integration junction search (nested in TEs) started.
2022-08-23 04:51:11,275:INFO:Filtering started.
2022-08-23 04:52:15,272:INFO:[1450, 1450] ME insertion candidates found.
2022-08-23 04:52:15,272:INFO:ME insertion search finished!
2022-08-23 04:52:15,273:INFO:Absent ME search started.
2022-08-23 04:52:24,224:INFO:1263 absent ME candidates found.
2022-08-23 04:52:24,224:INFO:Absent ME search finished!
2022-08-23 04:52:24,247:INFO:pME search finished!
2022-08-23 04:52:24,253:INFO:Limit BAM/CRAM started.
[E::idx_find_and_load] Could not retrieve index file for '/MEI2/MEGAnE/test/CN000440/only_necessary.bam'
2022-08-23 05:02:08,062:INFO:Evidence search started, insertion: /MEI2/MEGAnE/test/CN000440/MEI_final_gaussian.bed
[E::idx_find_and_load] Could not retrieve index file for '/MEI2/MEGAnE/test/CN000440/tmp.bam'
2022-08-23 05:02:18,955:ERROR:
Traceback (most recent call last):
  File "/software/MEGAnE-v1.0.2/scripts/allele_count_ins.py", line 719, in evaluate_spanning_read
    left_match=count_mismatch(ls[5], mdz, 0, s - start)
UnboundLocalError: local variable 'mdz' referenced before assignment

Do you have any clues about this? Thanks in advance.

Best, Yiwei

shohei-kojima commented 1 year ago

I am so sorry for the error. This is a kind of error that I have not encontered. Could you let me know two things?

  1. Does your log file contains "MD tag was not found." message?
  2. What version of samtools are you using?

MEGAnE adds MD tag to the small bam file named "only_necessary.bam" if MD tag is absent in the original input bam file. It seems something is wrong at this step.

YiweiNiu commented 1 year ago

Thank you for your quick reply.

  1. Does your log file contains "MD tag was not found." message?

I have pasted all the log messages above. So, "MD tag was not found" could not be found in the log file.

  1. What version of samtools are you using?

Here is the version of samtools and htslib I used: samtools 1.14 (Using htslib 1.14).

shohei-kojima commented 1 year ago

Thank you for the answers. About the Q1, I meant the log file, named "for_debug.log", which should be found in the result directory. That file contains some debug messages as well. Could you check this file?

YiweiNiu commented 1 year ago

I checked the file for_debug.log, but failed to find "MD tag was not found".

Here is this file: for_debug.log. Please contact me if any other information is needed.

Thanks a lot for your help.

shohei-kojima commented 1 year ago

Thank you very much for sharing the log file.

I still cannot figure out what caused this error, but I guess some few reads in the input bam file may not have MD tag. Could you try a code in a branch named _MEGAnE_debug_220904? MEGAnE checked only one first read to judge whether the "only_necessary.bam" contains MD tag, but this version now checks all reads.

Thank you.

YiweiNiu commented 1 year ago

I will try this version of MEGAnE, and give you the feedback about the logs and results later. Thank you.

YiweiNiu commented 1 year ago

Hi, I tried _MEGAnE_debug_220904 and it works without errors. Here is the message infor:

2022-09-04 21:51:48,480:INFO:Initial check started.
2022-09-04 21:52:04,119:INFO:estimated read lenth = 103
2022-09-04 21:52:04,145:INFO:-cov 31.150000 was rounded to 31
2022-09-04 21:52:04,156:INFO:All 24 main chromosome(s) were found in /MEI2/gatkout/CN000440/CN000440.final.bam.
2022-09-04 21:52:04,156:INFO:"chrX" was found in /MEI2/gatkout/CN000440/CN000440.final.bam. "chrX" will be considered as a female sex chromosome.
2022-09-04 21:52:04,156:INFO:"chrY" was found in /MEI2/gatkout/CN000440/CN000440.final.bam. "chrY" will be considered as a male sex chromosome.
2022-09-04 21:52:04,353:INFO:Preprocessing started.
2022-09-04 21:52:30,875:INFO:N=1155 repeats found in /RefData/MEI/megane_repeat_lib/Dfam_custom.rep. N=1155 will be analyzed. N=0 will be excluded due to non-ME repeats.
2022-09-04 21:56:01,088:INFO:Discordant read search started.
2022-09-04 22:30:13,589:INFO:Screening results:nonXY_reads=675617148,X_reads=17382654,Y_reads=4772766,chimeric_reads=4581543,hybrid_reads=3575876,pA_reads=63208,absent_reads=84389
2022-09-04 22:30:13,755:INFO:Clipped read processing started.

2022-09-04 23:00:45,103:INFO:Unmapped read processing started.
2022-09-04 23:02:48,753:INFO:Hybrid read processing started.
2022-09-04 23:09:12,846:INFO:Integration junction search (outside of TEs) started.
2022-09-04 23:11:49,964:INFO:Integration junction search (nested in TEs) started.
2022-09-04 23:25:32,699:INFO:Filtering started.
2022-09-04 23:27:29,304:INFO:[1458, 1458] ME insertion candidates found.
2022-09-04 23:27:29,306:INFO:ME insertion search finished!
2022-09-04 23:27:29,677:INFO:Absent ME search started.
2022-09-04 23:27:42,005:INFO:1263 absent ME candidates found.
2022-09-04 23:27:42,006:INFO:Absent ME search finished!
2022-09-04 23:27:42,305:INFO:pME search finished!
2022-09-04 23:27:42,471:INFO:Limit BAM/CRAM started.
[E::idx_find_and_load] Could not retrieve index file for '/MEI2/MEGAnE/test/CN000440/only_necessary.bam'
2022-09-04 23:42:54,699:INFO:Evidence search started, insertion: /MEI2/MEGAnE/test/CN000440/MEI_final_gaussian.bed
[E::idx_find_and_load] Could not retrieve index file for '/MEI2/MEGAnE/test/CN000440/tmp.bam'
2022-09-04 23:44:11,249:INFO:Evidence merge started, insertion: /MEI2/MEGAnE/test/CN000440/MEI_final_gaussian.bed
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_core.py:1326: UserWarning: Vertical orientation ignored with only `x` specified.
  warnings.warn(single_var_warning.format("Vertical", "x"))
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_core.py:1326: UserWarning: Vertical orientation ignored with only `x` specified.
  warnings.warn(single_var_warning.format("Vertical", "x"))
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_core.py:1326: UserWarning: Vertical orientation ignored with only `x` specified.
  warnings.warn(single_var_warning.format("Vertical", "x"))
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_core.py:1326: UserWarning: Vertical orientation ignored with only `x` specified.
  warnings.warn(single_var_warning.format("Vertical", "x"))
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_core.py:1326: UserWarning: Vertical orientation ignored with only `x` specified.
  warnings.warn(single_var_warning.format("Vertical", "x"))
2022-09-04 23:44:15,217:INFO:1427 high-confidence pMEI found.
2022-09-04 23:44:15,300:INFO:Did output VCF, insertion: /MEI2/MEGAnE/test/CN000440/MEI_final_gaussian.bed
2022-09-04 23:44:15,325:INFO:Evidence search started, insertion: /MEI2/MEGAnE/test/CN000440/MEI_final_percentile.bed
[E::idx_find_and_load] Could not retrieve index file for '/MEI2/MEGAnE/test/CN000440/tmp.bam'
2022-09-04 23:44:55,225:INFO:Evidence merge started, insertion: /MEI2/MEGAnE/test/CN000440/MEI_final_percentile.bed
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_core.py:1326: UserWarning: Vertical orientation ignored with only `x` specified.
  warnings.warn(single_var_warning.format("Vertical", "x"))
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_core.py:1326: UserWarning: Vertical orientation ignored with only `x` specified.
  warnings.warn(single_var_warning.format("Vertical", "x"))
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_core.py:1326: UserWarning: Vertical orientation ignored with only `x` specified.
  warnings.warn(single_var_warning.format("Vertical", "x"))
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_core.py:1326: UserWarning: Vertical orientation ignored with only `x` specified.
  warnings.warn(single_var_warning.format("Vertical", "x"))
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_core.py:1326: UserWarning: Vertical orientation ignored with only `x` specified.
  warnings.warn(single_var_warning.format("Vertical", "x"))
2022-09-04 23:44:58,652:INFO:1427 high-confidence pMEI found.
2022-09-04 23:44:58,735:INFO:Did output VCF, insertion: /MEI2/MEGAnE/test/CN000440/MEI_final_percentile.bed
2022-09-04 23:44:58,803:INFO:Evidence search started, absent MEs.
[E::idx_find_and_load] Could not retrieve index file for '/MEI2/MEGAnE/test/CN000440/tmp.bam'
2022-09-04 23:45:19,826:INFO:Evidence merge started, absent MEs.
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_core.py:1326: UserWarning: Vertical orientation ignored with only `x` specified.
  warnings.warn(single_var_warning.format("Vertical", "x"))
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_core.py:1326: UserWarning: Vertical orientation ignored with only `x` specified.
  warnings.warn(single_var_warning.format("Vertical", "x"))
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(
/software/anaconda3/lib/python3.9/site-packages/seaborn/_core.py:1326: UserWarning: Vertical orientation ignored with only `x` specified.
  warnings.warn(single_var_warning.format("Vertical", "x"))
2022-09-04 23:45:21,910:INFO:1107 high-confidence absent pME found.
2022-09-04 23:45:21,998:INFO:Did output VCF, absent MEs.
2022-09-04 23:45:22,034:INFO:Genotyping finished!
2022-09-04 23:45:22,036:INFO:

All analysis finished! Thank you for using MEGAnE!

  When you next generate joint VCF, please specify the output directory (/MEI2/MEGAnE/test/CN000440) in Step 2.
  When you analyze only this sample, please use: MEI_final_percentile_genotyped.vcf & absent_MEs_genotyped.vcf


And this is the for_debug.log: for_debug.log.

I hope these information will be useful to you. Thank you for your time and effort.

shohei-kojima commented 1 year ago

Thank you for the report! It is good to see it worked! Yes, there are lots to learn and your inputs helps me to keep improving this tool. Thank you very much for letting me know the errors!