search

HKU-BAL / Clair3

Clair3 - Symphonizing pileup and full-alignment for high-performance long-read variant calling
244 stars 26 forks source link

errors for illumina full alignment calling, questions for merging output. #158

Closed shiying-sxu closed 1 year ago

shiying-sxu commented 1 year ago

hello！ When I ran sh trainfullalignment_illumina_r1.sh, at the end Merge compressed binaries reported an error： Traceback (most recent call last): File "/work/Clair3-main1212/clair3.py", line 105, in main() File "/work/Clair3-main1212/clair3.py", line 99, in main submodule.main() File "/work/Clair3-main1212/preprocess/MergeBin.py", line 91, in main Run(args) File "/work/Clair3-main1212/preprocess/MergeBin.py", line 56, in Run table_dict = utils.write_table_file(table_file, table_dict, tensor_shape, param.label_size, float_type) File "/work/Clair3-main1212/clair3/utils.py", line 245, in write_table_file position_matrix = np.array(table_dict['position_matrix'], np.dtype(float_type)).reshape([-1] + tensor_shape) ValueError: cannot reshape array of size 7260000 into shape (89,33,8) Closing remaining open files:/work/Clair3-main-sy/data/datatest/Illumina/fullalign/outputref-HG001_GRCh38-1212//build/tensor_can/tensor_hg001_1000_22_1...done/work/Clair3-main-sy/data/datatest/Illumina/fullalign/outputref-HG001_GRCh38-1212//build/bins/bin_hg001_1000_22...done [INFO] Merging file /work/Clair3-main-sy/data/datatest/Illumina/fullalign/outputref-HG001_GRCh38-1212//build/tensor_can/tensor_hg002_1000_22_1 Traceback (most recent call last): File "/work/Clair3-main1212/clair3.py", line 105, in main() File "/work/Clair3-main1212/clair3.py", line 99, in main submodule.main() File "/work/Clair3-main1212/preprocess/MergeBin.py", line 91, in main Run(args) File "/work/Clair3-main1212/preprocess/MergeBin.py", line 56, in Run table_dict = utils.write_table_file(table_file, table_dict, tensor_shape, param.label_size, float_type) File "/work/Clair3-main1212/clair3/utils.py", line 245, in write_table_file position_matrix = np.array(table_dict['position_matrix'], np.dtype(float_type)).reshape([-1] + tensor_shape) ValueError: cannot reshape array of size 7260000 into shape (89,33,8) Closing remaining open files:/work/Clair3-main-sy/data/datatest/Illumina/fullalign/outputref-HG001_GRCh38-1212//build/tensor_can/tensor_hg002_1000_22_1...done/work/Clair3-main-sy/data/datatest/Illumina/fullalign/outputref-HG001_GRCh38-1212//build/bins/bin_hg002_1000_22...done

I tried changing to ont model and found it correct, but while testing sh clair3_ilmn_quick_demo.sh reported an error:. Total processed positions in chr20 (chunk 1/1) : 0 [ERROR] No full-alignment output for file chr20//work/Clair3-main/data/datatest/Illumina/clair3_illumina_quickDemo/output/tmp/full_alignment_output/full_alignment_chr20.0_1.vcf Total time elapsed: 0.00 s

zhengzhenxian commented 1 year ago

Hi, I'm afraid that the ONT model cannot be applied to Illumina data, as they have different input shapes.

Could you try to add --platform ilmn \ option for the mergebin submodule to see whether the issue persists, thanks.

shiying-sxu commented 1 year ago

 I tried adding the --platform ilmn \ option to the mergebin submodule and found that the program works fine, thanks a lot!   ------------------ Original ------------------ From: @.>; Date:  Wed, Dec 14, 2022 02:05 PM To: @.>; Cc: @.>; @.>; Subject:  Re: [HKU-BAL/Clair3] errors for illumina full alignment calling, questions for merging output. (Issue #158)

 

Hi, I'm afraid that the ONT model cannot be applied to Illumina data, as they have different input shapes.

Could you try to add --platform ilmn \ option for the mergebin submodule to see whether the issue persists, thanks.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

shiying-sxu commented 1 year ago

But another problem came up when I was testing（clair3_ilmn_quick_demo）： [INFO] 6/7 Call low-quality variants using full-alignment model [INFO] Delay 4 seconds before starting variant calling ... Traceback (most recent call last):   File "/work/Clair3-main-sy/clair3/../clair3.py", line 94, in <module>     main()   File "/work/Clair3-main-sy/clair3/../clair3.py", line 81, in main     submodule = import_module("%s.%s" % (directory, submodule_name))   File "/home/user/anaconda3/envs/attdeepcaller/lib-python/3/importlib/init.py", line 126, in import_module     return _bootstrap._gcd_import(name[level:], package, level)   File "<frozen importlib._bootstrap>", line 1003, in _gcd_import   File "<frozen importlib._bootstrap>", line 980, in _find_and_load   File "<frozen importlib._bootstrap>", line 964, in _find_and_load_unlocked   File "<frozen importlib._bootstrap>", line 674, in _load_unlocked   File "<builtin>/frozen importlib._bootstrap_external", line 691, in exec_module   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed   File "/work/Clair3-main-sy/preprocess/RealignReads.py", line 33, in <module>     realigner = ctypes.cdll.LoadLibrary(realigner_mod)   File "/home/user/anaconda3/envs/attdeepcaller/lib-python/3/ctypes/init.py", line 432, in LoadLibrary     return self._dlltype(name)   File "/home/user/anaconda3/envs/attdeepcaller/lib-python/3/ctypes/init.py", line 350, in init     pypy_dll = _ffi.CDLL(name, mode) OSError: Cannot load library /work/Clair3-main-sy/preprocess/realign/realigner: /work/Clair3-main-sy/preprocess/realign/realigner: cannot open shared object file: No such file or directory [mpileup] fail to read the header of - Calling variants ... Total processed positions in chr20 (chunk 1/1) : 0 [ERROR] No full-alignment output for file chr20//work/Clair3-main/data/datatest/Illumina/clair3_illumina_quickDemo/output/tmp/full_alignment_output/full_alignment_chr20.0_1.vcf Total time elapsed: 0.01 s [INFO] No vcf output for file /work/Clair3-main/data/datatest/Illumina/clair3_illumina_quickDemo/output/tmp/full_alignment_output/full_alignment_chr20.0_1.vcf, remove empty file

There is a warning at the beginning of the run：  [WARNING] Illumina platform will disable C implement to support short read realignment process!    ------------------ Original ------------------ From: @.>; Date:  Wed, Dec 14, 2022 02:05 PM To: @.>; Cc: @.>; @.>; Subject:  Re: [HKU-BAL/Clair3] errors for illumina full alignment calling, questions for merging output. (Issue #158)

 

Hi, I'm afraid that the ONT model cannot be applied to Illumina data, as they have different input shapes.

Could you try to add --platform ilmn \ option for the mergebin submodule to see whether the issue persists, thanks.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

zhengzhenxian commented 1 year ago

Hi,

Seem that the realigner package is not found in your environment, pls follow this instruction to install it, thanks!

shiying-sxu commented 1 year ago

I tried it and found that it works, thank you very much for your prompt reply!     ------------------ Original ------------------ From: @.>; Date:  Thu, Dec 15, 2022 09:56 AM To: @.>; Cc: @.>; @.>; Subject:  Re: [HKU-BAL/Clair3] errors for illumina full alignment calling, questions for merging output. (Issue #158)

 

Hi,

Seem that the realigner package is not found in your environment, pls follow this instruction to install it, thanks!

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>