Can't run read-simulator. Probably dependency problem

[shafayetrahat]

Description When running neat run-simulator -c config.yaml -o simulated data gives me error Traceback (most recent call last): File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/cli/cli.py", line 133, in main cmd(args) File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/cli/commands/read_simulator.py", line 47, in execute read_simulator_runner(arguments.config, arguments.output) File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/read_simulator/runner.py", line 268, in read_simulator_runner output_file_writer = OutputFileWriter(options=options, File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/read_simulator/utils/output_file_writer.py", line 138, in __init__ with open_output(self.vcf_fn, mode=mode) as vcf_file: File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/common/io.py", line 103, in open_output handle = open_(output_path, mode=mode) File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/Bio/bgzf.py", line 277, in open raise ValueError("Bad mode %r" % mode) ValueError: Bad mode 'xt' ERROR: read-simulator failed, showing the last error Traceback (most recent call last): File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/cli/cli.py", line 133, in main cmd(args) File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/cli/commands/read_simulator.py", line 47, in execute read_simulator_runner(arguments.config, arguments.output) File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/read_simulator/runner.py", line 268, in read_simulator_runner output_file_writer = OutputFileWriter(options=options, File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/read_simulator/utils/output_file_writer.py", line 138, in __init__ with open_output(self.vcf_fn, mode=mode) as vcf_file: File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/common/io.py", line 103, in open_output handle = open_(output_path, mode=mode) File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/Bio/bgzf.py", line 277, in open raise ValueError("Bad mode %r" % mode) ValueError: Bad mode 'xt'

To Reproduce Fresh installation gives me the error Installation steps

conda env create -f environment.yml -n neat conda activate neat poetry build pip install dist/neat*whl And follow [contents of neat_config.yml] reference: H1N1.fasta read_len: 126 produce_bam: True produce_vcf: True paired_ended: True fragment_mean: 300 fragment_st_dev: 30 input_variants: H1N1.vcf mutation_rate: 0

neat read-simulator \ -c neat_config.yml \ -o /home/me/simulated_reads

Additional context Suggest me how to solve this problem?

[joshfactorial]

Apologies, we recently discovered this bug as well and are working through it.

---

From: Shafayet Islam @.> Sent: Monday, October 23, 2023 4:22 PM To: ncsa/NEAT @.> Cc: Subscribed @.***> Subject: [ncsa/NEAT] Can't run read-simulator. Probably dependency problem (Issue #87)

Describe the bug When running neat run-simulator -c config.yaml -o simulated data it gives me error Traceback (most recent call last): File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/cli/cli.py", line 133, in main cmd(args) File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/cli/commands/read_simulator.py", line 47, in execute read_simulator_runner(arguments.config, arguments.output) File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/read_simulator/runner.py", line 268, in read_simulator_runner output_file_writer = OutputFileWriter(options=options, File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/read_simulator/utils/output_file_writer.py", line 138, in init with open_output(self.vcf_fn, mode=mode) as vcf_file: File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/contextlib.py", line 135, in enter return next(self.gen) File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/common/io.py", line 103, in openoutput handle = open(output_path, mode=mode) File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/Bio/bgzf.py", line 277, in open raise ValueError("Bad mode %r" % mode) ValueError: Bad mode 'xt' ERROR: read-simulator failed, showing the last error Traceback (most recent call last): File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/cli/cli.py", line 133, in main cmd(args) File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/cli/commands/read_simulator.py", line 47, in execute read_simulator_runner(arguments.config, arguments.output) File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/read_simulator/runner.py", line 268, in read_simulator_runner output_file_writer = OutputFileWriter(options=options, File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/read_simulator/utils/output_file_writer.py", line 138, in init with open_output(self.vcf_fn, mode=mode) as vcf_file: File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/contextlib.py", line 135, in enter return next(self.gen) File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/neat/common/io.py", line 103, in openoutput handle = open(output_path, mode=mode) File "/home/shafayetrahat/miniconda3/envs/neat/lib/python3.10/site-packages/Bio/bgzf.py", line 277, in open raise ValueError("Bad mode %r" % mode) ValueError: Bad mode 'xt'

To Reproduce Fresh installation gives me the error Installation steps

conda env create -f environment.yml -n neat conda activate neat poetry build pip install dist/neat*whl And follow [contents of neat_config.yml] reference: hg19.fa read_len: 126 produce_bam: True produce_vcf: True paired_ended: True fragment_mean: 300 fragment_st_dev: 30 input_variants: NA12878.vcf mutation_rate: 0

neat read-simulator -c neat_config.yml -o /home/me/simulated_reads

Expected behavior A clear and concise description of what you expected to happen.

Additional context Suggest me how to solve this problem?

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/ncsa/NEAT/issues/87__;!!DZ3fjg!_NeX4BDAPeqOFiFnm6XRz4hGrnPiBb4-d9_JZ2JuMlpypINXeTsJs99Q2J4BLUbwEFD_CkP0MwhJbpfBCjmWtra4k7lnmQ$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AGMI7274TI6J5DAVBMBXWITYA3NZHAVCNFSM6AAAAAA6MWRNTSVHI2DSMVQWIX3LMV43ASLTON2WKOZRHE2TQMBUGQ3TIOA__;!!DZ3fjg!_NeX4BDAPeqOFiFnm6XRz4hGrnPiBb4-d9_JZ2JuMlpypINXeTsJs99Q2J4BLUbwEFD_CkP0MwhJbpfBCjmWtrZvEHXIwQ$. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[joshfactorial]

For now, you can use version 3 of NEAT until we get the fix in place.

[shafayetrahat]

Thanks
Just a small confusion: if I want to simulate with specified snps without any introduced sequence error in gold standard vcf file, is the above mentioned command enough or do I need to use other option also. I am asking because of this --- Unless specified, the simulator will also insert some "sequencing error" -- random variants in some reads that represents false positive results from sequencing.

[joshfactorial]

Yes, -M 0 should set the random mutation rate to 0.

---

From: Shafayet Islam @.> Sent: Tuesday, October 24, 2023 1:04 PM To: ncsa/NEAT @.> Cc: Allen, Josh @.>; Assign @.> Subject: Re: [ncsa/NEAT] Can't run read-simulator. Probably dependency problem (Issue #87)

Thanks [image]https://urldefense.com/v3/__https://user-images.githubusercontent.com/12217769/277759947-f3059666-459c-4b7b-927d-8176ff6bb7c2.png__;!!DZ3fjg!6V0dRgnTE_8tE3LIHDfQso69iIn-4ikhQXpgnDhGi7ft1DJi9aWuGuts1oDZ8sRaf2Rj3YqTG01RlSMCGFMsjmRBo4AvKg$ Just a small confusion: if I want to simulate with specified snps without any introduced sequence error in gold standard vcf file, is the above mentioned command enough or do I need to use other option also. I am asking because of this --- Unless specified, the simulator will also insert some "sequencing error" -- random variants in some reads that represents false positive results from sequencing.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/ncsa/NEAT/issues/87*issuecomment-1777756871__;Iw!!DZ3fjg!6V0dRgnTE_8tE3LIHDfQso69iIn-4ikhQXpgnDhGi7ft1DJi9aWuGuts1oDZ8sRaf2Rj3YqTG01RlSMCGFMsjmQ-Mmdutg$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AGMI72YXRQC42G273TSTPNTYA77MRAVCNFSM6AAAAAA6MWRNTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZXG42TMOBXGE__;!!DZ3fjg!6V0dRgnTE_8tE3LIHDfQso69iIn-4ikhQXpgnDhGi7ft1DJi9aWuGuts1oDZ8sRaf2Rj3YqTG01RlSMCGFMsjmRWaJ8KEA$. You are receiving this because you were assigned.Message ID: @.***>

[joshfactorial]

I take it back, this is the parameter of interest for sequencing errors:

parser.add_argument('-E', type=float, required=False, metavar='Error rate', default=-1, help="Rescale avg sequencing error rate to this, must be between 0.0 and 0.3")

so it would be something like -E 0.0

---

From: Shafayet Islam @.> Sent: Tuesday, October 24, 2023 1:04 PM To: ncsa/NEAT @.> Cc: Allen, Josh @.>; Assign @.> Subject: Re: [ncsa/NEAT] Can't run read-simulator. Probably dependency problem (Issue #87)

Thanks [image]https://urldefense.com/v3/__https://user-images.githubusercontent.com/12217769/277759947-f3059666-459c-4b7b-927d-8176ff6bb7c2.png__;!!DZ3fjg!6V0dRgnTE_8tE3LIHDfQso69iIn-4ikhQXpgnDhGi7ft1DJi9aWuGuts1oDZ8sRaf2Rj3YqTG01RlSMCGFMsjmRBo4AvKg$ Just a small confusion: if I want to simulate with specified snps without any introduced sequence error in gold standard vcf file, is the above mentioned command enough or do I need to use other option also. I am asking because of this --- Unless specified, the simulator will also insert some "sequencing error" -- random variants in some reads that represents false positive results from sequencing.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/ncsa/NEAT/issues/87*issuecomment-1777756871__;Iw!!DZ3fjg!6V0dRgnTE_8tE3LIHDfQso69iIn-4ikhQXpgnDhGi7ft1DJi9aWuGuts1oDZ8sRaf2Rj3YqTG01RlSMCGFMsjmQ-Mmdutg$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AGMI72YXRQC42G273TSTPNTYA77MRAVCNFSM6AAAAAA6MWRNTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZXG42TMOBXGE__;!!DZ3fjg!6V0dRgnTE_8tE3LIHDfQso69iIn-4ikhQXpgnDhGi7ft1DJi9aWuGuts1oDZ8sRaf2Rj3YqTG01RlSMCGFMsjmRWaJ8KEA$. You are receiving this because you were assigned.Message ID: @.***>

[joshfactorial]

I pushed a change to the Develop branch that should resolve this. Please let me know if your run works on the Dev branch!

[shafayetrahat]

Now, I am facing another error. I'm not very sure if it's my configuration file problem or something else.

My config.yml file

[neat_config] reference: H1N1.fa read_len: 126 produce_bam: True produce_vcf: True paired_ended: True fragment_mean: 300 fragment_st_dev: 30 input_variants: H1N1.vcf mutation_rate: 0

neat read-simulator \ -c neat_config.yml \ -o /home/me/simulated_reads

[shafayetrahat]

Sorry, it was from my config.yml file. After using reference: H1N1.fa read_len: 126 produce_bam: True produce_vcf: True paired_ended: True fragment_mean: 300 fragment_st_dev: 30 input_variants: H1N1.vcf mutation_rate: 0 Config I am getting

[shafayetrahat]

Can you please confirm if there's something wrong with your codebase or from my end?

[joshfactorial]

Yes, this is a bug we're working through. Did you try checking out the Develop branch? Did it still give you that error? Apologies. If you would like, the previous release (3.4) should be stable. It works a bit different, more command-line flags oriented, but it should produce good results.

[shafayetrahat]

The develop branch still giving this error.

[joshfactorial]

Gotcha, I will take a look this afternoon.

[joshfactorial]

Okay, I pushed a change to dev that I think takes care of this.

[shafayetrahat]

I think it still needs some change cause I am still facing some errors.

[joshfactorial]

okay, I think I resolved that.

[sfragkoul]

Hey there. I was wonderig if the bug has been resolved.

my config file has these args:

read_len: 126
produce_bam: True
produce_vcf: True
paired_ended: True
fragment_mean: 300
fragment_st_dev: 30 

The command I am running is tehe following: neat read-simulator -c neat_config2.yml -o simulated_data

I am using NEAT develop branch.

Thank a lot for all your efforts so far!

Stella

[joshfactorial]

Yes, I believe it is. Please try the develop branch and let me know if the error persists!

[sfragkoul]

Goodmoning and thanks for the quick response.

I tried both main and develop branches!

[joshfactorial]

Did you try again in the last day or so? I pushed a change to develop on 11/28 that I think addresses this.