search

shahab-sarmashghi / RESPECT

Estimating repeat spectra and genome length from low-coverage genome skims
Other
11 stars 1 forks source link

Error. nthreads cannot be larger than environment variable "NUMEXPR_MAX_THREADS" (64) #15

Closed szf1993325 closed 1 year ago

szf1993325 commented 1 year ago

This error appear when i finish setup, i don't how to resolve. Anyone have suggestion?

shahab-sarmashghi commented 1 year ago

So this happens before even running?

szf1993325 commented 1 year ago

Every time. even if i'm not input data. just write the code "respect -h". If i give input data the other error appear, like this " Error. nthreads cannot be larger than environment variable "NUMEXPR_MAX_THREADS" (64)2023-07-05 23:34:49,825 INFO:Processing T80.fq.gz... 2023-07-05 23:35:12,501 INFO:compute_kmer_histogram finished in 13.619732141494751 seconds 2023-07-05 23:35:12,501 ERROR:Error occurred when processing /home/chrom/test/T80.fq.gz; it's skipped Traceback (most recent call last): File "/home/soft/miniconda3/lib/python3.9/site-packages/respect-1.3.0-py3.9.egg/respect/respect_functions.py", line 247, in run_respect parameter_estimator.set_kmer_histogram(args.threads, args.decomp) File "/home/soft/miniconda3/lib/python3.9/site-packages/respect-1.3.0-py3.9.egg/respect/paramter_estimator.py", line 217, in set_kmer_histogram self.compute_kmer_histogram(n_threads, decomp_util) File "/home/soft/miniconda3/lib/python3.9/site-packages/respect-1.3.0-py3.9.egg/respect/timer.py", line 68, in wrapper_timer return func(*args, *kwargs) File "/home/soft/miniconda3/lib/python3.9/site-packages/respect-1.3.0-py3.9.egg/respect/paramter_estimator.py", line 174, in compute_kmer_histogram profiler_output = kmer_profiler(self.input_file, self.sequence_type, self.output_name, self.tmp_dir, File "/home/soft/miniconda3/lib/python3.9/site-packages/respect-1.3.0-py3.9.egg/respect/profiling.py", line 120, in kmer_profiler histo_stderr = check_output(["jellyfish", "histo", "-h", str(highest_histo_multiplicity), mercnt], stderr=STDOUT, File "/home/soft/miniconda3/lib/python3.9/subprocess.py", line 420, in check_output return run(popenargs, stdout=PIPE, timeout=timeout, check=True, File "/home/soft/miniconda3/lib/python3.9/subprocess.py", line 524, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['jellyfish', 'histo', '-h', '100000000', '/home/chrom/test/tmp/T80.fq.gz/T80.fq.gz.jf']' returned non-zero exit status 1. Traceback (most recent call last): File "/home/soft/miniconda3/bin/respect", line 33, in sys.exit(load_entry_point('respect==1.3.0', 'console_scripts', 'respect')()) File "/home/soft/miniconda3/lib/python3.9/site-packages/respect-1.3.0-py3.9.egg/respect/main.py", line 139, in main File "/home/soft/miniconda3/lib/python3.9/site-packages/respect-1.3.0-py3.9.egg/respect/respect_functions.py", line 262, in run_respect File "/home/soft/miniconda3/lib/python3.9/multiprocessing/context.py", line 119, in Pool return Pool(processes, initializer, initargs, maxtasksperchild, File "/home/soft/miniconda3/lib/python3.9/multiprocessing/pool.py", line 205, in init raise ValueError("Number of processes must be at least 1") ValueError: Number of processes must be at least 1"

shahab-sarmashghi commented 1 year ago

It seems you have many cores available on your machine and RESPECT is trying to use all of them but there is a limit set by the system. Can you try setting the number of requested cores to a smaller number? For example by adding --threads 8 option when running RESPECT?

szf1993325 commented 1 year ago

This error have different when i set '--threads 8'. but i just set '--threads 2' before Error. nthreads cannot be larger than environment variable "NUMEXPR_MAX_THREADS" (64)2023-07-05 23:56:29,115 INFO:Processing T80.fq.gz... 2023-07-05 23:58:30,303 INFO:compute_kmer_histogram finished in 112.02907919883728 seconds 2023-07-05 23:58:30,725 INFO:Starting iterations to estimate parameters of T80.fq.gz 2023-07-05 23:58:31,093 INFO:Set parameter Username 2023-07-05 23:58:31,199 INFO:estimate_genome_skim_parameters finished in 0.844153881072998 seconds 2023-07-05 23:58:31,199 ERROR:Error occurred when estimating parameters for /home/chrom/T80.fq.gz; it's skipped Traceback (most recent call last): File "/home/soft/miniconda3/lib/python3.9/site-packages/respect-1.3.0-py3.9.egg/respect/paramter_estimator.py", line 344, in call return self.estimate_genomic_parameters(*args, *kwargs) File "/home/soft/miniconda3/lib/python3.9/site-packages/respect-1.3.0-py3.9.egg/respect/paramter_estimator.py", line 326, in estimate_genomic_parameters self.estimate_genome_skim_parameters(spectra_number, error_norm, iterations_number, min_r1l, temperature) File "/home/soft/miniconda3/lib/python3.9/site-packages/respect-1.3.0-py3.9.egg/respect/timer.py", line 68, in wrapper_timer return func(args, *kwargs) File "/home/soft/miniconda3/lib/python3.9/site-packages/respect-1.3.0-py3.9.egg/respect/paramter_estimator.py", line 281, in estimate_genome_skim_parameters optimizer.run_simulated_annealing(iterations_number, min_r1l, temperature) File "/home/soft/miniconda3/lib/python3.9/site-packages/respect-1.3.0-py3.9.egg/respect/optimizer.py", line 395, in run_simulated_annealing repeat_spectra_next = self.estimate_repeat_spectra(o[1:], poisson_matrix_next[1:, :]) File "/home/soft/miniconda3/lib/python3.9/site-packages/respect-1.3.0-py3.9.egg/respect/optimizer.py", line 335, in estimate_repeat_spectra spectral_residuals = [1.0 constrained_spectra[i] / norm(constrained_spectra[i:], ord=1) for i in range( File "/home/soft/miniconda3/lib/python3.9/site-packages/respect-1.3.0-py3.9.egg/respect/optimizer.py", line 335, in spectral_residuals = [1.0 * constrained_spectra[i] / norm(constrained_spectra[i:], ord=1) for i in range( TypeError: 'NoneType' object is not subscriptable 2023-07-05 23:58:31,202 ERROR:Error occurred while trying to get estimated parameters for a sample 2023-07-05 23:58:31,209 INFO:Writing the results to the output files...

shahab-sarmashghi commented 1 year ago

I just changed the version requirements for some of the prerequisites and made a commit to the master branch. Can you clone/download the repo again and install RESPECT into a new conda environment and see if the error is resolved?

szf1993325 commented 1 year ago

Thank you very much. I try again, the similar error appear "Error. nthreads cannot be larger than environment variable "NUMEXPR_MAX_THREADS" (64)" (this conda enverioment version was 23.1.0), but when i run my data not other error appear. I think maybe the conda version was too newly, because when i install download package in older conda version 4.8.4, no error appear and quickly get output file. In all, i think RESPECT was a convenient and quickly software.

shahab-sarmashghi commented 1 year ago

Okay I'm glad that it worked. I think the error is related to newer versions of pandas or numpy and I changed the requirements so that only versions that I had tested with are installed and not the latest.