Open erick-dorlass opened 2 years ago
Cantalapiedra
commented
2 years ago Hi @erick-dorlass ,
You may try hmm_server.py -d virus to check if the hmmer server works, and, if it doesn't, hopefully get some error message that could give us a clue.
Best, Carlos
Best, Carlos
erick-dorlass
commented
2 years ago Hi @Cantalapiedra Cantalapiedra I appreciate your quick response.
I tried the hmm_server.py -d virus, and the process also stuck in "Waiting for server to become ready at localhost:51700 ..."
When I kill the process, this is the message provided:
CPUs per worker: 1 1.0 hmm_server.py -d eggnog_data/virus/virus.hmm Preparing to query custom database eggnog_data/virus/virus.hmm create_servers: hmmdb:eggnog_data/virus/virus.hmm:localhost:51700-53200 Creating server number 1/1 Loading server at localhost, port 51700-51701 Creating hmmpgmd server at port 51700 ... Creating hmmpgmd workers (1) at port 51701 ... Waiting for server to become ready at localhost:51700 ... Loading master: /home/ec2-user/miniconda3/envs/emapper/bin/hmmpgmd --master --cport 51700 --wport 51701 --hmmdb eggnog_data/virus/virus.hmm Loading worker: /home/ec2-user/miniconda3/envs/emapper/bin/hmmpgmd --worker localhost --wport 51701 --cpu 1 ..^C Jaime Huerta Cepas GPL v2 : A server for HMMER3 in-memory searches Process Process-2: Process Process-1: Total time: 57.6704 secs Traceback (most recent call last): Traceback (most recent call last): File "/home/ec2-user/miniconda3/envs/emapper/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/ec2-user/miniconda3/envs/emapper/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/ec2-user/miniconda3/envs/emapper/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) File "/home/ec2-user/miniconda3/envs/emapper/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(self._args, **self._kwargs) File "/home/ec2-user/miniconda3/envs/emapper/lib/python3.10/site-packages/eggnogmapper/search/hmmer/hmmer_server.py", line 203, in __start_master_subprocess time.sleep(60) File "/home/ec2-user/miniconda3/envs/emapper/lib/python3.10/site-packages/eggnogmapper/search/hmmer/hmmer_server.py", line 217, in __start_worker_subprocess time.sleep(60)
I have tried running it with different versions of python (3.7, 3.8 and 3.10) however the result is the same Also, emapper was installed with conda. when I try the same command with emapper installed with pip, this is the message i get:
CPUs per worker: 1
1.0
hmm_server.py -d virus.hmm
Preparing to query custom database virus.hmm
create_servers: hmmdb:virus.hmm:localhost:51700-53200
Creating server number 1/1
Loading server at localhost, port 51700-51701
Creating hmmpgmd server at port 51700 ...
Creating hmmpgmd workers (1) at port 51701 ...
Waiting for server to become ready at localhost:51700 ...
Loading master: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/bin/hmmpgmd --master --cport 51700 --wport 51701 --hmmdb virus.hmm
Process Process-1:
Loading worker: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/bin/hmmpgmd --worker localhost --wport 51701 --cpu 1
Traceback (most recent call last):
Process Process-2:
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/search/hmmer/hmmer_server.py", line 201, in start_master_subprocess
CHILD_PROC = subprocess.Popen(cmd.split(), shell=False, stderr=OUT, stdout=OUT)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 966, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1842, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/bin/hmmpgmd'
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/search/hmmer/hmmer_server.py", line 215, in __start_worker_subprocess
CHILD_PROC = subprocess.Popen(cmd.split(), shell=False, stderr=OUT, stdout=OUT)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 966, in init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1842, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/bin/hmmpgmd'
Loading server at localhost, port 51702-51703
Creating hmmpgmd server at port 51702 ...
Creating hmmpgmd workers (1) at port 51703 ...
Waiting for server to become ready at localhost:51702 ...
Loading worker: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/bin/hmmpgmd --worker localhost --wport 51703 --cpu 1
Process Process-4:
Loading master: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/bin/hmmpgmd --master --cport 51702 --wport 51703 --hmmdb virus.hmm
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/search/hmmer/hmmer_server.py", line 215, in start_worker_subprocess
CHILD_PROC = subprocess.Popen(cmd.split(), shell=False, stderr=OUT, stdout=OUT)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 966, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1842, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/bin/hmmpgmd'
Process Process-3:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/search/hmmer/hmmer_server.py", line 201, in __start_master_subprocess
CHILD_PROC = subprocess.Popen(cmd.split(), shell=False, stderr=OUT, stdout=OUT)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 966, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1842, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/bin/hmmpgmd'
Loading server at localhost, port 51704-51705
Creating hmmpgmd server at port 51704 ...
Creating hmmpgmd workers (1) at port 51705 ...
Waiting for server to become ready at localhost:51704 ...
Loading worker: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/bin/hmmpgmd --worker localhost --wport 51705 --cpu 1
Loading master: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/bin/hmmpgmd --master --cport 51704 --wport 51705 --hmmdb virus.hmm
Process Process-6:
Process Process-5:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/search/hmmer/hmmer_server.py", line 215, in start_worker_subprocess
CHILD_PROC = subprocess.Popen(cmd.split(), shell=False, stderr=OUT, stdout=OUT)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 966, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1842, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/bin/hmmpgmd'
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/search/hmmer/hmmer_server.py", line 201, in __start_master_subprocess
CHILD_PROC = subprocess.Popen(cmd.split(), shell=False, stderr=OUT, stdout=OUT)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 966, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1842, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/bin/hmmpgmd'
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/search/hmmer/hmmer_server.py", line 73, in create_servers
sdbpath, shost, sport, master_pid, workers_pids = start_server(dbpath, host, sport, end_port, cpus_per_worker, num_workers, dbtype, silent = silent)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/eggnogmapper/search/hmmer/hmmer_server.py", line 141, in start_server
raise Exception("Could not start server after trying {ports_tried} ports.")
Exception: Could not start server after trying {ports_tried} ports.
Could not create server number 1/1. Fails: 1
Created 0 out of 1
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/bin/hmm_server.py", line 138, in
Jaime Huerta Cepas GPL v2 : A server for HMMER3 in-memory searches Total time: 3.05523 secs
It seems that are more tries to create a server with the pip install version, however the result is the same.
Cantalapiedra
commented
2 years ago Hi,
It might be the hmmer version. Which one are you using?
erick-dorlass
commented
2 years ago Hi,
I'm using HMMER 3.3.2, but i also tried with versions 3.1b2, 3.2, 3.2.1, 3.3 and 3.3.1. The same error was reported in all versions.
Cantalapiedra
commented
2 years ago Hi,
ok. Then it must be another thing. How do you swap between HMMER versions? I guess you are sure that you are trying different versions and not a single version is interfering from the PATH or conda.
Could you try putting the command which is under the eggnog-mapper directory, at eggnogmapper/bin in the PATH just to be sure that it is not due to hmmer version?
What is the complete command which you used to create the "viral" database? I could try to replicate with the same database.
Also, what is your OS distribution and version?
Best, Carlos
Cantalapiedra
commented
2 years ago Also, you may try to load the DB directly using the hmmer command "hmmgpmd", and see if it is causing the problem, or instead it is an eggnog-mapper problem itself.
erick-dorlass
commented
2 years ago Hi,
I was able to create a server (master and worker) with hmmpgmd:
hmmpgmd --master --cport 51700 --hmmdb virus.hmm
hmmpgmd --worker localhost --wport 51701 --cpu 2
However when I try using:
emapper.py -m hmmer -d virus --data_dir eggnog_db --servers_list servers.txt -i scaffolds.faa -o test
or
emapper.py -m hmmer -d virus.hmm:localserver:51700 --data_dir eggnog_db -i scaffolds.faa -o test
I receive the message: hmmer.py:search DB: virus.hmm:localserver:51700 idmap file virus.hmm.idmap not found
Is there a way to create de idmap file in the server or copy it from my local machine? And this will also have to be performed to all hmmpress files?
The viral hmm database is being downloaded from http://eggnog5.embl.de/download/eggnog_5.0/per_tax_level/10239/, all HMM files are concatenated with cat to a virus.hmm file and then I use hmmpress to create h3 files. I'm creating the idmap file by parsing the virus.hmm file as in hmmer_idmap.py script. Actually the "viral" database is being used as a test for a larger hmm database created with eggnog for metagenome analysis
Until now, the hmmer_server.py script don't seem to work, so the option I'm considering is creating the servers with hmmpgmd, create a server list file and pass it as an input to emapper.py.
I tested the HMMER versions with conda. I created a new env with eggnog-mapper, uninstall the HMMER downloaded together with eggnog-mapper and install different hmmer versions for testing
I'm using a MacOS Monterey v12.3.1
erick-dorlass
commented
2 years ago Hi @Cantalapiedra
I performed a few more tests, and indeed was the hmmer version. Once I installed eggnog-mapper version2.1.7 with pip in a conda env with hmmer v3.1b2 the --usemem, hmmpgmd and hmm_server.py started to work fine.
However I'm facing another problem. Once I use:
emapper.py -m hmmer -d viral --usemem --data_dir eggnog_db/ -i test.fa -o output
The servers are uploaded, however I receive the following message:
Error: Failed to open target sequence database /home/ec2-user/erick/eggnog_db/hmmer/viral/2756.fa for reading
multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/home/ec2-user/miniconda3/envs/eggnog/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "/home/ec2-user/miniconda3/envs/eggnog/lib/python3.7/site-packages/eggnogmapper/search/hmmer/hmmer_search.py", line 328, in refine_hit best_hit = get_best_hit(F.name, group_fasta, excluded_taxa, tempdir) File "/home/ec2-user/miniconda3/envs/eggnog/lib/python3.7/site-packages/eggnogmapper/search/hmmer/hmmer_search.py", line 360, in get_best_hit raise ValueError('Error running PHMMER') ValueError: Error running PHMMER """
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ec2-user/miniconda3/envs/eggnog/bin/emapper.py", line 704, in
The .hits file is generated normally, however the "hit" column reads: 2756 instead of a HMM ID (4QCXI):
query_name hit evalue sum_score query_length hmmfrom hmmto seqfrom seqto query_coverage LNCMODLF_00001 2756 1.7e-210 699.0 545 1 510 1 525 0.9614678899082569
It seems that the tool looks for a X.fa for phmmer to generate the .seed.orthologs file. The ".fa" files downloaded with the command download_eggnog_data.py have the HMM name (a.i 4QCXI.fa).
I used the following command to generate .fa files with the coordinate avaiable in the idmap file:
for line in $(ls *.fa | cut -d "." -f 1); do X=$(grep $line viral.hmm.idmap | awk -F ' ' '{print $1}'); cp ${line}.fa${X}.fa; done
The seed.orthologs file was generated, but a new error apperars:
Processed queries:26 total_time:1.509143352508545 rate:17.23 q/s 26 1.5095138549804688 17.22 q/s (% mem usage: 4.10, % mem avail: 95.94) Traceback (most recent call last): File "/home/ec2-user/miniconda3/envs/eggnog/lib/python3.7/site-packages/psutil/_common.py", line 449, in wrapper ret = self._cache[fun] AttributeError: _cache
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/ec2-user/miniconda3/envs/eggnog/lib/python3.7/site-packages/psutil/_pslinux.py", line 1516, in wrapper return fun(self, *args, kwargs) File "/home/ec2-user/miniconda3/envs/eggnog/lib/python3.7/site-packages/psutil/_common.py", line 452, in wrapper return fun(self) File "/home/ec2-user/miniconda3/envs/eggnog/lib/python3.7/site-packages/psutil/_pslinux.py", line 1558, in _parse_stat_file with open_binary("%s/%s/stat" % (self._procfs_path, self.pid)) as f: File "/home/ec2-user/miniconda3/envs/eggnog/lib/python3.7/site-packages/psutil/_common.py", line 713, in open_binary return open(fname, "rb", kwargs) FileNotFoundError: [Errno 2] No such file or directory: '/proc/6860/stat'
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/ec2-user/miniconda3/envs/eggnog/lib/python3.7/site-packages/psutil/init.py", line 373, in _init self.create_time() File "/home/ec2-user/miniconda3/envs/eggnog/lib/python3.7/site-packages/psutil/init.py", line 723, in create_time self._create_time = self._proc.create_time() File "/home/ec2-user/miniconda3/envs/eggnog/lib/python3.7/site-packages/psutil/_pslinux.py", line 1516, in wrapper return fun(self, *args, **kwargs) File "/home/ec2-user/miniconda3/envs/eggnog/lib/python3.7/site-packages/psutil/_pslinux.py", line 1728, in create_time ctime = float(self._parse_stat_file()['create_time']) File "/home/ec2-user/miniconda3/envs/eggnog/lib/python3.7/site-packages/psutil/_pslinux.py", line 1523, in wrapper raise NoSuchProcess(self.pid, self._name) psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=6860)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ec2-user/miniconda3/envs/eggnog/bin/emapper.py", line 704, in
Is there a way to generate te hits file with the correct HMM ID? Should I make an alteration in the downloaded bank? Thank you a lot for the help so far.
Cantalapiedra
commented
2 years ago Hi @erick-dorlass ,
sorry for the delay answering.
The .idmap file issue could be due to format. Actually, when using
download_eggnog_data.py -D -H -d 10239
The .idmap file generated is using tabs for separating the 2 fields, whereas the parser expects single whitespace.
So this looks like a bug we will need to fix. However, I am not sure whether it is the exact same problem you have, since you said that you used the generate_idmap function within hmmer_idmap.py, which should generate an idmap file with single whitespace as field separator.
What does your .idmap file look like?
I have been checking the psutil errors, and I am afraid this is also a bug. The results (annotations, etc) should be ok, since this is a final step in which the processes running hmmpgmd server and workers are shutdown. In this case, the servers are already down, and therefore psutil cannot find the process and raises and exception. I could help you to code a workaround in case that you need it fixed ASAP. I will try to have it fixed at least in the master branch soon. Hopefully also the conda version.
Best, Carlos
Cantalapiedra
commented
2 years ago I hope this is of help:
https://github.com/eggnogdb/eggnog-mapper/releases/tag/2.1.8 https://pypi.org/project/eggnog-mapper/2.1.8/
Hopefully the conda version will be ready soon.
Best, Carlos
Hello,
I'm using eggnog-mapper for a metagenomic pipeline with hmm searches.
For optimization, i'm using "--usemem" parameter, however the process seems to be stuck in "Waiting for server to become ready". I've seen this same problem been related in other issues as #287 and #33 but without a clear solution. Have this parameter ever worked?
I'm using eggnog-mapper v2.1.7 the command:
emapper.py -m hmmer -d virus --data_dir eggnog_data --usemem -i scaffolds.faa -o test