Closed TS404 closed 6 years ago
I have had similar experience with get_hmm
. Some days the hmmscan
server is very unresponsive. Batch upload to hmmscan
is even worse. Uploads are put in a queue and sometimes hours pass just to start.
Currently I would like to change get_hmm
to resubmit the sequence after some time if the result is not provided. If the 2nd submit hangs, the functions ends returning results for the sequences up to there and an Error message. I trust this is the best solution in this case.
For 10k+ sequences I recommend using hmmer
. Perhaps a function that will take output from hmmer
and import it in the same format as output of hmmscan
function?
I haven't had this problem with get_big_pi
, usually only sequences containing an N-sp are sent to big pi
and in my experience it works solid for up to 5k sequences.
Batch queries would be a good addition (perhaps faster), this requires a complete rewrite of the function. I will do some testing and then decide if we shall go in this direction.
Progress bar would be a nice addition, perhaps when verbose = FALSE
a progress bar is displayed. for both functions and even for the *_file
functions.
Good idea for get_hmm
! I think that's a very sensible way to do it.
I've not managed to recreate the get_big_pi
issue so perhaps there was something odd about the time it happened to me.
I have managed to speed up get_big_pi
significantly, and I have implemented the progress bar as per the suggestion. I just need to perform some checking before deployment. I trust it will be available during the weekend. I think I will update all the scraping functions with progress bars.
get_big_pi
has been updated. The update should provide a significant speed up, and it should be now more in line with the speed of get_phobius
.
For instance:
system.time(
test_big_pi <- ragp::get_big_pi(at_nsp[1:1000,],
sequence,
Transcript.id,
simplify = FALSE)
)
#output
user system elapsed
3.53 0.13 57.88
Bugs are possible, if you stumble upon any please report them.
get_hmm
has been updated.
New arguments are:
timeout
- time in seconds to wait for the server response (default = 10s).attempts
- number of attempts if server unresponsive (default = 2 times).If number of attempts is exhausted the function will issue a warning and return the queries finished so far.
Additionally a progress bar has been added.
Bugs are possible, if you stumble upon any please report them.
Thank you for suggestions.
Closing this issue.
get_hmm
andget_big_pi
can take a while. When submitting >1000 sequences, I've tended to do them individually in order to salvage the results if it becomes unresponsive after an hour. I don't know if tghe server prefers batch queries rather than repeated individual queries, but even so, sequences could be submitted in batches of 10 or 50 to give an idea of estimated time.