Job Queing

[xiy]

(Feature, Low Priority)

Not sure if this has been brought up before but I'm sure a lot of people would like the ability to queue BLAST jobs up, similar to the official service, and get a unique ID to refer back to the results.

This could be easily implemented with Resque/Redis.

Probably pretty low priority at the moment but just an idea. Thoughts?

[yeban]

I don't think it has, but it is always there at the back of my head. And surely @yannickwurm and @wwood have thought about it too give that BlastStation has it, and so does NCBI's web blast service.

Feel free to take the lead if it interests you. I will be glad to discuss the details. Though, I would find it more comforting if other devs can focus on priority stuff first :).

It's important that the installation doesn't get any complicated than what it is now, so that the ease of setup for individuals and smaller organizations isn't sacrificed. So you may first want to come up with something that will work on a minimal Linux installation + Ruby. And interested admins can swap queuing backend to Redis (or whatever) if they want.

[yannickwurm]

don't think we should implement queuing. It can be faked by replacing "e.g. blastx" binary with a script that interactively redirects the arguments to a cluster.

But having semi-permanent links to old searches (or search results) could indeed be very handy (for Electronic lab notebooks, for sharing...etc).

We need to get some of the other issues sorted first tho :D

Cheers!

[yeban]

On Fri, Sep 21, 2012 at 12:11 AM, Yannick Wurm notifications@github.comwrote:

don't think we should implement queuing. It can be faked by replacing "e.g. blastx" binary with a script that interactively redirects the arguments to a cluster.

I think the point of queuing is to submit a long search and be notified about it later when the results are ready.

But having semi-permanent links to old searches (or search results) could indeed be very handy.

This can be implemented directly on the file system (no external databases -> easier deployment). Maintain a directory of blast searches in Archive format. Search id (a UIUD prefixed with 'ss-' perhaps) being the file name of the result. Which in turn can be directly mapped to a URI. /searches/search_id. Of course, SS will have to periodically clean up that directory.

Anurag Priyam

[xiy]

Woops!

I do agree that it should be as easy a deployment as possible, although running a Redis server is quite trivial (could be done with setup scripts). The only problem with implementing the queuing through the file system (UNIX job queues?) is for Windows users. Unfortunately there are some people who are forced to run the software on that OS. Is that actually possible on Windows?

[malcook]

You might consider decoupling FE/BE by implementing NCBI's URL API

http://www.ncbi.nlm.nih.gov/staff/tao/URLAPI/new/BLAST_URLAPI.html

Doing so would enable other blast clients, both UIs and libraries, to nicely interoperate. Imagine: a SS site whose content is also exposed by URL API implemented by you potentially allowing usage by: Vector NTI biopython ... more

[yeban]

Closing this in favour of #181. We won't ever implement a full-blow queuing system within SequenceServer. And our URL API will likely take a much simpler approach than NCBI's. Something like /rid -> html results, /rid.xml -> XML results, /rid.tsv -> tabular results.

[malcook]

+1 for dropping the full-blown queuing system -1 for not adopting the NCBI's URL API. We're you to adopt it, sequence server would be inter-operable with other clients that employ this URL API (such as Vector NTI Advanced).

Just sayin'

+100 for sequencserver in the first place

[yeban]

Thanks for your input, @malcook. And I hope you will continue to remain involved :). I think a strong case for interop can really be made if our user base grows 1000 fold. In the meantime our focus is on simple things - improve the search UX (database widget, advanced params) and make incremental refinements to the results (visualisations, taxonomy data).

[malcook]

Fair enough. So, not -1 then. Only -.001 ;) Thanks!

On Wed, Mar 18, 2015 at 10:34 AM, Anurag Priyam notifications@github.com wrote:

Thanks for your input, @malcook https://github.com/malcook. And I hope you will continue to remain involved :). I think a strong case for interop can really be made if our user base grows 1000 fold. In the meantime our focus is on simple things - improve the search UX (database widget, advanced params) and make incremental refinements to the results (visualisations, taxonomy data).

— Reply to this email directly or view it on GitHub https://github.com/yannickwurm/sequenceserver/issues/96#issuecomment-83022936 .