search

vsivsi / meteor-job-collection

A persistent and reactive job queue for Meteor, supporting distributed workers that can run anywhere.
https://atmospherejs.com/vsivsi/job-collection
Other
387 stars 68 forks source link

restarting jobs without spawning new ones #139

Closed dasdeck closed 6 years ago

dasdeck commented 8 years ago

Hi!

I'd lie to just restart jobs without respawning them. Is this possible? For convenience I'd use just one collection for the job data and the actual jobs.

Cheers!

JM

vsivsi commented 8 years ago

I don't think I understand the question. Is this related to you trying to use job.pause() on a running job in the code you included with the other issue? Please explain what you are actually trying to accomplish, and I'll be happy to suggest some ways to do that using jC.

dasdeck commented 8 years ago

Yes, sorry, and thanks for responding to my poorly formulated question. I want to use my job collection in a persistent manner, and jobs that have been completed should be able to run again, without spawning another database entry. I have some jobs that are importing a large data file into a database, according to some parameters. The user may change paramters and run the job again to have a different importation result.

I think I'm supposed to keep two collections, one for the paramter sets, and one for the actual jobs doing the work. but having both in one collection solves a lot of porblems, for example not having to restrict number of jobs per setting, etc.

I hope I could be more concise this time.

vsivsi commented 8 years ago

Oh, I see. Well, within the architecture of job-collection this isn't possible. Each run of a given job needs its own state. If you wanted to implement this yourself on the server-side, you certainly could (by creating a new Meteor.method() to implement it, and then calling that from your worker to manipulate the job state.) However, I wouldn't recommend this, and certainly couldn't provide any technical support for it (ie. ensure that it did break with some future update to jC).

Another possibility that might work for you is to just keep your "result" documents in the job-collection mongoDB collection. MongoDB collections have no schema, and job-collection should completely ignore documents that don't match the queries it makes. So as long as your "result" documents don't collide with the attribute namespace of the job-collection data model then you should be fine. To ensure forward compatibility, I would use a unique prefix on attribute names to ensure that no collisions are likely in the future as well. This approach would keep everything in one collection, and has the advantage of requiring no changes to job-collection.

Hope that helps.