search

Meteor-Community-Packages / organization

Discussions on organization of the organization 🎩
https://meteorjs.community/
41 stars 1 forks source link

New package for inclusion - vsivsi:job-collection #30

Closed mitar closed 2 years ago

mitar commented 4 years ago

Package/project name & description

job-collection is a powerful and easy to use job manager designed and built for Meteor.js. It solves the following problems (and more):

Links

Current status of the project

I think this is the best background task package for Meteor, supporting many use cases.

Progress

StorytellerCZ commented 4 years ago

Have you looked on jobs from Max Savin? https://github.com/msavin/SteveJobs

StorytellerCZ commented 4 years ago

Also I think at this point we need to get things moving on packages that we have move into the org before admitting any new packages.

mitar commented 4 years ago

We have people interested in maintaining, and they are already doing that in their fork, so not sure why making it more official and organized would be a bad thing?

LarsBuur commented 4 years ago

vsivsi:job-collection is mature and heavily battle-tested, also already written in CoffeeScript.

sebakerckhof commented 4 years ago

@StorytellerCZ Last time I looked steve jobs package was lacking some features compared to jobs collection. Most notably jobs collection allows you to have work distributed amongst multiple servers and to have workers outside of Meteor. There's a package for pure nodejs workers or you can even have workers in any other language using a rest api to fetch jobs.

vsivsi commented 4 years ago

job-collection author here. I fully support transitioning this package into community support. I'm no longer doing any active Meteor development, so this just makes sense given that others are still actively using it and willing to keep it going.

I would hope to be able to similarly transition its sister package file-collection.

Sorry for being late to the discussion, I've been out of action with a terrible case of the flu for a couple of weeks. Get those shots, it's awful!

vsivsi commented 4 years ago

A couple of wrinkles... The job-collection package has dependencies on some more generic node-js libs.

1) https://github.com/vsivsi/meteor-job is my own base library that enables pure node-js code to play with job-collection. But job-collection itself is also built on this, so it needs to somehow come along, even though it's not a pure Meteor package.

2) https://github.com/oortcloud/node-ddp-client The pure node-js DDP client (and everything it depends on, eg. https://github.com/oortcloud/ejson) is required for pure node-js worker authentication and interoperability.

3) https://github.com/vsivsi/ddp-login Which streamlines robust worker authorization flow.

There are probably more. The point being that just maintaining job-collection but neglecting these other pieces will probably lead to difficulty down the line, so be aware.

SimonSimCity commented 4 years ago

Any update on this? I've already created my own personal fork of this and am running it in production nowadays (https://atmospherejs.com/simonsimcity/job-collection). I'd like to get it back into a non-forked maintained stream.

I'm also on @StorytellerCZ that we might have to do some research here to investigate on where the different job packages are good in and where the individual implementation falls short in. I'm all in for diversity, but it should be reasonable when to join which path.

sebakerckhof commented 4 years ago

We are also using the fork from @SimonSimCity in our main product for a while now.

wildhart commented 4 years ago

Just a note on this - I am using my own fork/rewrite of msavin/sjobs, mainly due to this significant inefficiency in sjobs which is WONTFIX (every new job type/queue you create results in a 3 second setInterval which hits the db - my app currently has 20 different job queues/types!). Also, despite claiming to be multi-server, job queues cannot be paused across servers and the list of paused jobs do not survive a server reboot.

My own version is much more efficient as it uses a single observer on the job queue instead of 20 db lookups every 3 seconds! Mine can also persists paused job queues across multiple servers.

I haven't used job-collection and it looks awesome, but I personally would not recommend endorsing msavin/sjobs. Having said that, I wouldn't endorse mine either because I haven't written tests for it!

Edit: it looks like jobs-collection uses polling as well, which in my case could be very inefficient. The readme says:

job-collection must poll for this operation because it is time that is changing, not the contents of the database, so there are no database updates to listen for

My solution to that is to use an observer to watch for changes to a query looking for the next job across all queues, and within that observer set a timeout for the next due job. One mostly idle observer, one mostly idle timeout, for as many different job queues as you want. Simple, efficient and effective.

copleykj commented 4 years ago

I think we can move ahead with this. Does someone want to take the lead on getting the repo transferred and setting up orgs, teams, and permissions?

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

mitar commented 4 years ago

Where are we about this one?

copleykj commented 4 years ago

@mitar If we have maintainers in place, feel free to move forward.

mitar commented 4 years ago

I think maybe we could share it between @SimonSimCity and me? I am still interested in it, but at the moment a bit low on time.

SimonSimCity commented 3 years ago

@mitar I could help with the initial setup, but I'm no longer working on a project requiring it, so I'm not all too eager investing much time into it.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

mitar commented 3 years ago

Still planing. :-)

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

mitar commented 3 years ago

Unstale.

holmrenser commented 3 years ago

@mitar, I am using a copied version of job-collection as a local package at the moment, but a community-hosted package would have my preference. I can help out from time to time.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

holmrenser commented 3 years ago

Not stale from my perspective

StorytellerCZ commented 3 years ago

The setup is no problem. @mitar and @holmrenser if you want to become the maintainers on this then that is enough for me to get this started.

holmrenser commented 3 years ago

Sure thing! Let me know what I can do!

mitar commented 3 years ago

I am still interested in this (using it in some of my legacy apps) but it seems I will not have time to do anything concrete about this anytime soon. :-(

holmrenser commented 3 years ago

Proposal: @mitar and myself become maintainers, we put some (limited) effort into migrating this to meteor-community-packages as is, which I can probably mostly take care of, but might want to have some feedback from @mitar on. Afterwards we can think about updates/changes and review pull requests. I have a few modifications that might be interesting to include, but we can think about that once this is safely incorporated into meteor-community-packages.

@StorytellerCZ what do you think?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

holmrenser commented 3 years ago

Not stale. Pinging @mitar and @StorytellerCZ

holmrenser commented 3 years ago

@StorytellerCZ Reminder

StorytellerCZ commented 3 years ago

Alright, first step is to add communitypackages to the package on Atmosphere and then we can start on the migration.

StorytellerCZ commented 3 years ago

On another note, we can now also migrate in: https://github.com/percolatestudio/meteor-synced-cron

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.