Thoughts on management UI?

[dandv]

I'm eager to get a queue management UI working and wanted to check with you for the general architecture and any guidance you might have. Final goal would be to integrate this with Houston, but it can function standalone for now:

• [ ] list queues and jobs
• [ ] show overall progress
• [ ] show progress per job
• [ ] pause/resume/remove job from queue
• [ ] pause entire queue
• [ ] bonus points for further editing, e.g. of the data payload

Our use case is infinitely repeating jobs: ~100 of one type (feed parsing), and several jobs of different types. Makes sense to group by type.

I'm thinking of using jQuery DataTables for its filtering, sorting and editing capabilities, and an admin theme like SB Admin 2.

[vsivsi]

The sample app is a good place to start if you haven't looked at it yet. It certainly doesn't do everything on your list above, but it'll give you an idea of how I've connected jobCollection method calls to a reactive UI, etc. as a first cut. https://github.com/vsivsi/meteor-file-job-sample-app/

[LarsBuur]

I am trying to build a simple admin ui for the package but I run into a couple of problems:

1) The example is rather old - but when I convert it into a running meteor 1.0.1 then the list of running jobs is empty (also when I create a user) and drop images on the gallery. The console reports of many jobs being processed but they are never exposed in the jobcollection.find.

The docs might be out of sync. "job-collections are backed by Meteor Collections and may be used in similar ways. .find() and .findOne() work as you would expect and are fully reactive on the client just as with a normal collection."

I am unable to call a find og findOne method on my jobcollection.

[vsivsi]

@LarsBuur Hi, thanks for your question. I'm sorry but I'm on vacation at the moment and won't be back home until January 3rd. I only have my iPad with me, so I am very limited in the help I can give right now.

The main thing to know is that to use a jobCollection on the client, you must publish it as you would any other kind of collection. And to manipulate it, you must write allow/deny rules to give the proper permissions to the authenticated user. If your jobCollection is working on the server, but appears empty on the client, it is almost certainly a problem with how it is being published on the server. Without seeing your code it is hard for me to speculate why your code isn't working as expected. But as I said above, I have no ability to debug a meteor app until I return home.

[LarsBuur]

Hi Vaughn,

Thank you very much for your reply. Please enjoy you vacation and reply when back if possible.

I think my problem is more general than not being able define the correct allow/deny rules.

The following code fails with "undefined is not a function" on the .find() call - and it seems that is not possible to call .find() on a JobCollection on the client. I wont expect the following code to return any jobs - but I would expect it to just return an empty result or an empty array.

if(Meteor.isClient){
    var myJobs = JobCollection('myJobQueue');
    Meteor.subscribe('allJobs');
    var jobs = myJobs.find();
    console.log('myJobs:' + JSON.stringify(jobs));
}

For now I have made an meteor method that returns the job collection from the server and then parse it to a reative variable so I can iterate on it using {{#each jobs}} - but it is definitely a hack that I would like to avoid in the future.

Happy holidays, Lars Buur

[kennyatbusheldirect]

Hello,

I recently had a similar issue and thought I'd share my fix to see if it might help. I was getting records returned on the server, but nothing on the client. After lots of testing and setting up a new project with just the job collection files, it turned out to be a small issue on the server.

In my "Collections" folder i have a file called "jobs.js" where i was setting up the jobs collection.

Previously, this is how that file started:

myJobs = JobCollection('myJobQueue');

That is setup so that "myJobs" is available on both the client and the server, meaning all you need to do is subscribe on the client and publish from the server. That worked for earlier versions, but after upgrading i had to change it to:

myJobs = new JobCollection('myJobQueue');

Once i did that, all my jobs were showing up on the client as i had expected. It took me days of tryng to figure it out and it was so simple in the end. Maybe it could save you some time as well.

-Kenny

Date: Thu, 18 Dec 2014 01:15:50 -0800 From: notifications@github.com To: meteor-job-collection@noreply.github.com Subject: Re: [meteor-job-collection] Thoughts on management UI? (#14)

Hi Vaughn,

Thank you very much for your reply. Please enjoy you vacation and reply when back if possible.

I think my problem is more general than not being able define the correct allow/deny rules.

The following code fails with "undefined is not a function" on the .find() call - and it seems that is not possible to call .find() on a JobCollection on the client.

I wont expect the following code to return any jobs - but I would expect it to just return an empty result or an empty array.

if(Meteor.isClient){ var myJobs = JobCollection('myJobQueue'); Meteor.subscribe('allJobs'); var jobs = myJobs.find(); console.log('myJobs:' + JSON.stringify(jobs)); }

For now I have made an meteor method that returns the job collection from the server and then parse it to a reative variable so I can iterate on it using {{#each jobs}} - but it is definitely a hack that I would like to avoid in the future.

Happy holidays,

Lars Buur

— Reply to this email directly or view it on GitHub.

                  =

[LarsBuur]

Hi Kenny,

Thank you very much for your effort. I will try to create a completely fresh clean setup with only job-collection and post the project to github if I keep running into this problem.

Can you confirm that you are using the current version from atmospherejs.com which is: vsivsi:job-collection 0.0.18 ?

Best regards, Lars Buur

[kennyatbusheldirect]

Yep, that is exactly the version I am using!

-Kenny Date: Thu, 18 Dec 2014 06:09:53 -0800 From: notifications@github.com To: meteor-job-collection@noreply.github.com CC: kenny@busheldirect.com Subject: Re: [meteor-job-collection] Thoughts on management UI? (#14)

Hi Kenny,

Thank you very much for your effort. I will try to create a completely fresh clean setup with only job-collection and post the project to github if I keep running into this problem.

Can you confirm that you are using the current version from atmospherejs.com which is:

vsivsi:job-collection 0.0.18 ?

Best regards,

Lars Buur

— Reply to this email directly or view it on GitHub.

                  =

[mitar]

@LarsBuur: Is your interface open? If so, share the link to the code so that we can check the code and help. It is much easier to debug in that way.

[LarsBuur]

The example from the README.md works with the only exception that you need to declare

var myJobs = new JobCollection('myJobQueue');

on the client as @kennyatbusheldirect suggested.

I think it is strange that I need to define the JobCollection og both the server and the client but I am definitely willing to live with it :-) - I am unable to define the JobCollection in a lib folder and have it working....

If someone runs into this sometimes in the future then this is what is working for me - and steps.

Steps:

meteor create testJobCollection
cd testJobCollection
meteor remove autopublish
meteor remove insecure
meteor add vsivsi:job-collection

Edit testJobCollection.js and insert this content:

///////////////////
// Server
if (Meteor.isServer) {

    var myJobs = JobCollection('myJobQueue');
    myJobs.allow({
        // Grant full permission to anyone
        admin: function (userId, method, params) {
            return (userId ? true : true);
        }
    });

    Meteor.startup(function () {
        // Normal Meteor publish call, the server always
        // controls what each client can see
        Meteor.publish('allJobs', function () {

            var retVal = myJobs.find({});
            console.log(JSON.stringify(retVal.fetch()));

            return retVal;
        });

        // Create a job:
        var job = myJobs.createJob('sendEmail', // type of job
            // Job data that you define, including anything the job
            // needs to complete. May contain links to files, etc...
            {
                address: 'bozo@clowns.com',
                subject: 'Critical rainbow hair shortage',
                message: 'LOL; JK, KThxBye.'
            }
        );

        // Set some properties of the job and then submit it
        job.priority('normal')
            .retry({
                retries: 5,
                wait: 15 * 60 * 1000
            })  // 15 minutes between attempts
            .delay(60 * 60 * 1000)     // Wait an hour before first try
            .save();

        // Start the myJobs queue running
        return myJobs.startJobs();
    });

}

///////////////////
// Client
if (Meteor.isClient) {
    var myJobs = JobCollection('myJobQueue');
    Meteor.subscribe('allJobs');

    // lets just wait for the subscription to happen....
    setTimeout(function(){
        console.log(JSON.stringify(myJobs.find().fetch()));
    }, 1000);

}

Now both your server console and browser console should be filled with job json.

[mitar]

I think it is normal that in Meteor you have to define a Meteor collection both on a client and server? This is how it works.

[vsivsi]

Yes, collections need to be declared in both places if you want access on the client.. JobCollection is a collection internally, so goes by the same rules.

[LarsBuur]

The main point of my notes to this report got lost (maybe in translation).

What I found strange was that I need to create the JobCollection on both the server and the client - and that I am unable to share a JobCollection on both the server and client using a file in e.g. /collections which should be picked up by both the server and the client.

I am definitely willing to live with this - but it is still strange in my mind.

[vsivsi]

@LarsBuur Thanks for clarifying. Have you looked at the JobCollection sample app? It has only a single source file for both client and server and they share a single definition of the JobCollection: https://github.com/vsivsi/meteor-file-job-sample-app/blob/master/sample.coffee#L18

I've not encountered any problems with doing it this way. If you are seeing an issue, please provide some code in a sample app that replicates the issue and I'd be happy to take a look.

[vsivsi]

@LarsBuur Actually I see your steps to reproduce above. Sorry, I lost track of this issue. I'll take a look and see what I can find. Thanks.

[vsivsi]

So, when I try this using Meteor 1.0.3.1 and job-collection 0.0.18, I am able to partly reproduce what you are describing. It seems that the reason is three-fold:

1) The subscribe on the client should also be inside of a Meteor.startup() code block (at least that seems to help...)

2) The console.log in the publish function may not print the new job that is saved below it because there's a race condition there.

3) The 1000ms timeout on the setTimeout in the client code you provided is often inadequate. Upping that to 10000ms always resulted in the find returning a value on my machine. When you restart the server, the client may or may not reload (depending on the reason for the restart), and that will only run when it does. And even when it does, the timeout may expire before the local collection has been synchronized. So, the key seems to be actually waiting on the onReady callback of the subscribe, rather than picking an arbitrary timeout.

Another observation is that Meteor seems to be pretty lazy in syncing client subscriptions when there are no reactive cursors (or observers) registered to consume the records, as in this example.

So these changes to your repro code above make it work for me, so long as the client reloads. Also, note that a single JobCollection declaration works for both Server and Client.

var myJobs = new JobCollection('myJobQueue');

///////////////////
// Server

if (Meteor.isServer) {

    myJobs.allow({
        // Grant full permission to anyone
        admin: function (userId, method, params) {
            return (userId ? true : true);
        }
    });

    Meteor.startup(function () {
        // Normal Meteor publish call, the server always
        // controls what each client can see
        Meteor.publish('allJobs', function () {
            var retVal = myJobs.find({});
            return retVal;
        });

        // Create a job:
        var job = myJobs.createJob('sendEmail', // type of job
            // Job data that you define, including anything the job
            // needs to complete. May contain links to files, etc...
            {
                address: 'bozo@clowns.com',
                subject: 'Critical rainbow hair shortage',
                message: 'LOL; JK, KThxBye.'
            }
        );

        // Set some properties of the job and then submit it
        job.priority('normal')
            .retry({
                retries: 5,
                wait: 15 * 60 * 1000
            })  // 15 minutes between attempts
            .delay(60 * 60 * 1000)     // Wait an hour before first try
            .save();

        console.log(JSON.stringify(myJobs.find({}).fetch()));

        // Start the myJobs queue running
        return myJobs.startJobs();
    });
}

///////////////////
// Client
if (Meteor.isClient) {
    Meteor.startup(function () {
        Meteor.subscribe('allJobs', function () {
            console.log("Subscription Ready!");  // This only runs once per reload!
            console.log(JSON.stringify(myJobs.find().fetch()));  // Not reactive!
        });
    });
}

[vsivsi]

One other to note on this issue is that the JobCollection sample application has been fully updated in the past week or so to work with Meteor 1.0.x Sorry for letting it get out of date, Meteor 1.0 dropped just before the holidays and my long vacation and I didn't have time to update everything before I left.

[LarsBuur]

Thank you so much for clarifying this. You example definitely works and we have only one collection defined. Great!

[aldeed]

FYI, we've published a queue management UI package using aldeed:tabular here: https://atmospherejs.com/dispatch/tabular-job-collection

It is functional, a work in progress, but we're using it in production.

[vsivsi]

@aldeed Nice! Thanks for the heads-up!