[Question] Scalability

[abnerjacobsen]

I am interested in using Cronicle in a solution that I am developing but I have the following requirement:

1) My app will use the Cronicle API to generate about 30,000 schedules per day within 2 hours time frame.

2) These schedules will run only once and their purpose will be to send an attendance confirmation email to a customer.

3) Schedules will have their execution start date and time distributed over a time period of 6 hours during the day.

4) After the schedule is successfully executed it can be deleted from the Cronicle.

Does the Cronicle support this workload?

Thank you.

[jhuckaby]

Hello there, and thank you for your interest in Cronicle. However, I am sorry to say, I think your needs are far beyond what it can realistically deliver, at least in its current pre-release state. I've only pushed it to about 2,000 jobs in a full day.

Cronicle does a lot of "bookkeeping" for each scheduled job run. Items are pushed to multiple lists, which require locking and serialized access. I would be very hesitant to launch more than 10 jobs per minute, and you need about 1000X times that amount.

It sounds like what you really need is an industrial strength message queue system, such as RabbitMQ or ZeroMQ.

I apologize for Cronicle. It is still in beta (v0.86), and really isn't designed to scale up to the levels you need. However, I will look into this for version 2.0.

Thanks!

[yisraelU]

Dont mean to crash someone's thread ...however, scalability is also a concern for me. Have you ever considered layering your api over an enterprise scheduler like Quartz,Dkron,Chronos etc..?

[jhuckaby]

Well, the products you mention, especially Chronos, are really super complete, full-featured scheduler applications that do virtually everything, and are designed for extreme scale. I don't see how my silly little app could add anything to them, because they already do it all.

To be honest, my Cronicle app shouldn't even exist, when there are free, open source products like Chronos already available. I only made Cronicle as an experiment, and to learn more about Node.js. It isn't meant to compete with the big boys. It just wants to play alone in its little kiddie pool :)

[yisraelU]

if this is your idea of a 'silly little app' , i can barely imagine how awesome it will be once its out of beta! I have worked with other schedulers and I do find usually , either no UI at all , or very limited ,thats why i am suggesting actually layering it on top of them.

[abnerjacobsen]

@jhuckaby Cronicle is a very interesting app, of course it can not compete with the big brothers yet, but even so in its current stage of development it can be very useful.

In my case, for example, I am looking for a solution for sending notifications that are triggered at a specific date and time, and Cronicle allows this in a simple and easy way, without much infrastructure configuration work and without a very complex solution.

Yesterday I ran a test with it generating a workload that consisted of sending 2000 notifications to a Slack channel in a 5 hour period. It worked perfectly, with no error, and only in some periods did the computer become overloaded. The test was done using this Docker image: intelliops/cronicle - using the file system as storage, I do not know if using CouchBase would improve performance.

One thing I noticed is that if I open Dasboard in the browser the CPU consumption increases, even without having any open scheduling.

I do not know if it is useful but here is the source of the Python script I used to generate the test:

import requests, json, cuid, random
from faker import Faker

fake = Faker()
url = "http://localhost:3012/api/app/create_event/v1"

headers = {
    'x-api-key': "116317e9c7b1de700a1e2d525f7c2621",
    'content-type': "application/json",
    'cache-control': "no-cache"
}

for number in range(2000):
    rowid = cuid.cuid()
    json_post = {
        'title': f'slack-{rowid}',
        'enabled': True,
        'category': 'general',
        'target': 'allgrp',
        'plugin': 'shellplug',
        'params': {
            'annotate': 0,
            'json': 0,
            'script': f"#!/bin/sh\n\n# Enter your shell script code here\n/opt/cronicle/bin/slacker -h \"https://hooks.slack.com/services/slack_key\" -c \"#slackchannel\" -u \"Confirmin\" -i \":heavy_check_mark:\" -t \"Dear {fake.name()}, you confirm the appointment with {fake.name()} today at {fake.time(pattern='%H:%M', end_datetime=None)} hours ?\" || exit 1"
        },
        'timing': {
            'years': [2018],
            'months': [7],
            'days': [5],
            'hours': [random.randint(13,17)],
            'minutes': [random.randint(0,59)]
        },
        'timezone': 'America/Sao_Paulo'
    }

    payload = json.dumps(json_post)

    response = requests.request("POST", url, data=payload, headers=headers)

    print(response.text)

/opt/cronicle/bin/slacker is a shell script I got from https://github.com/neilgupta/slacker

Some questions:

1) Does using the plugin for shell add a significant overhead to a plugin written in JS? 2) Does using CounchBase improve overall performance? 3) How much would a Single Master with Slaves configuration impact Master's CPU usage?

Anyway, congratulations for it, Cronicle is an application with a lot of potential, even without having all the features of Chronos, Dkron, etc, perhaps its implementation simplicity is its great differential.

[jhuckaby]

Thank you @abnerjacobsen and @yisraelU for your kind words. I am working hard to make Cronicle a great product -- I just never fathomed that anyone would ever want to schedule so many events at once 😝

@yisraelU I would be happy to discuss and consider some kind of integration with these other apps, if something made sense. I was under the impression that Chronos, Quartz et al already had a full-featured scheduling system and UI, and that Cronicle really didn't bring anything new to the table. Chronos supports full ISO 8601, for example, while Cronicle has a much simpler implementation, based on Unix cron. Could you elaborate on how you envisioned an integration with Chronos, Quartz or Dkron to work? I'm just having trouble visualizing how Cronicle could ever add anything to these "super apps".

@abnerjacobsen Thank you for your scale test and providing your test script source. Regarding Filesystem vs. Couchbase, I believe Couchbase could be much, much faster, as long as you configured the bucket to be 100% in memory, and ran it on a beefy cluster. That may help considerably with scale. However, a super fast SSD on Linux isn't too bad either.

Regarding scalability, here are my specific concerns about scaling Cronicle beyond 1,000 jobs per day, and where I think it may fall on its face:

• Total number of events in schedule at one time
  • The schedule system wasn't really designed for more than a few hundred entries total. It is more geared toward a small set of "repeating" events, like hourly / daily / weekly. Adding 30,000 one-time events is something I never imagined ever happening.
  • The schedule is kept entirely in RAM, but also synced to disk. So we may run into high memory usage issues, high CPU, as well as disk bashing.
  • The create_event API only allows you to add a single event at a time, and is quite wasteful on disk operations. It would be much better if I designed some sort of "bulk add" API, so you could add 1,000 events at a time. Possible v2.0 feature here.
• Home tab shows ALL upcoming jobs in next 24 hours
  • Currently the "Home" tab in the UI simulates 24 hours of jobs in the future, in order to present the "Upcoming Events" display. While this is simulated in a worker thread, I expect with 30,000 jobs per hour this would do something bad. Either eat CPU for a long time, or never display anything, or crash the browser.
  • What I can do here is "cap" the schedule simulation to the next 1,000 jobs max or some sane ceiling, then abort the loop before going into a tailspin. Possible v1.0 feature here.
• No pagination on Schedule tab in UI
  • The "Schedule" tab in the UI has no pagination, and instead lists every event. I just never imagined this growing large enough to warrant something like pagination. With 30,000+ events, this page would probably lag the browser terribly, crash it, or at the least burn up the user's CPU.
  • I could add pagination here, but many things in the UI require the entire schedule to be in browser memory, for example the Home screen "Upcoming Events" simulation, among others. So even with pagination, I think something would still get overloaded, with such a huge JSON object getting passed from server to client, and stored in browser memory.
  • Perhaps for v2.0 I can redesign the schedule to only send "chunks" of it over the wire, instead of the entire thing as one huge blob. However this will require redesign on a number of components.
• Total number of current active jobs at one time
  • This one is important, because the current "active" (running jobs) is designed to only scale to 100 at most. Not only is this data kept in memory (on server and client), but it's also refreshed (sent from server to client) every second, in its entirety. So you can imagine if this was thousands or tens of thousands of active jobs, things would definitely be quite unhappy.
  • Also, the UI "Home" tab shows all active jobs, without pagination, and every single one has an animated "progress bar" (CSS background gradient). These chew up CPU and GPU, and with more than 100 of these, I expect the "Home" tab would become unresponsive, crash the browser, or other.
  • All slave servers also send information about all their active jobs to the master server every second, so this adds up as well.
  • Not sure what I can do about this in the short term. For Cronicle v2.0, I can try to redesign this to only send chunks over the wire, and paginate the Home tab. However, I expect the total number of active jobs will always be a scalability breaking point, and it'll probably never scale up well, unless I do a ground-up redesign.
• Bookkeeping system (disk)
  • Cronicle has a rather extensive bookkeeping systems for jobs, where it writes to several linked lists on disk (or other Storage medium). This data powers the "Completed Events" page, the "Event History" page, the "Activity Log" (if the job failed) and also the job metadata and gzipped job log itself, not to mention expiration schedule metadata for the latter two.
  • This ends up being around 30 storage write operations per job (not to mention 40 reads), and many of these need to be locked and serialized, for maintaining the integrity of compound structures like linked lists and disk hashes. This is really tough to scale up to 30,000 jobs per hour, without a large redesign.
  • One of the decisions I made very early on was to not use any kind of database for Cronicle, and instead read and write my own data, but still allow for things like huge completed lists, activity logs and fast pagination. This is what led to the creation of my pixl-server-storage module, which is neat, I guess, but unfortunately it does NOT scale anywhere near what a real database could deliver. I made this tradeoff for simplicity vs. scalability.
• Expiration system (disk)
  • Cronicle has an "expiration" system for job metadata and job log files, where those storage records are deleted job_data_expire_days days after their creation. This requires writing metadata into two different disk structures, a linked list for each day, and a global disk-based hash for keeping track of all expiration dates (in case they change).
  • Unfortunately these operations add a number of extra disk operations for every single job, and are responsible for 20 of the disk writes (and 10 of the reads) per job. These also require global locks, so they cannot be parallelized.
  • I will work on ways to optimize this system. I think I can eliminate the use of the disk hash entirely, because Cronicle never changes its job expiration dates after the fact. That may help things a bit, but I am still worried about scaling this system to 30,000 jobs per hour.

In summary, I think there are some considerable scalability concerns here, to get up to the level you want, @abnerjacobsen. Some of them I can address in v1.0, others in v2.0, but I still think 30,000 per hour is a pretty insane goal to reach. It may be possible, but I forsee a great many redesigns here, possibly even a ground-up. This number was never even in my wildest dreams. I thought 1,000 jobs per day was a lot!

Here are answers to your 3 questions, @abnerjacobsen:

• Does using the plugin for shell add a significant overhead to a plugin written in JS?
  • The only overhead is one additional Node.js process, and an additional "shell" process (bash, sh, etc.). I didn't consider this to be much, but when you are talking about 30,000 jobs per hour, then VERY MUCH YES this may cause additional memory/CPU issues. I would recommend writing an actual Plugin in this case, to eliminate both the Node.js process and shell process.
• Does using CounchBase improve overall performance?
  • Oh, I talked about this above. Yes, I think it would indeed help quite a bit, as long as you configured the Couchbase bucket to store everything in RAM 100%, and also ran it on a beefy cluster.
• How much would a Single Master with Slaves configuration impact Master's CPU usage?
  • That's a good question. Obviously slave servers help with distributing the load of running jobs, but they also have to communicate with the master server. Slave servers maintain a constant WebSocket connection to the master, and send metadata about all their active jobs every second. So that's a consideration. Also, when jobs are complete, the slave server has to "upload" the job log file to the master. So that's additional disk and bandwidth for each job.
  • I think if your job processes themselves consume considerable CPU / memory, and didn't generate huge (multi-MB) log files, then running them on slaves would be beneficial. However, if they are usually fast processes that start/stop quickly, then running everything on a single beefy master server may be better overall.

Thanks again for your interest and ideas on improving Cronicle. I'll start thinking more about scalability, and add some TODO items for these low-hanging fruit items for v1.0.

[yisraelU]

@jhuckaby sure. My primary experience is with Quartz so ill talk to that end.

1. It has no UI to speak of. There are some commercial ($$$) products that created a UI for it.
2. It does not have a rest API so integration with other software requires building it yourself. 3.It has no persistent job history even with DB persistence enabled. 4.Overall it has no SDK like environment that you plug into i.e performance data, table return etc..

while I have a few other gripes, these are the glaring holes I do think they have a couple of ideas worth absorbing, i.e.Calendars to black out Holidays etc..

[jhuckaby]

@yisraelU Thank you for that information. I will do some research on Quartz, Chronos and others, and see what I can figure out. To be honest, I don't really know much about any of them.

I just released Cronicle Version 0.8.9, which has a few low-hanging fruit items plucked off, in regards to performance and scalability. We're still nowhere near the goal of 30,000 jobs per hour, but this should at least help reduce the number of storage operations per job, and also help the UI a bit in dealing with huge schedules.

[yisraelU]

cool , thanks ill check it out On a different note I know its pretty simple to write a plugin , however perhaps there would be benefit in having a plugins repo for community supplied pugins

[abnerjacobsen]

@jhuckaby I ran a test with the new version creating 2,000 schedules to run over a period of 1 hour.

The time it took to create the schedules using the API was 19 seconds.

The cpu usage was:

17:00:02 all 1,53 0,00 0,51 0,14 0,00 97,82 17:01:02 all 2,72 0,00 1,32 0,31 0,00 95,65 17:02:02 all 1,56 0,00 0,59 0,15 0,00 97,70 17:03:02 all 1,61 0,00 0,53 0,27 0,00 97,58 17:04:02 all 1,64 0,00 0,59 0,34 0,00 97,42 17:05:02 all 1,24 0,00 0,46 0,34 0,00 97,97 17:06:02 all 2,10 0,00 1,09 0,50 0,00 96,32 17:07:02 all 1,43 0,00 0,46 0,12 0,00 97,99 17:08:02 all 1,37 0,00 0,39 0,25 0,00 97,99 17:09:02 all 1,49 0,00 0,49 0,20 0,00 97,82 17:10:02 all 1,57 0,00 0,52 0,22 0,00 97,69 17:11:02 all 1,47 0,00 0,46 0,10 0,00 97,97 17:12:02 all 1,36 0,00 0,43 0,20 0,00 98,00 17:13:02 all 1,57 0,00 0,49 0,21 0,00 97,73 17:14:02 all 1,20 0,00 0,41 0,11 0,00 98,27 17:15:02 all 1,38 0,00 0,42 0,16 0,00 98,03 17:16:02 all 1,52 0,00 0,47 0,14 0,00 97,88 17:17:02 all 1,28 0,00 0,43 0,12 0,00 98,17 17:18:02 all 1,44 0,00 0,45 0,22 0,00 97,89 17:19:02 all 1,61 0,00 0,51 0,30 0,00 97,58 17:20:02 all 1,27 0,00 0,45 0,21 0,00 98,08 17:21:02 all 1,79 0,00 0,50 0,12 0,00 97,60 17:22:02 all 2,55 0,00 0,61 0,18 0,00 96,66 17:23:02 all 1,48 0,00 0,51 0,44 0,00 97,58 17:24:02 all 1,80 0,00 0,63 0,13 0,00 97,45 17:25:02 all 1,55 0,00 0,53 0,29 0,00 97,63 17:26:02 all 1,83 0,00 0,62 0,14 0,00 97,41 17:27:02 all 1,84 0,00 0,96 0,24 0,00 96,97 17:28:02 all 1,78 0,00 0,55 0,62 0,00 97,05 17:29:02 all 1,83 0,00 0,55 0,21 0,00 97,42 17:30:02 all 1,84 0,00 0,54 0,54 0,00 97,08 17:31:02 all 1,39 0,00 0,48 0,18 0,00 97,95 17:32:02 all 1,66 0,00 0,48 0,16 0,00 97,70 17:33:02 all 1,49 0,00 0,49 0,33 0,00 97,69 17:34:02 all 1,45 0,00 0,46 0,18 0,00 97,91 17:35:02 all 1,72 0,00 0,53 0,15 0,00 97,59 17:36:02 all 1,26 0,00 0,39 0,27 0,00 98,09 17:37:02 all 1,57 0,00 0,47 0,20 0,00 97,76 17:38:02 all 1,35 0,00 0,47 0,11 0,00 98,08 17:39:02 all 1,57 0,00 0,47 0,21 0,00 97,75 17:40:02 all 1,59 0,00 0,45 0,15 0,00 97,81 17:41:02 all 1,56 0,00 0,47 0,15 0,00 97,83 17:42:02 all 1,66 0,00 0,48 0,21 0,00 97,65 17:43:02 all 1,26 0,00 0,43 0,24 0,00 98,07 17:44:02 all 1,52 0,00 0,45 0,12 0,00 97,91 17:45:02 all 1,06 0,00 0,37 0,24 0,00 98,33 17:46:02 all 1,98 0,00 0,64 0,25 0,00 97,13 17:47:02 all 1,52 0,00 0,49 0,12 0,00 97,87 17:48:02 all 1,43 0,00 0,45 0,24 0,00 97,88 17:49:02 all 1,63 0,00 0,50 0,10 0,00 97,77 17:50:02 all 1,64 0,00 0,53 0,14 0,00 97,69 17:51:02 all 1,40 0,00 0,45 0,12 0,00 98,03 17:52:02 all 1,38 0,00 0,41 0,39 0,00 97,82 17:53:02 all 1,66 0,00 0,47 0,17 0,00 97,70 17:54:02 all 1,67 0,00 0,50 0,14 0,00 97,69 17:55:02 all 1,23 0,00 0,41 0,19 0,00 98,17 17:56:02 all 1,33 0,00 0,41 0,18 0,00 98,08 17:57:02 all 1,40 0,00 0,47 0,17 0,00 97,96 17:58:02 all 1,42 0,00 0,44 0,15 0,00 97,98 17:59:02 all 1,52 0,00 0,47 0,12 0,00 97,88 18:00:02 all 0,37 0,00 0,17 0,28 0,00 99,18

[jhuckaby]

@yisraelU A community plugin repository sounds like a great idea, if Cronicle ever becomes popular. Currently I think you are one of only 4 or 5 total users :)

@abnerjacobsen I'm not sure what your test output means, or how to read your CPU usage spreadsheet. 12 seconds to add 2,000 items seems.... slow? Not sure. However, I can't think of any other low-hanging fruit items to increase efficiency. I will come back around and try to address scalability for v2.0. I think several low-level systems need a complete redesign, to handle the extreme scale you need.

[guns2410]

Hey @jhuckaby. We have started using cronicle at work in production after trying it for a month and it has worked great for us... I think this product has a great potential and I am looking forward to contributing to this.