Cron schedule runs 3-5 minutes late

[ghost]

cron schedule with specific date and time seems to run about 3-5 minutes late, and schedule for running every minute, seems to be running every 5 minutes. Is that normal? Thank you.

Configuration example

Schedule for specific datetime:

{
  "bistro_settings": {
    "resources": {
      "instance": {"concurrency": {"limit": 10, "default": 1}},
      "level1": {
        "my_resource": {"limit": 3, "default": 0}
      }
    },
    "nodes": {
      "levels": ["level1", "level2"],
      "node_sources": [
        {
          "source": "manual",
          "prefs": {
            "node1": []
          }
        },

        {
          "source": "add_time",
          "prefs": {
            "parent_level": "level1",
            "schedule": [
              {
                "cron": {
                  "year": 2017,
                  "month": 5,
                  "day_of_month": 17,
                  "hour": 8,
                  "minute": 50,
                  "dst_fixes": ["unskip", "repeat_use_only_early"]
                },
                "lifetime": 6000,
                "tags": ["tag_job1"]
              }
            ]
          }
        }

      ]
    },
    "enabled" : true
  },

  "bistro_job->job1" : {
    "owner" : "test",
    "enabled" : true,
    "command" : ["/bs/job_script.sh"],
    "priority": 1,
    "resources": {
      "my_resource": 1
    },
    "filters": {
      "level2": {
        "tag_whitelist": ["tag_job1"]
      }
    }
  }

}

Here is my configuration for the task running every minute:

        {
          "source": "add_time",
          "prefs": {
            "parent_level": "level1",
            "schedule": [
              {
                "cron": {
                  "minute": {"period": 1},
                  "dst_fixes": ["unskip", "repeat_use_only_early"]
                },
                "lifetime": 40
              }
            ]
          }
        }

Example Output:

Here is example timestamps showing approx. 5 minute gaps between job runs:

$ date -d +@1494852746
2017-05-15T08:52:26
$ date -d +@1494853041
2017-05-15T08:57:21
$ date -d +@1494853342
2017-05-15T09:02:22

[snarkmaster]

Could you share the scheduler logs?

Here is what I expect is happening:

• I bet your add_time nodes are getting made with the correct timestamps. Is that right?
• The actual runtimes of the tasks get delayed because: (i) the node refresh period is too low, (ii) the scheduler is heavily loaded or sleeping too long, or (iii) all the workers are busy.

In other words, the "cron" feature makes the task available to run at the promised time, but it may be a while before it can actually be executed.

We should be able to get this delay down to under 20 seconds, though, if you don't mind sharing some more details.

[snarkmaster]

Oh, actually, here is your entire problem -- :D

DEFINE_int32(nodes_update_ms, 300000, "How often to re-read the nodes");

Just pass --nodes_update_ms=5000 or something similar, et voilà!