search

StackStorm / st2

StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, incident responses, troubleshooting, deployments, and more for DevOps and SREs. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html
https://stackstorm.com/
Apache License 2.0
5.96k stars 744 forks source link

st2 cron timer was missed #5831

Open antonbezkrovny opened 1 year ago

antonbezkrovny commented 1 year ago

Have problem:

2022-12-02 09:00:01,696 140410779819872 WARNING base [-] Run time of job "St2Timer._emit_trigger_instance (trigger: cron[day_of_week='*', hour='9', minute='0', second='0'], next run at: 2022-12-03 09:00:00 MSK)" was missed by 0:00:01.696190

How can im configure APScheduler misfire_grace_time ?

antonbezkrovny commented 1 year ago

up

nzlosh commented 1 year ago

Can you provide the following information about your set up with the below command:

echo StackStorm Version; \
st2 --version; \
echo Distribution; \
egrep 'NAME|ID|VER' /etc/os-release; \
echo RabbitMQ; \
rabbitmqctl --version; \
echo MongoDB; \
mongo --version

What is the StackStorm execution environment. E.g. bare-metal, VM, container, kubernetes (on-premise / cloud)?

antonbezkrovny commented 1 year ago

Ola!

StackStorm Version st2 3.6.0, on Python 3.8.10 Distribution NAME="Ubuntu" VERSION="20.04.3 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.3 LTS" VERSION_ID="20.04" VERSION_CODENAME=focal UBUNTU_CODENAME=focal RabbitMQ 3.9.15 MongoDB MongoDB shell version v4.4.13 Build Info: { "version": "4.4.13", "gitVersion": "df25c71b8674a78e17468f48bcda5285decb9246", "openSSLVersion": "OpenSSL 1.1.1f 31 Mar 2020", "modules": [], "allocator": "tcmalloc", "environment": { "distmod": "ubuntu2004", "distarch": "x86_64", "target_arch": "x86_64" } }

lemeur commented 1 year ago

I've opened a similar discussion 11 days ago: https://github.com/StackStorm/st2/discussions/5824

the pb for me is that the VM Host may be overloaded sometimes, and thus misses clock ticks. The current implementation of the CrontTimer in stackstorm uses APScheduler, which has a misfire_grace_time parameter. But I can't see where to pass it to the Stackstorm rule.

jonathangull commented 1 year ago

HI Team I am also facing this issue . Here are my environment details

st2 3.7.0, on Python 3.6.9 Distribution NAME="Ubuntu" VERSION="18.04.6 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.6 LTS" VERSION_ID="18.04" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic MongoDB shell version v4.0.28 git version: af1a9dc12adcfa83cc19571cb3faba26eeddac92 OpenSSL version: OpenSSL 1.1.1 11 Sep 2018 allocator: tcmalloc modules: none build environment: distmod: ubuntu1804 distarch: x86_64 target_arch: x86_64

lemeur commented 1 year ago

@jonathangull, Are you running st2 in a virtual machine ?

If so, can you check on your hypervisor, if the host was busy at that time (CPU, I/O) ?

antonbezkrovny commented 1 year ago

up, problem appears bot if the host was busy or not

chain312 commented 1 year ago

@jonathangull, Are you running st2 in a virtual machine ?

If so, can you check on your hypervisor, if the host was busy at that time (CPU, I/O) ?

@lemeur You can look at my pr and add a parameter to the codehttps://github.com/StackStorm/st2/pull/5780