search

cookpad / kuroko2

Kuroko2 is a web-based job scheduler / workflow engine.
MIT License
313 stars 72 forks source link

Kuruko2 schedule job commands sometime runs twice #122

Open algiecaballesbbo opened 6 years ago

algiecaballesbbo commented 6 years ago

When running testing executing a simple command execute: node -v and scheduled it to run every minute, there are instance that the job runs twice. please refer to the attached images for more details

screen shot 2018-08-23 at 12 26 19 pm screen shot 2018-08-23 at 1 52 47 pm

Info: ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux] Rails 5.2.1 rvm 1.29.4 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io] node v8.4.0 RAILS_ENV=production NUM_OF_WORKERS=1

eagletmt commented 6 years ago

I never see such situation... Could you paste log/job-scheduler.log and log/production.log? Which RDB are you using for Kuroko2?

Kuroko2 currently does not support Rails 5.2 series, so you cannot use Kuroko2 with Rails 5.2.1. https://github.com/cookpad/kuroko2/blob/v0.5.0/kuroko2.gemspec#L20 You can check installed gems by running bundle show command.

algiecaballesbbo commented 6 years ago

For you question Kuroko2 currently does not support Rails 5.2 series, so you cannot use Kuroko2 with Rails 5.2.1.? , its currently working in our production using this version on rails. the only problem we have is it has duplicate process each commands when running scheduled jobs where we only use only 1 worker. Second question Which RDB are you using for Kuroko2?, we are using MariaDB v5.5.56 Our installed gems in the production:

Gems included by the bundle:
  * actioncable (5.2.1)
  * actionmailer (5.2.1)
  * actionpack (5.2.1)
  * actionview (5.2.1)
  * activejob (5.2.1)
  * activemodel (5.2.1)
  * activerecord (5.2.1)
  * activestorage (5.2.1)
  * activesupport (5.2.1)
  * addressable (2.5.2)
  * archive-zip (0.11.0)
  * arel (9.0.0)
  * aws-sdk (2.11.96)
  * aws-sdk-core (2.11.96)
  * aws-sdk-resources (2.11.96)
  * aws-sigv4 (1.0.3)
  * bindex (0.5.0)
  * bootsnap (1.3.1)
  * builder (3.2.3)
  * bundler (1.16.3)
  * byebug (10.0.2)
  * capybara (3.4.2)
  * childprocess (0.9.0)
  * chromedriver-helper (1.2.0)
  * chrono (0.4.0)
  * coffee-rails (4.2.2)
  * coffee-script (2.3.0)
  * coffee-script-source (1.12.2)
  * commonmarker (0.16.8)
  * concurrent-ruby (1.0.5)
  * crass (1.0.4)
  * dalli (2.7.8)
  * dotenv (0.11.1)
  * dotenv-deployment (0.0.2)
  * dotenv-rails (0.11.1)
  * erubi (1.7.1)
  * execjs (2.7.0)
  * faraday (0.12.2)
  * ffi (1.9.25)
  * font-awesome-rails (4.7.0.4)
  * foreman (0.85.0)
  * globalid (0.4.1)
  * haml (5.0.4)
  * hashie (3.5.7)
  * hipchat (1.3.0)
  * html-pipeline (2.8.4)
  * http_accept_language (2.1.1)
  * httparty (0.16.2)
  * i18n (1.1.0)
  * io-like (0.3.0)
  * jbuilder (2.7.0)
  * jmespath (1.4.0)
  * jquery-rails (4.3.3)
  * json (2.1.0)
  * jwt (1.5.6)
  * kaminari (1.1.1)
  * kaminari-actionview (1.1.1)
  * kaminari-activerecord (1.1.1)
  * kaminari-core (1.1.1)
  * kuroko2 (0.4.6)
  * listen (3.1.5)
  * loofah (2.2.2)
  * mail (2.7.0)
  * marcel (0.3.2)
  * method_source (0.9.0)
  * mimemagic (0.3.2)
  * mini_mime (1.0.1)
  * mini_portile2 (2.3.0)
  * minitest (5.11.3)
  * momentjs-rails (2.20.1)
  * msgpack (1.2.4)
  * multi_json (1.13.1)
  * multi_xml (0.6.0)
  * multipart-post (2.0.0)
  * mysql2 (0.5.2)
  * nio4r (2.3.1)
  * nokogiri (1.8.4)
  * oauth2 (1.4.0)
  * oj (3.6.5)
  * omniauth (1.8.1)
  * omniauth-google-oauth2 (0.2.10)
  * omniauth-oauth2 (1.3.1)
  * passenger (5.3.3)
  * public_suffix (3.0.2)
  * rack (2.0.5)
  * rack-accept-default (0.0.2)
  * rack-store (0.0.4)
  * rack-test (1.1.0)
  * rails (5.2.1)
  * rails-dom-testing (2.0.3)
  * rails-html-sanitizer (1.0.4)
  * rails_bootstrap_sortable (2.0.3)
  * railties (5.2.1)
  * rake (12.3.1)
  * rb-fsevent (0.10.3)
  * rb-inotify (0.9.10)
  * redcarpet (3.4.0)
  * redis (4.0.1)
  * responders (2.4.0)
  * retryable (3.0.1)
  * rinku (2.0.4)
  * ruby-enum (0.7.2)
  * ruby_dep (1.5.0)
  * rubyzip (1.2.1)
  * sass (3.4.25)
  * sass-rails (5.0.7)
  * select2-rails (4.0.3)
  * selenium-webdriver (3.13.1)
  * serverengine (1.5.11)
  * sigdump (0.2.4)
  * slim (3.0.9)
  * slim-rails (3.1.3)
  * spring (2.0.2)
  * spring-watcher-listen (2.0.1)
  * sprockets (3.7.2)
  * sprockets-rails (3.2.1)
  * temple (0.8.0)
  * the_garage (2.4.3)
  * thor (0.19.4)
  * thread_safe (0.3.6)
  * tilt (2.0.8)
  * tzinfo (1.2.5)
  * uglifier (2.7.2)
  * visjs-rails (4.21.0.0)
  * weak_parameters (0.5.0)
  * web-console (3.6.2)
  * websocket-driver (0.7.0)
  * websocket-extensions (0.1.3)
  * xpath (3.1.0)
riseshia commented 6 years ago

Could you paste log/job-scheduler.log and log/production.log?

Please check this one, too.

joefrey25 commented 6 years ago

Hi @riseshia san and @eagletmt san

This is lately the incident we have encountered in our production server. log.txt

duplicate
joefrey25 commented 6 years ago

hi @eagletmt san @riseshia san This is just a follow up question. How do you properly restart kuroko2 services daemon in the server? Do you have any proper commands that we can restart these services?

TIA

riseshia commented 6 years ago

I read pasted logs, It seems that more than one job-scheduler is running

D, [2018-08-31T04:00:26.906791 #3825] DEBUG -- :    (0.2ms)  BEGIN
D, [2018-08-31T04:00:26.907739 #3825] DEBUG -- :   Kuroko2::Tick Load (0.4ms)  SELECT  `kuroko2_ticks`.* FROM `kuroko2_ticks` ORDER BY `kuroko2_ticks`.`id` ASC LIMIT 1
D, [2018-08-31T04:00:26.908511 #3825] DEBUG -- :   Kuroko2::Tick Update (0.2ms)  UPDATE `kuroko2_ticks` SET `at` = '2018-08-30 19:00:26' WHERE `kuroko2_ticks`.`id` = 1
D, [2018-08-31T04:00:26.909129 #3825] DEBUG -- :   Kuroko2::JobSchedule Load (0.2ms)  SELECT  `kuroko2_job_schedules`.* FROM `kuroko2_job_schedules` ORDER BY `kuroko2_job_schedules`.`id` ASC LIMIT 1000
D, [2018-08-31T04:00:26.909960 #3825] DEBUG -- :   Kuroko2::JobDefinition Load (0.2ms)  SELECT `kuroko2_job_definitions`.* FROM `kuroko2_job_definitions` WHERE `kuroko2_job_definitions`.`id` IN (2, 5)
D, [2018-08-31T04:00:26.910801 #3825] DEBUG -- :   Kuroko2::JobSuspendSchedule Load (0.2ms)  SELECT `kuroko2_job_suspend_schedules`.* FROM `kuroko2_job_suspend_schedules` WHERE `kuroko2_job_suspend_schedules`.`job_definition_id` IN (2, 5)
D, [2018-08-31T04:00:26.922660 #3825] DEBUG -- :    (0.9ms)  COMMIT
D, [2018-08-31T04:00:26.934955 #3855] DEBUG -- :    (0.1ms)  BEGIN
D, [2018-08-31T04:00:26.935703 #3855] DEBUG -- :   Kuroko2::Tick Load (0.3ms)  SELECT  `kuroko2_ticks`.* FROM `kuroko2_ticks` ORDER BY `kuroko2_ticks`.`id` ASC LIMIT 1
D, [2018-08-31T04:00:26.936373 #3855] DEBUG -- :   Kuroko2::Tick Update (0.2ms)  UPDATE `kuroko2_ticks` SET `at` = '2018-08-30 19:00:26' WHERE `kuroko2_ticks`.`id` = 1
D, [2018-08-31T04:00:26.936966 #3855] DEBUG -- :   Kuroko2::JobSchedule Load (0.2ms)  SELECT  `kuroko2_job_schedules`.* FROM `kuroko2_job_schedules` ORDER BY `kuroko2_job_schedules`.`id` ASC LIMIT 1000
D, [2018-08-31T04:00:26.937774 #3855] DEBUG -- :   Kuroko2::JobDefinition Load (0.2ms)  SELECT `kuroko2_job_definitions`.* FROM `kuroko2_job_definitions` WHERE `kuroko2_job_definitions`.`id` IN (2, 5)
D, [2018-08-31T04:00:26.938588 #3855] DEBUG -- :   Kuroko2::JobSuspendSchedule Load (0.2ms)  SELECT `kuroko2_job_suspend_schedules`.* FROM `kuroko2_job_suspend_schedules` WHERE `kuroko2_job_suspend_schedules`.`job_definition_id` IN (2, 5)
D, [2018-08-31T04:00:26.947863 #3855] DEBUG -- :    (0.7ms)  COMMIT

Base on your comment (https://github.com/cookpad/kuroko2/issues/122#issuecomment-418591794), It may fail to restart job scheduler, but just launch one more job scheduler.

we using systemd to manage these processes, except executor which need to be restarted when it's not working. Please check example service unit files: https://github.com/cookpad/kuroko2/blob/master/docs/admin_guide.md#daemons

joefrey25 commented 6 years ago

@riseshia san

Thanks for the response,

Base on your comment (#122 (comment)), It may fail to restart job scheduler, but just launch one more job scheduler.

Yes we just launch 1 job in scheduler and it will only run everyday at 3am

we using systemd to manage these processes, except executor which need to be restarted when it's not working. Please check example service unit files: https://github.com/cookpad/kuroko2/blob/master/docs/admin_guide.md#daemons

Yes we used this systemd file in our server and we also do have a cronjob to restart the services

BTW. may we know the versions of rails, ruby and web server(what you used) you guys running for this? Thanks!

joefrey25 commented 6 years ago

Hi @riseshia san

I just want to confirm this if this is another issue in kuroko2. I'm currently downgraded my rails version. current version of rails 5.1.6 current version of ruby 2.5.1

new_job_definition_ _kuroko_2

As I can see there's a new table kuroko2_script_revisions and there's no documentation on how to do this script revisions.

Sorry for too much question.

TIA