Add new "daily_0800" Jenkins job to run tasks.

[swirtSJW]

Description

We currently have a "daily" jenkins job that runs every day at midnight. That timing is correct for our forms migrations and needs to stay running then. However the timing is too soon for our facilities migrations so we need another task "daily_0800" that will run every day at 8:00AM ET.

Acceptance Criteria

• [x] daily_0800 jenkins job is created in unison with https://github.com/department-of-veterans-affairs/va.gov-cms/pull/9701 reaching production.
• [x] The job should run daily at 8:00AM.
• [x] The existing "daily" job timing should continue running as it does now, at midnight so that the upstream and downstream dependencies of the VA Forms migration are not interfered with.
  • Forms DB dumps CSV at 11:30
  • We migrate that at midnight
  • Forms API pulls out data at 1:00AM.

Tasks

• [x] Update CMS PR to change daily_two to daily_0800
  • https://github.com/department-of-veterans-affairs/va.gov-cms/pull/9701
• [x] Create devops PR to create new http://jenkins.vfs.va.gov/job/cms/job/cms-daily-0800 job
  • https://github.com/department-of-veterans-affairs/va.gov-cms/issues/9703
• [x] Run Seedjob to get job activated
• [x] Deploy tasks-periodic.yml to PROD manually
• [x] Test new Jenkins job
• [x] Get devops PR approved
• [x] Merge devops PR
• [x] Merge CMS PR

Slacks

• https://dsva.slack.com/archives/C03LFSPGV16/p1662050124637869
• https://dsva.slack.com/archives/C0FQSS30V/p1661522268787919
• https://dsva.slack.com/archives/C02HX4AQZ33/p1660923084200089
• https://dsva.slack.com/archives/C02HX4AQZ33/p1657641141719559

CMS Team

Please check the team(s) that will do this work.

• [ ] Program
• [x] Platform CMS Team
• [ ] Sitewide Crew
• [ ] ⭐️ Sitewide CMS
• [ ] ⭐️ Public Websites
• [x] ⭐️ Facilities
• [ ] ⭐️ User support

[ElijahLynn]

I'm assuming you have thought about this already, can you explain why we can't just change the timing of the daily job and use that one? Just so we have something documented.

[kevwalsh]

@ElijahLynn: @swirtSJW is out, but my sense is that there are other things that need to happen at midnight.

[cweagans]

related slack thread here: https://dsva.slack.com/archives/C02HX4AQZ33/p1657641141719559

[swirtSJW]

@elijah, we can not just change the timing of the daily job because we need the forms migrations to continue to run at the time they do now (midnight) There are both downstream and upstream dependencies that rely on that one running at midnight.

[swirtSJW]

More related threads here https://dsva.slack.com/archives/C02HX4AQZ33/p1660923084200089 and here https://dsva.slack.com/archives/C0FQSS30V/p1661522268787919

[ElijahLynn]

Steve and I just chatted on Zoom and we agreed to rename the task block from daily_two to daily_0800 to be a bit more specific about when it has to run so that it is self-documenting. I'll update https://github.com/department-of-veterans-affairs/va.gov-cms/pull/9701/files with the right code and make a PR for the Jenkins stuff and make sure it is all good so we don't miss a day.

[ElijahLynn]

K, task list is created and updated. And this is ready for review. Devops PR is created here https://github.com/department-of-veterans-affairs/devops/pull/11900 and I updated @swirtSJW's PR here with a rename of daily_two to daily_0800 as discussed. I'm testing a bit at http://jenkins.vfs.va.gov/job/cms/job/cms-daily-0800/ but can't remember/figure out how to have it checkout the specific branch in https://github.com/department-of-veterans-affairs/va.gov-cms/pull/9701. We may just have to merge it and test that way.

[ElijahLynn]

K, I just tested this by taking the file contents of #9701 and manually copying to PROD (via clipboaord, cut/paste) and running the job here http://jenkins.vfs.va.gov/job/cms/job/cms-daily-0800/2/console. The output is below and no changes were processessed. But, it ran successfully and I think that is good for merging.

task: [va/background/daily_0800/migrate/health_care_local_facility] drush $DRUSH_ALIAS migrate:reset-status va_node_health_care_local_facility [warning] Migration va_node_health_care_local_facility is already Idle task: [va/background/daily_0800/migrate/health_care_local_facility] drush $DRUSH_ALIAS migrate:import va_node_health_care_local_facility [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'va_node_health_care_local_facility' task: [va/background/daily_0800/migrate/vet_centers_facility] drush $DRUSH_ALIAS migrate:reset-status va_node_facility_vet_centers [warning] Migration va_node_facility_vet_centers is already Idle task: [va/background/daily_0800/migrate/vet_centers_facility] drush $DRUSH_ALIAS migrate:import va_node_facility_vet_centers [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'va_node_facility_vet_centers' task: [va/background/daily_0800/migrate/vet_centers_facility] drush $DRUSH_ALIAS migrate:reset-status va_node_facility_vet_centers_mvc [warning] Migration va_node_facility_vet_centers_mvc is already Idle task: [va/background/daily_0800/migrate/vet_centers_facility] drush $DRUSH_ALIAS migrate:import va_node_facility_vet_centers_mvc [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'va_node_facility_vet_centers_mvc' task: [va/background/daily_0800/migrate/vet_centers_facility] drush $DRUSH_ALIAS migrate:reset-status va_node_facility_vet_centers_os [warning] Migration va_node_facility_vet_centers_os is already Idle task: [va/background/daily_0800/migrate/vet_centers_facility] drush $DRUSH_ALIAS migrate:import va_node_facility_vet_centers_os [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'va_node_facility_vet_centers_os' task: [va/background/daily_0800/migrate/vba_facility] drush $DRUSH_ALIAS migrate:reset-status va_node_facility_vba [warning] Migration va_node_facility_vba is already Idle task: [va/background/daily_0800/migrate/vba_facility] drush $DRUSH_ALIAS migrate:import va_node_facility_vba [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'va_node_facility_vba' task: [va/background/daily_0800/migrate/nca_facility] drush $DRUSH_ALIAS migrate:reset-status va_node_facility_nca [warning] Migration va_node_facility_nca is already Idle task: [va/background/daily_0800/migrate/nca_facility] drush $DRUSH_ALIAS migrate:import va_node_facility_nca [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'va_node_facility_nca' task: [va/background/daily_0800/migrate/flag_removed_facilities] drush $DRUSH_ALIAS va_gov_migrate:flag-missing-facilities

[ElijahLynn]

Merged everything, tested after merge, we are all good now to close this out. The updated file is on PROD and will be until the next PROD deploy, which will replace it with the same file. The Jenkins job will continue to run fine until then, which it will do one more time before the PROD deploy.

[swirtSJW]

I checked on prod and have confirmed that the new timing ran perfectly. Nicely done Elijah.