Identify and document orphaned sidekiq jobs for ZSF initiative

rmtolmach commented 1 month ago

partially written by AI

User Story

As a member of the Platform team, I want to identify any orphaned Sidekiq jobs in the vets-api repo so that we can support the Zero Silent Failure (ZSF) initiative.

Issue Description

As a follow up to https://github.com/department-of-veterans-affairs/va.gov-team/issues/92876, we need to review the 100+ Sidekiq jobs in vets-api. The goal is to identify any jobs that are not currently owned by any team and document them. Once the orphaned jobs are identified, we will pass this information on to OCTO for further action (maybe the IIR can take action?). Before proceeding, we should run this task by Matt Dingee to make him aware of the effort and make sure someone else isn't already doing this.

Jobs can be found in vets-api by looking in lib/sidekiq, app/sidekiq by running the command find . -name "*_job.rb" to find the ones in modules.

Tasks

[x] Inform Matt Dingee that this work is being done.
[x] Identify all Sidekiq jobs
[x] Cross-check job ownership using git history and/or Atlas and identify any orphaned jobs.
[x] ~Document all orphaned Sidekiq jobs and prepare a handover to OCTO.~ was done by OCTO here.
[x] Provide the final report in either Confluence or GitHub (something OCTO can access).
[x] Create follow-up tickets as needed

Success Metrics

Matt Dingee is aware and has approved of the plan.
All orphaned Sidekiq jobs are identified and documented.
A clear handover is prepared for OCTO with the necessary information for orphaned jobs.

Acceptance Criteria

[x] Matt Dingee has been notified and has approved moving forward with the task.
[x] A list of orphaned Sidekiq jobs has been created.
[x] Documentation is available in Confluence or GitHub.
[x] Follow-up tickets created.

Validation

Assignee to add steps to this section. List the actions that need to be taken to confirm this issue is complete. Include any necessary links or context. State the expected outcome.

humancompanion-usds commented 1 month ago

@va-albers - Can you confirm that this is a task that is not being done by anyone else?

rmtolmach commented 1 month ago

I'm taking inventory of the sidekiq jobs in vets-api. I've organized them by how/where they're called.

Called only in PERIODIC JOBS	Called elsewhere in the app	Called in other Sidekiq job(s)	Don't appear to be called anywhere directly	Definitely not veteran-facing
AccountLoginStatisticsJob	Form526ConfirmationEmailJob	DeleteAttachmentJob	DirectDepositEmailJob	FeatureCleanerJob (Flipper)
BenefitsIntakeRemediationStatusJob	Form526SubmissionFailedEmailJob	CallbackUrlJob	CreateCh31SubmissionsReportJob	SidekiqStatsJob
BenefitsIntakeStatusJob	GIBillFeedbackSubmissionJob	NotificationsJob	PollForm526PdfStatus	DailyMaintenance (TUD)
DeleteOldPiiLogsJob	ProcessFileJob	SchedulerJob	DisabilityCompensationForm::Metrics	RotateKeysJob (KMS)
DeleteOldTransactionsJob	VANotifyDdEmailJob	BaseSubmissionJob	DisabilityCompensationForm::JobStatus	BatchInitiatorJob (KMS)
ExportBreakerStatus	VANotifyEmailJob	SubmitForm526	Create10203ApplicantDecisionLetters	8 files in CypressViewportUpdater
ExternalServicesStatusJob	Submit1900Job	DisabilityCompensationForm::Job	VA5495, VA5490, VA1995, VA1995, VA1990n, VA1990e, VA1990, VA0994, VA0993 (education forms)
Form526FailureStateSnapshotJob	SubmitDependentsPdfJob	Form4142DocumentUploadFailureEmail	NodEmailLoaderJob
Form526ParanoidSuccessPollingJob	SignUpServiceUpdaterJob	Form0781DocumentUploadFailureEmail	McpNotificationEmailJob
Form526StatusPollingJob	ProcessDataJob	Form526DocumentUploadFailureEmail	DeleteOldClaims (CentralMail)
Form526SubmissionProcessingReportJob	AuditLoginJob	EVSSDocument	BGS::Job
InProgressFormCleaner	AuditLogoutJob	VA10203	ApplicationJob (CovidResearch)
LoadAverageDaysForClaimCompletionJob	PhrUpdateJob		VHASubmissionJob
TransactionalEmailAnalyticsJob	SubmitCareerCounselingJob
DeleteExpiredSessionsJob	SubmitBenefitsIntakeClaim
CertificateCheckerJob	IncomeAndAssetsIntakeJob
DeleteValidationRecordsJob	DocumentUpload
ValidationJob	DocumentUploadSynchronous
DeleteOldUploads	CreateIntentToFileJob
CacheGlobalDowntime	SubmissionJob (HCA)
PollMaintenanceWindows	LogEmailDiffJob
Form526DocumentUploadPollingJob	EzrSubmissionJob
StdZipcodeImport	AnonSubmissionJob
StdStateImport	SubmissionJob (Form1010cg)
StdIncomeThresholdImport	UpdateClaimFromRemoteJob
StdCountyImport	RetrieveClaimsFromRemoteJob
GmtThresholdsImport	RequestDecision
UserAcceptableVerifiedCredentialTotalsJob	DocumentUpload (EVSS)
StdInstitutionImportJob	DependentsApplicationJob
New1095BsJob	UploadBddInstructions (DisabilityCompensationForm EVSS)
DeleteOldUploadsJob	SubmitUploads (DisabilityCompensationForm EVSS)
DeleteOldClaims (EVSS)	SubmitForm8940
Process10203Submissions	SubmitForm0781
DeleteOldApplications	SubmitForm526Cleanup
Create10203SpoolSubmissionsReport	SubmitForm526AllClaim
CreateSpoolSubmissionsReport	Form8940Document
CreateDailyFiscalYearToDateReport	SendSchoolCertifyingOfficialsEmail
SavedClaimScStatusUpdaterJob	EducationFacility
SavedClaimNodStatusUpdaterJob	CreateDailyYearToDateReport
SavedClaimHlrStatusUpdaterJob	CreateDailySpoolFiles
DeleteSavedClaimRecordsJob	SubmitUpload (DecisionReview)
ExpandedSubmissionStateJob	NodSendEmailJob
ScheduledBatchJob	Form4142Submit
ExpandedScheduledSubmissionJob	ParseNewStatementsJob
OptionsetDataJob	NewStatementNotificationJob
TopicsDataJob	SubmitForm4142Job
FindPoasJob	SubmitCentralForm686cJob
MissingFormStatusJob	SubmitForm686cJob
	SubmitForm674Job
	FlashUpdater (BGS)
	VANotifyEmailJob
	GenisisDeliveryJob
	ConfirmationMailerJob
	VBASubmissionJob
	VBSSubmissionJob
	SharepointSubmissionJob
	ExpandedSubmissionJob
	EnrollmentUploadJob
	ExpandedRegistrationEmailJob
	CovidVaccine::SubmissionJob
	RegistrationEmailJob
	PoaFormBuilderJob (v2)
	PoaFormBuilderJob (V1)
	PoaVBMSUploadJob
	EvidenceWaiverBuilderJob
	EmailJob (used A LOT)
	IcnJob
	UserAccountJob
	PreCacheClaimsAndAppealsJob
	Vet360LinkingJob
	PdfSubmitJob
	AppealReceivedJob
	StatusUpdatedJob
	PensionBenefitIntakeJob
	ExtendSessionJob
	Form526BackupSubmissionProcess

va-albers commented 1 month ago

@va-albers - Can you confirm that this is a task that is not being done by anyone else?

@humancompanion-usds I don't know if anyone else doing this. Bill Chapman did this previously though for Code Yellow 2 - he created a spreadsheet. Jeff Keene set up a project board for reviewing them as well.

rmtolmach commented 1 month ago

@va-albers - Can you confirm that this is a task that is not being done by anyone else?

@humancompanion-usds I don't know if anyone else doing this. Bill Chapman did this previously though for Code Yellow 2 - he created a spreadsheet. Jeff Keene set up a project board for reviewing them as well.

Thanks for the insight, Steve! I think I found the Board: https://github.com/orgs/department-of-veterans-affairs/projects/1166. Very nice!! I will try to identify any platform-owned and/or orphaned jobs so the Platform can move forward with any ZSF work that we need to do.

rmtolmach commented 1 month ago

Follow-up on some jobs that don't appear to be called anywhere directly. The board I reference is the one I linked in my previous comment. The sidekiq-exception-handling-review board.

DirectDepositEmailJob - deleted in https://github.com/department-of-veterans-affairs/vets-api/pull/18855
CreateCh31SubmissionsReportJob - commented on the sidekiq-exception-handling-review GitHub Projects board.
PollForm526PdfStatus - New-ish. Added a draft to the board for this job.
Create10203ApplicantDecisionLetters - commented on board.
NodEmailLoaderJob - New-ish. Added a draft to the board for this job.
McpNotificationEmailJob - recently deleted. moved to DONE on board.
DeleteOldClaims (CentralMail) - cron job deleted. Commented on board.
VHASubmissionJob - recently deleted. moved to DONE on board.

rmtolmach commented 1 month ago

Now I'm searching the board by "owner". I will take a look at the ones categorized as owned by Platform, Orphaned (owner-(team):"Orphaned?") and the ones without an owner. There are 12 Platform, 36 orphaned, and 13 with no owner.

Platform

There were some listed as being owned by Platform that weren't. And some that had been deleted. I provided evidence of that and then changed the team. Listed below are the ones that are still considered "Platform". The only two that might need more attention are in bold.

FeatureCleanerJob Not veteran-facing and not run. This job is supposed to delete features from the database that were removed from features.yaml, but sometimes it would delete ones that were still in features.yaml, causing problems, so this job was disabled in 2023.
PagerDuty::CacheGlobalDowntime - Not veteran-facing. Created in 2020 (in https://github.com/department-of-veterans-affairs/vets-api/pull/3954) to access the downtime schedule without invoking the maintenance_windows endpoint. This is needed when the API goes down for maintenance in order to display the proper messaging on the frontend. I assume it's still relevant. Logs are sent to Datadog, but we don't have any alerting set up for failures.
TestUserDashboard::DailyMaintenance - Not veteran-facing. There doesn't appear to be any error handling. The TUD is accessible to developers, but I'm not sure how often it's used.
InProgressFormCleaner - I don't think this is ours. It's not being actively maintained, but it's run once a day. edit: moved to orphaned.
AccountLoginStatisticsJob - Not veteran-facing. The original description for this job was to "Gather account login statistics for statsd". (initial commit)
PagerDuty::PollMaintenanceWindows https://github.com/department-of-veterans-affairs/va.gov-team/issues/89547 will cover this. There were silent failures. The impact was that maintenance windows would not be set for any service. While this isn't directly veteran facing, the trickle-down effects could be veteran-facing.
SidekiqStatsJob - Not veteran-facing. The job gathers metrics like processed, failed, and scheduled. It's also not called anywhere in the app. It could be called directly in the rails console by a developer. According to DD, it hasn't been called in any environment in the past 15 days. Personally, I don't think I've ever used this job.

No Owner

I either assigned a team, did not assign a team (but assured that it was not platform-owned). The two jobs below need our attention:

CypressViewportUpdater::UpdateCypressViewportsJob - Not veteran-facing. Added by Holden in https://github.com/department-of-veterans-affairs/vets-api/pull/5846 in 2021. It seems like it's owned by Platform. This job, "Updates Cypress files in vets-website with data from Google Analytics." Cypress is a JavaScript testing framework. I don't see any error handling, so there could be silent failures (but no veteran impact). I do see this in the dead queue with the error: OpenSSL::PKey::RSAError: Neither PUB key nor PRIV key. Added to Platform.
TransactionalEmailAnalyticsJob - This file and the spec are owned by backend-review-group in CODEOWNERS, so I've assigned this to Platform. PR where it was added (in 2019 https://github.com/department-of-veterans-affairs/vets-api/pull/3101) says it "Adds a mailer to generate a confirmation email to the end user upon successful updating of their PPIU Direct Deposit details." Seems veteran-facing.
I assigned teams to many other team-less jobs on the board.

Orphaned?

I ended up assigning these two to Platform. Neither are veteran facing.

ExternalServicesStatusJob - Not veteran-facing. 🤔 But probably should be owned by Platform.
ExportBreakerStatus - ditto ⬆

There are now only 21 items in the "Orphaned?" category (down from 36!). I was able to move some to done b/c they were deleted. And I was able to determine team assignment based on CODEOWNERS. The rest seem truly orphaned and/or definitely not Platform.

jwoodman5 commented 1 month ago

@rmtolmach is done with Platform owned and now focusing on orphaned jobs.

rmtolmach commented 1 month ago

Status: I went through all of the jobs on the board and edited my previous comment with updates. Any jobs Platform owns are labeled as such on the board (see here). Since they are "Drafts", I converted the Cyprus job to an issue (https://github.com/department-of-veterans-affairs/va.gov-team/issues/95353), and we already had an open issue for the PollMaintenance window job (#89547), and then I reached out to Jeff about TransactionalEmailAnalyticsJob here. The rest of the Platform-owned jobs are Low in priority so don't need to be addressed as part of this initiative.

The board was created in January, so I searched thorough the sidekiq history to see if anyone on the Platform had added a new job, but nope! No new Platform jobs.

rmtolmach commented 3 weeks ago

TransactionalEmailAnalyticsJob is not veteran facing. The only two jobs that have come out of this ticket are:

There are other jobs Platform owns that wouldn't be categorized as Silent Failures under the ZSF initiative because they're not veteran-facing and haven't ended up in the dead queue. Do we want to do an audit of these anyway?

rmtolmach commented 3 weeks ago

All items for this ticket are complete. I created this follow-up ticket for more work:

https://github.com/department-of-veterans-affairs/va.gov-team/issues/95476

department-of-veterans-affairs / va.gov-team