Migrate cronjobs to Argo CronWorkflows

[pacospace]

Is your feature request related to a problem? Please describe.

• [ ] deploy workflow-controller to namespaces hosting cronworkflows

Migrate cronjob in Thoth to CronWorkflow:

• [ ] increase quota on thoth-infra
• [ ] move all the -job to thoth-infra namespace
• [ ] deploy Argo's workflow-controller to thoth-infra
• [ ] graph-update-job
• [ ] graph-refresh-job
• [ ] package-release-job
• [ ] cve-update-job
• [ ] graph-backup
• [ ] slo-reporter
• [ ] advise-reporter

Describe the solution you'd like Use Argo Cronworkflow for flexibility, monitoring and extensibility: https://github.com/thoth-station/thoth-application/issues/429#issuecomment-727820282

Describe alternatives you've considered

Additional context

[goern]

/triage accepted

[harshad16]

constraint: this would require workflow-controller to be deployed in other namespaces as well.

[goern]

/milestone 2021.07.19

[goern]

/milestone 2021.08.02

[sesheta]

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

/close

[sesheta]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[sesheta]

@sesheta: Closing this issue.

In response to [this](https://github.com/thoth-station/thoth-application/issues/654#issuecomment-905047287): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[VannTen]

constraint: this would require workflow-controller to be deployed in other namespaces as well.

Is a cluster-scoped controller not possible as an alternative ?

[harshad16]

As we work on shared cluster, cluster-scoped controller are most of the time hard to get deployed.

[VannTen]

As we work on shared cluster, cluster-scoped controller are most of the time hard to get deployed.

Could operate-first offer that as a cluster-level service (aka, this would become part of the operate-first platform offer) ?

I don't mean technically, but would that work with the operate-first organization ?

[harshad16]

Yes correct, operate-first platform can provide these cluster-scoped resources.

[Gregory-Pereira]

/cc @Gregory-Pereira

[VannTen]

So to resume the discussion yesterday, although operate-first could spin up an argo-workflow controller as a cluster service, we deploy to other environments (I didn't know that ^) where it would be more complicated.

[Gregory-Pereira]

Yes, I guess I should follow your example to document the result of our discussion. The issue prompting this discussion in the first place is that by migrating our Cronjobs to Argo Workflows, we would want the components to run in their correct namespaces. As such there we would need access to the Argo workflows controller in those namespaces. It is possible to deploy a single cluster scoped instance in operate first, which would eliminate the need to redeploy the controller in every namespace but this comes with its own issues, mainly that this is not reproduce-able in other non-operate-first environments, particularly our staging OCP4 environment. This would in turn decouple our deployment for our staging and production clusters which honestly seems like more trouble than its worth. As a result, it seems like a smarter idea to simply deploy an instance of the workflow controller in the namespace that dont currently have it (this would be a good first step). Let me know if I misunderstood something, or am missing anything.