client: allow users to manually specify resources for their workflows

[diegodelemos]

In https://github.com/reanahub/reana/issues/490 we introduced cluster-wide memory limits.

The goal of this task would be to make this memory limits overridable by users, use case:

• As a user, I want to set a custom memory limit to my workflow which needs 10 GiB so that it doesn't get OOMKilled by the default 8 GiB limit.

Expected behavior:

• User specifies a new option in reana.yaml to set a memory limit for jobs. This would be done similarly as HTCondor limits.
• The limit should be respected for the user jobs (here you can find instructions on how to see whether limits are set in jobs) in both following cases:
  • If the cluster has already a default memory limit, the user jobs should override it with this config
  • Even if the cluster has no limits, the user job should still have the specified limit

TODO:

• [x] Validation
  • [x] Format
  • [x] Support fixed-point number values for k8s memory limits, e.g. 1.5Gi
  • [x] Max memory limits
    • [x] Configured max memory limit. Admin knows how big are cluster nodes (MVP)
    • [x] Helm values.yaml
    • [ ] ❓Smart discovery (no MVP) => ticket for later?
• [x] reana-client schema validation
• [x] Documentation (docs.reana.io) in Kubernetes section. Get inspiration from HTCondor. Reference https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-memory for possible kubernetes_memory_size values.

[lukasheinrich]

ideally this would be something that the engine understands on a per-jobs basis, not only per-workfllow

I.e. the engine parses the wflow description and submits for each job a suitable job definition. If the engine cannot to job-lelvel granularity it could do a wflow wide setting, but if it knows some jobs are lighter than others that would be useful to exploit

[mvidalgarcia]

ideally this would be something that the engine understands on a per-jobs basis, not only per-workfllow

I.e. the engine parses the wflow description and submits for each job a suitable job definition. If the engine cannot to job-lelvel granularity it could do a wflow wide setting, but if it knows some jobs are lighter than others that would be useful to exploit

Yep, that's exactly the direction we're taking. This issue aims at allowing the user to set something like e.g. kubernetes_memory_limit: 8Gi per job.