gitlab: verify behaviour when user exceeds quota

[tiborsimko]

When a user exceeds disk quota, the CLI way of submitting workflows will prevent user from creating new workflow runs.

The behaviour was verified to work well. However, some improvements were proposed in https://github.com/reanahub/reana-client/issues/607.

This goal of this issue is to do the same verification for GitLab <-> REANA way of submitting workflows:

• will the workflow be prevented from starting?
• will the error message back to GitLab web informative enough?
• will the user sees the core of the issue on the GitLab side?
• will the user know what to do next when this happens?

[VMois]

Preliminary answering questions after reading the code (as testing is complicated)

will the workflow be prevented from starting?

Yes, it will be prevented.

will the error message back to GitLab web be informative enough? will the user sees the core of the issue on the GitLab side?

Not sure what will happen here, definitely some sort of error. How GitLab will record and display the error is also a question.

This is where we respond with success or failure to GitLab when a workflow is finished/failed - https://github.com/reanahub/reana-workflow-controller/blob/f35e70f98c56ebca757131cdd64e51c3ccf0cdf6/reana_workflow_controller/consumer.py#L173. But I am curious what will happen if 4xx error with our custom message (REANAQuotaExceed) will be returned to GitLab webhook in reana-server, way before workflows is even created. Needs testing.

will the user know what to do next when this happens?

It depends on what you mean by "what to do next". A clear message like we will have in https://github.com/reanahub/reana-server/pull/496 should be enough, in my view, to direct users to check their quota usage.

[VMois]

I have triggered and checked how Gitlab will treat our error. It will be silenced and only appear deep in "Webhook" settings. Nothing will appear on a build status page.

We can fix this by applying similar code we have in r-w-controller/consumer.py for Gitlab in r-server to properly fail build status and pass a detailed description of the error - https://docs.gitlab.com/ee/api/commits.html#post-the-build-status-to-a-commit

[VMois]

There are two issues to solve:

1. Currently, the code that sends results back to GitLab is in r-w-controller/consumer.py, we need it in r-server. So we need to answer the question - duplicate code or move to r-commons (again...)?

2. Quota error is thrown even before we execute endpoint code:

https://github.com/reanahub/reana-server/blob/38fdf2c039c42ea39c5534460b0c65a3ffe2d3f3/reana_server/rest/workflows.py#L277-L278

It means we will need to remove the decorator here and check for quota error at the beginning of a create_workflow function.

[VMois]

This issue raises another problem. What will happen if on Gitlab hook workflow fails for other reasons. This is something to discuss in a separate issue, I guess. I can try to fail Gitlab with incorrect reana.yaml and see what will happen.