epam / cloud-pipeline

Cloud agnostic genomics analysis, scientific computation and storage platform
https://cloud-pipeline.com
Apache License 2.0
144 stars 58 forks source link

Introduce hard/soft limits for commiting run and run tool/pipeline #3573

Open SilinPavel opened 6 days ago

SilinPavel commented 6 days ago

Background Docker images can be extremely big, so it is nice to have a mechanism to warn/reject user from creation or running such big images.

Approach

  1. Let's introduce new System preference - commit.tool.size.limits (Object) with structure:

    {
      "commitToolSoftLimit": <size value in bytes, long>,
      "commitToolHardLimit": <size value in bytes, long>,
      "runToolSoftLimit": <size value in bytes, long>,
      "runToolHardLimit": <size value in bytes, long>
    }

    Provide validation for the values of the properties`

  2. Modify a method in PipelineRunController GET /run/{runId}/commit/check to provide more extended result like:

    {
        "enoughSpace": {
           "status": <ENUM (FAIL, WARN, OK)>,
           "message": <can be null, message in case of warning or fail>,
           "value": <raw value of the metric, for enoughSpace true/false>
        },
        "containerSize":  {
            "status": <ENUM (FAIL, WARN, OK)>,
            "message": <can be null, message in case of warning or fail>,
            "value": <raw value of the metric, for containerSize size in bytes>
        }
    }

    To do so, let's introduce new server method in DockerContainerOperationManager to be able to ask node which is host the run, current size of the run container.

    • This method should work similar as commit procedure (ssh to the node) and utilize command docker inspect ${CONTAINER_ID} -s --format "{{.SizeRw}}"
    • Timeout should be implemented for the method to not to hang much time
    • If something goes wrong (timeout or error) return -1 as containerSize
  3. On UI:

    • During commit change logic to verify not only enoughSpace but also check that containerSize < commitToolSoftLimit/commitToolHardLimit
      • If bigger:
      • For commitToolSoftLimit : Warn user that container image will be too big, but let to commit if needed
      • For commitToolHardLimit : Restrict to commit
    • During Tool/Pipeline run, check if tool version size < runToolSoftLimit/runToolHardLimit
      • If bigger:
      • For runToolSoftLimit : Warn user that container image is too big, but let to run if needed
      • For runToolHardLimit : Restrict to run