search

FNNDSC / pman

A process management system written in python
MIT License
22 stars 33 forks source link

Kubernetes memory limit #199

Closed jennydaman closed 2 years ago

jennydaman commented 2 years ago

Logs from pfcon:

[2022-03-04 17:31:29,568] [INFO][services:47 10 140453802506048] Sending RUN job request to pman at -->http://pman:5010/api/v1/<-- for job crispy-126
[2022-03-04 17:31:29,568] [INFO][services:49 10 140453802506048] Payload sent: {
    "cmd_args": "--saveinputmeta --saveoutputmeta --quiet",
    "cmd_path_flags": "",
    "auid": "jennings",
    "number_of_workers": 1,
    "cpu_limit": 1000,
    "memory_limit": 100,
    "gpu_limit": 0,
    "image": "fnndsc/ep-interpolate-surface-with-sphere:0.1.0",
    "selfexec": "isws",
    "selfpath": "/opt/conda/bin",
    "execshell": "/opt/conda/bin/python3.1",
    "type": "ds",
    "jid": "crispy-126"
}
[2022-03-04 17:31:30,620] [ERROR][services:60 10 140453802506048] Error response from pman service while submitting job crispy-126, detail: {"message": "(422)\nReason: Unprocessable Entity\nHTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '68c1055e-085f-4a7f-9938-4254064d5ce6', 'X-Kubernetes-Pf-Prioritylevel-Uid': '296737dd-595c-4f96-93b6-bb9c25ac5bcb', 'Date': 'Fri, 04 Mar 2022 17:31:30 GMT', 'Content-Length': '516'})\nHTTP response body: {\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"Job.batch \\\"crispy-126\\\" is invalid: spec.template.spec.containers[0].resources.requests: Invalid value: \\\"150Mi\\\": must be less than or equal to memory limit\",\"reason\":\"Invalid\",\"details\":{\"name\":\"crispy-126\",\"group\":\"batch\",\"kind\":\"Job\",\"causes\":[{\"reason\":\"FieldValueInvalid\",\"message\":\"Invalid value: \\\"150Mi\\\": must be less than or equal to memory limit\",\"field\":\"spec.template.spec.containers[0].resources.requests\"}]},\"code\":422}\n\n"}
jennydaman commented 2 years ago

https://github.com/FNNDSC/pman/commit/962a98d98081eb88bf52b1e3ac7d9314316787d8