Closed jjacob7734 closed 2 years ago
Added new shell wrapper to encapsulate running calrissian and dumping the docker usage statistics out to STDOUT so that it is recorded in the K8s job's logs: https://github.com/Duke-GCB/calrissian/compare/master...pymonger:handle-unsupported-file-ops?expand=1#diff-bf650190aac2a87b32f218ff937bae76c8a2362a022dc8086f9f94b8f63e4027
Screenshot from K8s dashboard:
and alternatively via kubectl:
[Tue May 10 14:07:15 gmanipon@MT-202634 calrissian]$ kc logs job/calrissian-job-h43neo2jeheg
Defaulted container "calrissian-job" out of: calrissian-job, init-volumes (init)
INFO calrissian 0.10.0 (cwltool 3.1.20220303001946)
https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl:8:3: Warning: checking
item
Warning: Field
`class` contains
undefined reference
to
`http://commonwl.org/cwltool#Secrets`
INFO https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl:8:3: Unknown hint
http://commonwl.org/cwltool#Secrets
https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/stage_out.cwl:8:3: Warning: checking
item
Warning: Field
`class` contains
undefined reference
to
`http://commonwl.org/cwltool#Secrets`
INFO https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/stage_out.cwl:8:3: Unknown hint
http://commonwl.org/cwltool#Secrets
DEBUG Parsed job order from command line: {
"id": "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl",
"workflow_aws_access_key_id": "(secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)",
"workflow_aws_secret_access_key": "(secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)",
"workflow_base_dataset_url": "s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades",
"workflow_input_url": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"workflow_max_spin_time": 30,
"workflow_min_spin_time": 15
}
DEBUG Starting ThreadPoolJobExecutor.run_jobs: total_resources=[ram: 1073.741824, cores: 1.0], max_workers=None
DEBUG [workflow ] initialized from https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl
DEBUG wait_for_completion with 0 futures
DEBUG [workflow ] inputs {
"workflow_aws_access_key_id": "(secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)",
"workflow_aws_secret_access_key": "(secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)",
"workflow_base_dataset_url": "s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades",
"workflow_input_url": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"workflow_max_spin_time": 30,
"workflow_min_spin_time": 15
}
DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat not ready
DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out not ready
INFO [workflow ] starting step stage_in
DEBUG [step stage_in] job input {
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_in/input_url": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF"
}
DEBUG [step stage_in] evaluated job input to {
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_in/input_url": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF"
}
INFO [step stage_in] start
DEBUG [job stage_in] initializing from https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/stage_in.cwl as part of step stage_in
DEBUG [job stage_in] {
"input_url": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF"
}
DEBUG [job stage_in] path mappings is {}
DEBUG [job stage_in] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "papermill"
},
{
"position": [
-1000000,
1
],
"datum": "/home/jovyan/downsample-landsat/stage_in.ipynb"
},
{
"position": [
-1000000,
2
],
"datum": "output_nb.ipynb"
},
{
"position": [
1,
"input_url"
],
"shellQuote": false,
"prefix": "--parameters",
"valueFrom": "input_url \"$(self)\"\n",
"datum": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF"
}
]
DEBUG wait_for_completion with 0 futures
DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat not ready
DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out not ready
DEBUG allocate [ram: 0, cores: 0] from available [ram: 1073.741824, cores: 1.0]
INFO [workflow ] start
DEBUG restore [ram: 0, cores: 0] to available [ram: 1073.741824, cores: 1.0]
DEBUG allocate [ram: 256, cores: 1] from available [ram: 1073.741824, cores: 1.0]
DEBUG [job stage_in] initial work dir {}
DEBUG wait_for_completion with 2 futures
DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat not ready
DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out not ready
DEBUG wait_for_completion with 1 futures
Building resources spec from {'cores': 1, 'ram': 256}
--------------------------------------------------------------------------------
apiVersion: v1
kind: Pod
metadata:
labels: {}
name: stage-in-pod-ksubablj
spec:
containers:
- args:
- /bin/sh -c 'papermill /home/jovyan/downsample-landsat/stage_in.ipynb output_nb.ipynb
--parameters input_url "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF"'
command:
- /bin/sh
- -c
env:
- name: HOME
value: /wXQPqe
- name: TMPDIR
value: /tmp
image: !!python/object/new:ruamel.yaml.scalarstring.SingleQuotedScalarString
- pymonger/downsample-landsat:2.0.3
imagePullPolicy: Always
name: stage-in-container
resources:
requests:
cpu: '1'
memory: 256Mi
volumeMounts:
- mountPath: /wXQPqe
name: tmpout-h43neo2jeheg
readOnly: false
subPath: tmpout-h43neo2jeheg/pu2mkflz
- mountPath: /tmp
name: tmpdir
workingDir: /wXQPqe
initContainers: []
restartPolicy: Never
securityContext:
runAsGroup: 0
runAsUser: 1001
volumes:
- name: tmpout-h43neo2jeheg
persistentVolumeClaim:
claimName: tmpout-h43neo2jeheg
readOnly: false
- name: output-data-h43neo2jeheg
persistentVolumeClaim:
claimName: output-data-h43neo2jeheg
readOnly: false
- emptyDir: {}
name: tmpdir
--------------------------------------------------------------------------------
Created k8s pod name stage-in-pod-ksubablj with id 6ecfb76e-33a8-4da9-a1d9-019b8b73ec74
PodMonitor adding stage-in-pod-ksubablj
k8s pod 'stage-in-pod-ksubablj' started
[stage-in-pod-ksubablj] follow_logs start
[stage-in-pod-ksubablj] Input Notebook: /home/jovyan/downsample-landsat/stage_in.ipynb
[stage-in-pod-ksubablj] Output Notebook: output_nb.ipynb
[stage-in-pod-ksubablj] Generating grammar tables from /srv/conda/envs/notebook/lib/python3.7/site-packages/blib2to3/Grammar.txt
[stage-in-pod-ksubablj] Writing grammar tables to /wXQPqe/.cache/black/22.1.0/Grammar3.7.12.final.0.pickle
[stage-in-pod-ksubablj] Writing failed: [Errno 2] No such file or directory: '/wXQPqe/.cache/black/22.1.0/tmp2naxidzt'
[stage-in-pod-ksubablj] Generating grammar tables from /srv/conda/envs/notebook/lib/python3.7/site-packages/blib2to3/PatternGrammar.txt
[stage-in-pod-ksubablj] Writing grammar tables to /wXQPqe/.cache/black/22.1.0/PatternGrammar3.7.12.final.0.pickle
[stage-in-pod-ksubablj] Writing failed: [Errno 2] No such file or directory: '/wXQPqe/.cache/black/22.1.0/tmpp1qmff2o'
Executing: 0%| | 0/10 [00:00<?, ?cell/s]Executing notebook with kernel: python3
Executing: 100%|██████████| 10/10 [00:03<00:00, 2.83cell/s]
[stage-in-pod-ksubablj] follow_logs end
Handling terminated pod name stage-in-pod-ksubablj with id 6ecfb76e-33a8-4da9-a1d9-019b8b73ec74
handling completion with 0
CALRISSIAN_DELETE_PODS=false(<class 'str'>)
returning False
DEBUG [step stage_in] produced output {
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_in/output_nb_file": {
"location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/output_nb.ipynb",
"basename": "output_nb.ipynb",
"nameroot": "output_nb",
"nameext": ".ipynb",
"class": "File",
"checksum": "sha1$3a517655d5bf8673ade15bd09c529b82566a798f",
"size": 7374,
"http://commonwl.org/cwltool#generation": 0
},
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_in/image_file": {
"location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL",
"nameext": ".TIF",
"class": "File",
"checksum": "sha1$8724b4a1707e93006cb6b5fc96171154bed1ff1c",
"size": 2121031,
"http://commonwl.org/cwltool#generation": 0
}
}
INFO [step stage_in] completed success
shutil.rmtree(/tmp/y1a8z_dw, True)
DEBUG restore [ram: 256, cores: 1] to available [ram: 817.741824, cores: 0.0]
INFO [workflow ] starting step downsample_landsat
DEBUG [step downsample_landsat] job input {
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/input_file": {
"location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL",
"nameext": ".TIF",
"class": "File",
"checksum": "sha1$8724b4a1707e93006cb6b5fc96171154bed1ff1c",
"size": 2121031,
"http://commonwl.org/cwltool#generation": 0
},
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/max_spin_time": 30,
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/min_spin_time": 15
}
DEBUG [step downsample_landsat] evaluated job input to {
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/input_file": {
"location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL",
"nameext": ".TIF",
"class": "File",
"checksum": "sha1$8724b4a1707e93006cb6b5fc96171154bed1ff1c",
"size": 2121031,
"http://commonwl.org/cwltool#generation": 0
},
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/max_spin_time": 30,
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/min_spin_time": 15
}
INFO [step downsample_landsat] start
DEBUG [job downsample_landsat] initializing from https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/downsample_landsat.cwl as part of step downsample_landsat
DEBUG [job downsample_landsat] {
"input_file": {
"location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL",
"nameext": ".TIF",
"class": "File",
"checksum": "sha1$8724b4a1707e93006cb6b5fc96171154bed1ff1c",
"size": 2121031,
"http://commonwl.org/cwltool#generation": 0
},
"max_spin_time": 30,
"min_spin_time": 15
}
DEBUG [job downsample_landsat] path mappings is {
"file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF": [
"/calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"/wXQPqe/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"File",
false
]
}
DEBUG [job downsample_landsat] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "papermill"
},
{
"position": [
-1000000,
1
],
"datum": "/home/jovyan/downsample-landsat/downsample_landsat.ipynb"
},
{
"position": [
-1000000,
2
],
"datum": "output_nb.ipynb"
},
{
"position": [
1,
"input_file"
],
"shellQuote": false,
"prefix": "--parameters",
"valueFrom": "input_file \"$(self.basename)\"\n",
"datum": {
"location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL",
"nameext": ".TIF",
"class": "File",
"checksum": "sha1$8724b4a1707e93006cb6b5fc96171154bed1ff1c",
"size": 2121031,
"http://commonwl.org/cwltool#generation": 0,
"path": "/wXQPqe/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"dirname": "/wXQPqe"
}
},
{
"position": [
2,
"min_spin_time"
],
"shellQuote": false,
"prefix": "--parameters",
"valueFrom": "min_spin_time \"$(self)\"\n",
"datum": 15
},
{
"position": [
3,
"max_spin_time"
],
"shellQuote": false,
"prefix": "--parameters",
"valueFrom": "max_spin_time \"$(self)\"\n",
"datum": 30
}
]
DEBUG wait_for_completion with 0 futures
DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out not ready
DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out not ready
DEBUG allocate [ram: 256, cores: 1] from available [ram: 1073.741824, cores: 1.0]
DEBUG wait_for_completion with 1 futures
DEBUG [job downsample_landsat] initial work dir {
"file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF": [
"/calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"/wXQPqe/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
"File",
true
]
}
Building resources spec from {'cores': 1, 'ram': 256}
--------------------------------------------------------------------------------
apiVersion: v1
kind: Pod
metadata:
labels: {}
name: downsample-landsat-pod-klceaaah
spec:
containers:
- args:
- /bin/sh -c 'papermill /home/jovyan/downsample-landsat/downsample_landsat.ipynb
output_nb.ipynb --parameters input_file "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF"
--parameters min_spin_time "15" --parameters max_spin_time "30"'
command:
- /bin/sh
- -c
env:
- name: HOME
value: /wXQPqe
- name: TMPDIR
value: /tmp
image: !!python/object/new:ruamel.yaml.scalarstring.SingleQuotedScalarString
- pymonger/downsample-landsat:2.0.3
imagePullPolicy: Always
name: downsample-landsat-container
resources:
requests:
cpu: '1'
memory: 256Mi
volumeMounts:
- mountPath: /wXQPqe
name: tmpout-h43neo2jeheg
readOnly: false
subPath: tmpout-h43neo2jeheg/wlolfpk7
- mountPath: /tmp
name: tmpdir
- mountPath: /wXQPqe/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF
name: tmpout-h43neo2jeheg
readOnly: true
subPath: tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF
workingDir: /wXQPqe
initContainers: []
restartPolicy: Never
securityContext:
runAsGroup: 0
runAsUser: 1001
volumes:
- name: tmpout-h43neo2jeheg
persistentVolumeClaim:
claimName: tmpout-h43neo2jeheg
readOnly: false
- name: output-data-h43neo2jeheg
persistentVolumeClaim:
claimName: output-data-h43neo2jeheg
readOnly: false
- emptyDir: {}
name: tmpdir
--------------------------------------------------------------------------------
Created k8s pod name downsample-landsat-pod-klceaaah with id 57dba616-a26d-41fe-a644-b5f455979419
PodMonitor adding downsample-landsat-pod-klceaaah
k8s pod 'downsample-landsat-pod-klceaaah' started
[downsample-landsat-pod-klceaaah] follow_logs start
[downsample-landsat-pod-klceaaah] Input Notebook: /home/jovyan/downsample-landsat/downsample_landsat.ipynb
[downsample-landsat-pod-klceaaah] Output Notebook: output_nb.ipynb
[downsample-landsat-pod-klceaaah] Generating grammar tables from /srv/conda/envs/notebook/lib/python3.7/site-packages/blib2to3/Grammar.txt
[downsample-landsat-pod-klceaaah] Writing grammar tables to /wXQPqe/.cache/black/22.1.0/Grammar3.7.12.final.0.pickle
[downsample-landsat-pod-klceaaah] Writing failed: [Errno 2] No such file or directory: '/wXQPqe/.cache/black/22.1.0/tmpii7o7abc'
[downsample-landsat-pod-klceaaah] Generating grammar tables from /srv/conda/envs/notebook/lib/python3.7/site-packages/blib2to3/PatternGrammar.txt
[downsample-landsat-pod-klceaaah] Writing grammar tables to /wXQPqe/.cache/black/22.1.0/PatternGrammar3.7.12.final.0.pickle
[downsample-landsat-pod-klceaaah] Writing failed: [Errno 2] No such file or directory: '/wXQPqe/.cache/black/22.1.0/tmpsd7eqmcb'
Executing: 0%| | 0/14 [00:00<?, ?cell/s]Executing notebook with kernel: python3
Executing: 100%|██████████| 14/14 [00:28<00:00, 2.01s/cell]
[downsample-landsat-pod-klceaaah] follow_logs end
Handling terminated pod name downsample-landsat-pod-klceaaah with id 57dba616-a26d-41fe-a644-b5f455979419
handling completion with 0
CALRISSIAN_DELETE_PODS=false(<class 'str'>)
returning False
DEBUG [step downsample_landsat] produced output {
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/output_nb_file": {
"location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/output_nb.ipynb",
"basename": "output_nb.ipynb",
"nameroot": "output_nb",
"nameext": ".ipynb",
"class": "File",
"checksum": "sha1$c9c7fb003df4c0704c85638dbddd876cc956182e",
"size": 12349,
"http://commonwl.org/cwltool#generation": 0
},
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/dataset_dir": {
"location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"nameext": "",
"class": "Directory"
}
}
INFO [step downsample_landsat] completed success
shutil.rmtree(/tmp/ax5s3b_e, True)
DEBUG restore [ram: 256, cores: 1] to available [ram: 817.741824, cores: 0.0]
INFO [workflow ] starting step stage_out
DEBUG [step stage_out] job input {
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/aws_access_key_id": "(secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)",
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/aws_secret_access_key": "(secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)",
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/base_dataset_url": "s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades",
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/dataset_dir": {
"location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"nameext": "",
"class": "Directory"
}
}
DEBUG [step stage_out] evaluated job input to {
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/aws_access_key_id": "(secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)",
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/aws_secret_access_key": "(secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)",
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/base_dataset_url": "s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades",
"https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/dataset_dir": {
"location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"nameext": "",
"class": "Directory"
}
}
INFO [step stage_out] start
DEBUG [job stage_out] initializing from https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/stage_out.cwl as part of step stage_out
DEBUG [job stage_out] {
"aws_access_key_id": "(secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)",
"aws_secret_access_key": "(secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)",
"base_dataset_url": "s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades",
"dataset_dir": {
"location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"nameext": "",
"class": "Directory"
}
}
DEBUG [job stage_out] path mappings is {
"file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled": [
"/calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"/var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"Directory",
true
]
}
DEBUG [job stage_out] command line bindings is [
{
"position": [
-1000000,
0
],
"datum": "sh"
},
{
"position": [
0,
0
],
"datum": "-c"
},
{
"position": [
0,
1
],
"valueFrom": "if [ ! -d $HOME/.aws ]; then cp -rp .aws $HOME/; fi && aws s3 cp --recursive $(inputs.dataset_dir.path) $(inputs.base_dataset_url)/$(inputs.dataset_dir.basename)"
}
]
DEBUG wait_for_completion with 0 futures
DEBUG allocate [ram: 256, cores: 1] from available [ram: 1073.741824, cores: 1.0]
DEBUG [job stage_out] initial work dir {
"_:c2799287-b7c5-4bb5-abe3-55cf6b3d70a6": [
"[default]\noutput = json\nregion = us-west-2\naws_access_key_id = (secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)\naws_secret_access_key = (secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)\n",
"/wXQPqe/.aws/credentials",
"CreateFile",
true
]
}
DEBUG wait_for_completion with 1 futures
Building resources spec from {'cores': 1, 'ram': 256}
--------------------------------------------------------------------------------
apiVersion: v1
kind: Pod
metadata:
labels: {}
name: stage-out-pod-ntekqdpz
spec:
containers:
- args:
- sh -c 'if [ ! -d $HOME/.aws ]; then cp -rp .aws $HOME/; fi && aws s3 cp --recursive
/var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled
s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled'
command:
- /bin/sh
- -c
env:
- name: HOME
value: /wXQPqe
- name: TMPDIR
value: /tmp
image: pymonger/aws-cli
imagePullPolicy: Always
name: stage-out-container
resources:
requests:
cpu: '1'
memory: 256Mi
volumeMounts:
- mountPath: /wXQPqe
name: tmpout-h43neo2jeheg
readOnly: false
subPath: tmpout-h43neo2jeheg/m4h0_4nu
- mountPath: /tmp
name: tmpdir
- mountPath: /var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled
name: tmpout-h43neo2jeheg
readOnly: true
subPath: tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled
workingDir: /wXQPqe
initContainers: []
restartPolicy: Never
securityContext:
runAsGroup: 0
runAsUser: 1001
volumes:
- name: tmpout-h43neo2jeheg
persistentVolumeClaim:
claimName: tmpout-h43neo2jeheg
readOnly: false
- name: output-data-h43neo2jeheg
persistentVolumeClaim:
claimName: output-data-h43neo2jeheg
readOnly: false
- emptyDir: {}
name: tmpdir
--------------------------------------------------------------------------------
Created k8s pod name stage-out-pod-ntekqdpz with id 197bc5a8-25d7-4ff6-8eb4-d6273a83416c
PodMonitor adding stage-out-pod-ntekqdpz
k8s pod 'stage-out-pod-ntekqdpz' started
[stage-out-pod-ntekqdpz] follow_logs start
upload: ../var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.met.json to s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.met.json
upload: ../var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.TIF to s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.TIF
upload: ../var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.browse.png to s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.browse.png
[stage-out-pod-ntekqdpz] follow_logs end
Handling terminated pod name stage-out-pod-ntekqdpz with id 197bc5a8-25d7-4ff6-8eb4-d6273a83416c
handling completion with 0
CALRISSIAN_DELETE_PODS=false(<class 'str'>)
returning False
DEBUG [step stage_out] produced output {}
INFO [step stage_out] completed success
INFO [workflow ] completed success
DEBUG [workflow ] outputs {
"final_dataset_dir": {
"location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"nameext": "",
"class": "Directory"
}
}
shutil.rmtree(/tmp/m92_fax1, True)
DEBUG restore [ram: 256, cores: 1] to available [ram: 817.741824, cores: 0.0]
DEBUG wait_for_completion with 0 futures
DEBUG wait_for_completion with 0 futures
DEBUG Finishing ThreadPoolExecutor.run_jobs: total_resources=[ram: 1073.741824, cores: 1.0], available_resources=[ram: 1073.741824, cores: 1.0]
INFO Final process status is success
{
"final_dataset_dir": {
"location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
"class": "Directory",
"listing": [
{
"class": "File",
"location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.met.json",
"basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.met.json",
"path": "/calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.met.json"
},
{
"class": "File",
"location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.browse.png",
"basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.browse.png",
"path": "/calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.browse.png"
},
{
"class": "File",
"location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.TIF",
"basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.TIF",
"path": "/calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.TIF"
}
],
"path": "/calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled"
}
}
CALRISSIAN_DELETE_PODS=false(<class 'str'>)
returning False
# BEGIN docker-usage.json
{
"cores_allowed": 1.0,
"ram_mb_allowed": 1073.741824,
"children": [
{
"cpus": 1.0,
"ram_megabytes": 268.435456,
"disk_megabytes": 2.128405,
"name": "stage_in",
"start_time": "2022-05-10T20:53:28+00:00",
"finish_time": "2022-05-10T20:53:32+00:00",
"elapsed_hours": 0.0011111111111111111,
"elapsed_seconds": 4.0,
"ram_megabyte_hours": 0.29826161777777777,
"cpu_hours": 0.0011111111111111111
},
{
"cpus": 1.0,
"ram_megabytes": 268.435456,
"disk_megabytes": 0.012349,
"name": "downsample_landsat",
"start_time": "2022-05-10T20:53:35+00:00",
"finish_time": "2022-05-10T20:54:04+00:00",
"elapsed_hours": 0.008055555555555555,
"elapsed_seconds": 29.0,
"ram_megabyte_hours": 2.1623967288888886,
"cpu_hours": 0.008055555555555555
},
{
"cpus": 1.0,
"ram_megabytes": 268.435456,
"disk_megabytes": 0.0,
"name": "stage_out",
"start_time": "2022-05-10T20:54:08+00:00",
"finish_time": "2022-05-10T20:54:09+00:00",
"elapsed_hours": 0.0002777777777777778,
"elapsed_seconds": 1.0,
"ram_megabyte_hours": 0.07456540444444444,
"cpu_hours": 0.0002777777777777778
}
],
"start_time": "2022-05-10T20:53:28+00:00",
"finish_time": "2022-05-10T20:54:09+00:00",
"elapsed_hours": 0.01138888888888889,
"elapsed_seconds": 41.0,
"total_cpu_hours": 0.009444444444444445,
"total_ram_megabyte_hours": 2.535223751111111,
"total_disk_megabytes": 2.140754,
"total_tasks": 3,
"max_parallel_cpus": 1.0,
"max_parallel_ram_megabytes": 268.435456,
"max_parallel_tasks": 1
}
# END docker-usage.json
Added new shell wrapper to encapsulate running calrissian and dumping the docker usage statistics out to STDOUT so that it is recorded in the K8s job's logs: https://github.com/Duke-GCB/calrissian/compare/master...pymonger:handle-unsupported-file-ops?expand=1#diff-bf650190aac2a87b32f218ff937bae76c8a2362a022dc8086f9f94b8f63e4027
Screenshot from K8s dashboard:
and alternatively via kubectl: