R1: Collect metrics on ADES/K8S

Added new shell wrapper to encapsulate running calrissian and dumping the docker usage statistics out to STDOUT so that it is recorded in the K8s job's logs: https://github.com/Duke-GCB/calrissian/compare/master...pymonger:handle-unsupported-file-ops?expand=1#diff-bf650190aac2a87b32f218ff937bae76c8a2362a022dc8086f9f94b8f63e4027

Screenshot from K8s dashboard: Screen Shot 2022-05-10 at 2.06.33 PM.png

and alternatively via kubectl:

[Tue May 10 14:07:15 gmanipon@MT-202634 calrissian]$ kc logs job/calrissian-job-h43neo2jeheg
Defaulted container "calrissian-job" out of: calrissian-job, init-volumes (init)
INFO calrissian 0.10.0 (cwltool 3.1.20220303001946)
https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl:8:3: Warning: checking
                                                                                      item
                                                                                      Warning:   Field
                                                                                      `class` contains
                                                                                      undefined reference
                                                                                      to
                                                                                      `http://commonwl.org/cwltool#Secrets`
INFO https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl:8:3: Unknown hint
                                                                                      http://commonwl.org/cwltool#Secrets
https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/stage_out.cwl:8:3: Warning: checking
                                                                                       item
                                                                                       Warning:   Field
                                                                                       `class` contains
                                                                                       undefined reference
                                                                                       to
                                                                                       `http://commonwl.org/cwltool#Secrets`
INFO https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/stage_out.cwl:8:3: Unknown hint
                                                                                       http://commonwl.org/cwltool#Secrets
DEBUG Parsed job order from command line: {
    "id": "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl",
    "workflow_aws_access_key_id": "(secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)",
    "workflow_aws_secret_access_key": "(secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)",
    "workflow_base_dataset_url": "s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades",
    "workflow_input_url": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
    "workflow_max_spin_time": 30,
    "workflow_min_spin_time": 15
}
DEBUG Starting ThreadPoolJobExecutor.run_jobs: total_resources=[ram: 1073.741824, cores: 1.0], max_workers=None
DEBUG [workflow ] initialized from https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl
DEBUG wait_for_completion with 0 futures
DEBUG [workflow ] inputs {
    "workflow_aws_access_key_id": "(secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)",
    "workflow_aws_secret_access_key": "(secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)",
    "workflow_base_dataset_url": "s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades",
    "workflow_input_url": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
    "workflow_max_spin_time": 30,
    "workflow_min_spin_time": 15
}
DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat not ready
DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out not ready
INFO [workflow ] starting step stage_in
DEBUG [step stage_in] job input {
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_in/input_url": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF"
}
DEBUG [step stage_in] evaluated job input to {
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_in/input_url": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF"
}
INFO [step stage_in] start
DEBUG [job stage_in] initializing from https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/stage_in.cwl as part of step stage_in
DEBUG [job stage_in] {
    "input_url": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF"
}
DEBUG [job stage_in] path mappings is {}
DEBUG [job stage_in] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "papermill"
    },
    {
        "position": [
            -1000000,
            1
        ],
        "datum": "/home/jovyan/downsample-landsat/stage_in.ipynb"
    },
    {
        "position": [
            -1000000,
            2
        ],
        "datum": "output_nb.ipynb"
    },
    {
        "position": [
            1,
            "input_url"
        ],
        "shellQuote": false,
        "prefix": "--parameters",
        "valueFrom": "input_url \"$(self)\"\n",
        "datum": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF"
    }
]
DEBUG wait_for_completion with 0 futures
DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat not ready
DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out not ready
DEBUG allocate [ram: 0, cores: 0] from available [ram: 1073.741824, cores: 1.0]
INFO [workflow ] start
DEBUG restore [ram: 0, cores: 0] to available [ram: 1073.741824, cores: 1.0]
DEBUG allocate [ram: 256, cores: 1] from available [ram: 1073.741824, cores: 1.0]
DEBUG [job stage_in] initial work dir {}
DEBUG wait_for_completion with 2 futures
DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat not ready
DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out not ready
DEBUG wait_for_completion with 1 futures
Building resources spec from {'cores': 1, 'ram': 256}
--------------------------------------------------------------------------------
apiVersion: v1
kind: Pod
metadata:
  labels: {}
  name: stage-in-pod-ksubablj
spec:
  containers:
  - args:
    - /bin/sh -c 'papermill /home/jovyan/downsample-landsat/stage_in.ipynb output_nb.ipynb
      --parameters input_url "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF"'
    command:
    - /bin/sh
    - -c
    env:
    - name: HOME
      value: /wXQPqe
    - name: TMPDIR
      value: /tmp
    image: !!python/object/new:ruamel.yaml.scalarstring.SingleQuotedScalarString
    - pymonger/downsample-landsat:2.0.3
    imagePullPolicy: Always
    name: stage-in-container
    resources:
      requests:
        cpu: '1'
        memory: 256Mi
    volumeMounts:
    - mountPath: /wXQPqe
      name: tmpout-h43neo2jeheg
      readOnly: false
      subPath: tmpout-h43neo2jeheg/pu2mkflz
    - mountPath: /tmp
      name: tmpdir
    workingDir: /wXQPqe
  initContainers: []
  restartPolicy: Never
  securityContext:
    runAsGroup: 0
    runAsUser: 1001
  volumes:
  - name: tmpout-h43neo2jeheg
    persistentVolumeClaim:
      claimName: tmpout-h43neo2jeheg
      readOnly: false
  - name: output-data-h43neo2jeheg
    persistentVolumeClaim:
      claimName: output-data-h43neo2jeheg
      readOnly: false
  - emptyDir: {}
    name: tmpdir
--------------------------------------------------------------------------------

Created k8s pod name stage-in-pod-ksubablj with id 6ecfb76e-33a8-4da9-a1d9-019b8b73ec74
PodMonitor adding stage-in-pod-ksubablj
k8s pod 'stage-in-pod-ksubablj' started
[stage-in-pod-ksubablj] follow_logs start
[stage-in-pod-ksubablj] Input Notebook:  /home/jovyan/downsample-landsat/stage_in.ipynb
[stage-in-pod-ksubablj] Output Notebook: output_nb.ipynb
[stage-in-pod-ksubablj] Generating grammar tables from /srv/conda/envs/notebook/lib/python3.7/site-packages/blib2to3/Grammar.txt
[stage-in-pod-ksubablj] Writing grammar tables to /wXQPqe/.cache/black/22.1.0/Grammar3.7.12.final.0.pickle
[stage-in-pod-ksubablj] Writing failed: [Errno 2] No such file or directory: '/wXQPqe/.cache/black/22.1.0/tmp2naxidzt'
[stage-in-pod-ksubablj] Generating grammar tables from /srv/conda/envs/notebook/lib/python3.7/site-packages/blib2to3/PatternGrammar.txt
[stage-in-pod-ksubablj] Writing grammar tables to /wXQPqe/.cache/black/22.1.0/PatternGrammar3.7.12.final.0.pickle
[stage-in-pod-ksubablj] Writing failed: [Errno 2] No such file or directory: '/wXQPqe/.cache/black/22.1.0/tmpp1qmff2o'
Executing:   0%|          | 0/10 [00:00<?, ?cell/s]Executing notebook with kernel: python3
Executing: 100%|██████████| 10/10 [00:03<00:00,  2.83cell/s]
[stage-in-pod-ksubablj] follow_logs end
Handling terminated pod name stage-in-pod-ksubablj with id 6ecfb76e-33a8-4da9-a1d9-019b8b73ec74
handling completion with 0
CALRISSIAN_DELETE_PODS=false(<class 'str'>)
returning False
DEBUG [step stage_in] produced output {
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_in/output_nb_file": {
        "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/output_nb.ipynb",
        "basename": "output_nb.ipynb",
        "nameroot": "output_nb",
        "nameext": ".ipynb",
        "class": "File",
        "checksum": "sha1$3a517655d5bf8673ade15bd09c529b82566a798f",
        "size": 7374,
        "http://commonwl.org/cwltool#generation": 0
    },
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_in/image_file": {
        "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
        "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
        "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL",
        "nameext": ".TIF",
        "class": "File",
        "checksum": "sha1$8724b4a1707e93006cb6b5fc96171154bed1ff1c",
        "size": 2121031,
        "http://commonwl.org/cwltool#generation": 0
    }
}
INFO [step stage_in] completed success
shutil.rmtree(/tmp/y1a8z_dw, True)
DEBUG restore [ram: 256, cores: 1] to available [ram: 817.741824, cores: 0.0]
INFO [workflow ] starting step downsample_landsat
DEBUG [step downsample_landsat] job input {
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/input_file": {
        "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
        "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
        "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL",
        "nameext": ".TIF",
        "class": "File",
        "checksum": "sha1$8724b4a1707e93006cb6b5fc96171154bed1ff1c",
        "size": 2121031,
        "http://commonwl.org/cwltool#generation": 0
    },
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/max_spin_time": 30,
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/min_spin_time": 15
}
DEBUG [step downsample_landsat] evaluated job input to {
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/input_file": {
        "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
        "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
        "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL",
        "nameext": ".TIF",
        "class": "File",
        "checksum": "sha1$8724b4a1707e93006cb6b5fc96171154bed1ff1c",
        "size": 2121031,
        "http://commonwl.org/cwltool#generation": 0
    },
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/max_spin_time": 30,
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/min_spin_time": 15
}
INFO [step downsample_landsat] start
DEBUG [job downsample_landsat] initializing from https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/downsample_landsat.cwl as part of step downsample_landsat
DEBUG [job downsample_landsat] {
    "input_file": {
        "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
        "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
        "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL",
        "nameext": ".TIF",
        "class": "File",
        "checksum": "sha1$8724b4a1707e93006cb6b5fc96171154bed1ff1c",
        "size": 2121031,
        "http://commonwl.org/cwltool#generation": 0
    },
    "max_spin_time": 30,
    "min_spin_time": 15
}
DEBUG [job downsample_landsat] path mappings is {
    "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF": [
        "/calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
        "/wXQPqe/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
        "File",
        false
    ]
}
DEBUG [job downsample_landsat] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "papermill"
    },
    {
        "position": [
            -1000000,
            1
        ],
        "datum": "/home/jovyan/downsample-landsat/downsample_landsat.ipynb"
    },
    {
        "position": [
            -1000000,
            2
        ],
        "datum": "output_nb.ipynb"
    },
    {
        "position": [
            1,
            "input_file"
        ],
        "shellQuote": false,
        "prefix": "--parameters",
        "valueFrom": "input_file \"$(self.basename)\"\n",
        "datum": {
            "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
            "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
            "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL",
            "nameext": ".TIF",
            "class": "File",
            "checksum": "sha1$8724b4a1707e93006cb6b5fc96171154bed1ff1c",
            "size": 2121031,
            "http://commonwl.org/cwltool#generation": 0,
            "path": "/wXQPqe/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
            "dirname": "/wXQPqe"
        }
    },
    {
        "position": [
            2,
            "min_spin_time"
        ],
        "shellQuote": false,
        "prefix": "--parameters",
        "valueFrom": "min_spin_time \"$(self)\"\n",
        "datum": 15
    },
    {
        "position": [
            3,
            "max_spin_time"
        ],
        "shellQuote": false,
        "prefix": "--parameters",
        "valueFrom": "max_spin_time \"$(self)\"\n",
        "datum": 30
    }
]
DEBUG wait_for_completion with 0 futures
DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out not ready
DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out not ready
DEBUG allocate [ram: 256, cores: 1] from available [ram: 1073.741824, cores: 1.0]
DEBUG wait_for_completion with 1 futures
DEBUG [job downsample_landsat] initial work dir {
    "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF": [
        "/calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
        "/wXQPqe/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF",
        "File",
        true
    ]
}
Building resources spec from {'cores': 1, 'ram': 256}
--------------------------------------------------------------------------------
apiVersion: v1
kind: Pod
metadata:
  labels: {}
  name: downsample-landsat-pod-klceaaah
spec:
  containers:
  - args:
    - /bin/sh -c 'papermill /home/jovyan/downsample-landsat/downsample_landsat.ipynb
      output_nb.ipynb --parameters input_file "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF"
      --parameters min_spin_time "15" --parameters max_spin_time "30"'
    command:
    - /bin/sh
    - -c
    env:
    - name: HOME
      value: /wXQPqe
    - name: TMPDIR
      value: /tmp
    image: !!python/object/new:ruamel.yaml.scalarstring.SingleQuotedScalarString
    - pymonger/downsample-landsat:2.0.3
    imagePullPolicy: Always
    name: downsample-landsat-container
    resources:
      requests:
        cpu: '1'
        memory: 256Mi
    volumeMounts:
    - mountPath: /wXQPqe
      name: tmpout-h43neo2jeheg
      readOnly: false
      subPath: tmpout-h43neo2jeheg/wlolfpk7
    - mountPath: /tmp
      name: tmpdir
    - mountPath: /wXQPqe/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF
      name: tmpout-h43neo2jeheg
      readOnly: true
      subPath: tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF
    workingDir: /wXQPqe
  initContainers: []
  restartPolicy: Never
  securityContext:
    runAsGroup: 0
    runAsUser: 1001
  volumes:
  - name: tmpout-h43neo2jeheg
    persistentVolumeClaim:
      claimName: tmpout-h43neo2jeheg
      readOnly: false
  - name: output-data-h43neo2jeheg
    persistentVolumeClaim:
      claimName: output-data-h43neo2jeheg
      readOnly: false
  - emptyDir: {}
    name: tmpdir
--------------------------------------------------------------------------------

Created k8s pod name downsample-landsat-pod-klceaaah with id 57dba616-a26d-41fe-a644-b5f455979419
PodMonitor adding downsample-landsat-pod-klceaaah
k8s pod 'downsample-landsat-pod-klceaaah' started
[downsample-landsat-pod-klceaaah] follow_logs start
[downsample-landsat-pod-klceaaah] Input Notebook:  /home/jovyan/downsample-landsat/downsample_landsat.ipynb
[downsample-landsat-pod-klceaaah] Output Notebook: output_nb.ipynb
[downsample-landsat-pod-klceaaah] Generating grammar tables from /srv/conda/envs/notebook/lib/python3.7/site-packages/blib2to3/Grammar.txt
[downsample-landsat-pod-klceaaah] Writing grammar tables to /wXQPqe/.cache/black/22.1.0/Grammar3.7.12.final.0.pickle
[downsample-landsat-pod-klceaaah] Writing failed: [Errno 2] No such file or directory: '/wXQPqe/.cache/black/22.1.0/tmpii7o7abc'
[downsample-landsat-pod-klceaaah] Generating grammar tables from /srv/conda/envs/notebook/lib/python3.7/site-packages/blib2to3/PatternGrammar.txt
[downsample-landsat-pod-klceaaah] Writing grammar tables to /wXQPqe/.cache/black/22.1.0/PatternGrammar3.7.12.final.0.pickle
[downsample-landsat-pod-klceaaah] Writing failed: [Errno 2] No such file or directory: '/wXQPqe/.cache/black/22.1.0/tmpsd7eqmcb'
Executing:   0%|          | 0/14 [00:00<?, ?cell/s]Executing notebook with kernel: python3
Executing: 100%|██████████| 14/14 [00:28<00:00,  2.01s/cell]
[downsample-landsat-pod-klceaaah] follow_logs end
Handling terminated pod name downsample-landsat-pod-klceaaah with id 57dba616-a26d-41fe-a644-b5f455979419
handling completion with 0
CALRISSIAN_DELETE_PODS=false(<class 'str'>)
returning False
DEBUG [step downsample_landsat] produced output {
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/output_nb_file": {
        "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/output_nb.ipynb",
        "basename": "output_nb.ipynb",
        "nameroot": "output_nb",
        "nameext": ".ipynb",
        "class": "File",
        "checksum": "sha1$c9c7fb003df4c0704c85638dbddd876cc956182e",
        "size": 12349,
        "http://commonwl.org/cwltool#generation": 0
    },
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/dataset_dir": {
        "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "nameext": "",
        "class": "Directory"
    }
}
INFO [step downsample_landsat] completed success
shutil.rmtree(/tmp/ax5s3b_e, True)
DEBUG restore [ram: 256, cores: 1] to available [ram: 817.741824, cores: 0.0]
INFO [workflow ] starting step stage_out
DEBUG [step stage_out] job input {
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/aws_access_key_id": "(secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)",
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/aws_secret_access_key": "(secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)",
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/base_dataset_url": "s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades",
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/dataset_dir": {
        "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "nameext": "",
        "class": "Directory"
    }
}
DEBUG [step stage_out] evaluated job input to {
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/aws_access_key_id": "(secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)",
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/aws_secret_access_key": "(secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)",
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/base_dataset_url": "s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades",
    "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/dataset_dir": {
        "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "nameext": "",
        "class": "Directory"
    }
}
INFO [step stage_out] start
DEBUG [job stage_out] initializing from https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/stage_out.cwl as part of step stage_out
DEBUG [job stage_out] {
    "aws_access_key_id": "(secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)",
    "aws_secret_access_key": "(secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)",
    "base_dataset_url": "s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades",
    "dataset_dir": {
        "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "nameext": "",
        "class": "Directory"
    }
}
DEBUG [job stage_out] path mappings is {
    "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled": [
        "/calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "/var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "Directory",
        true
    ]
}
DEBUG [job stage_out] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "sh"
    },
    {
        "position": [
            0,
            0
        ],
        "datum": "-c"
    },
    {
        "position": [
            0,
            1
        ],
        "valueFrom": "if [ ! -d $HOME/.aws ]; then cp -rp .aws $HOME/; fi && aws s3 cp --recursive $(inputs.dataset_dir.path) $(inputs.base_dataset_url)/$(inputs.dataset_dir.basename)"
    }
]
DEBUG wait_for_completion with 0 futures
DEBUG allocate [ram: 256, cores: 1] from available [ram: 1073.741824, cores: 1.0]
DEBUG [job stage_out] initial work dir {
    "_:c2799287-b7c5-4bb5-abe3-55cf6b3d70a6": [
        "[default]\noutput = json\nregion = us-west-2\naws_access_key_id = (secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)\naws_secret_access_key = (secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)\n",
        "/wXQPqe/.aws/credentials",
        "CreateFile",
        true
    ]
}
DEBUG wait_for_completion with 1 futures
Building resources spec from {'cores': 1, 'ram': 256}
--------------------------------------------------------------------------------
apiVersion: v1
kind: Pod
metadata:
  labels: {}
  name: stage-out-pod-ntekqdpz
spec:
  containers:
  - args:
    - sh -c 'if [ ! -d $HOME/.aws ]; then cp -rp .aws $HOME/; fi && aws s3 cp --recursive
      /var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled
      s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled'
    command:
    - /bin/sh
    - -c
    env:
    - name: HOME
      value: /wXQPqe
    - name: TMPDIR
      value: /tmp
    image: pymonger/aws-cli
    imagePullPolicy: Always
    name: stage-out-container
    resources:
      requests:
        cpu: '1'
        memory: 256Mi
    volumeMounts:
    - mountPath: /wXQPqe
      name: tmpout-h43neo2jeheg
      readOnly: false
      subPath: tmpout-h43neo2jeheg/m4h0_4nu
    - mountPath: /tmp
      name: tmpdir
    - mountPath: /var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled
      name: tmpout-h43neo2jeheg
      readOnly: true
      subPath: tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled
    workingDir: /wXQPqe
  initContainers: []
  restartPolicy: Never
  securityContext:
    runAsGroup: 0
    runAsUser: 1001
  volumes:
  - name: tmpout-h43neo2jeheg
    persistentVolumeClaim:
      claimName: tmpout-h43neo2jeheg
      readOnly: false
  - name: output-data-h43neo2jeheg
    persistentVolumeClaim:
      claimName: output-data-h43neo2jeheg
      readOnly: false
  - emptyDir: {}
    name: tmpdir
--------------------------------------------------------------------------------

Created k8s pod name stage-out-pod-ntekqdpz with id 197bc5a8-25d7-4ff6-8eb4-d6273a83416c
PodMonitor adding stage-out-pod-ntekqdpz
k8s pod 'stage-out-pod-ntekqdpz' started
[stage-out-pod-ntekqdpz] follow_logs start
upload: ../var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.met.json to s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.met.json
upload: ../var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.TIF to s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.TIF
upload: ../var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.browse.png to s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.browse.png
[stage-out-pod-ntekqdpz] follow_logs end
Handling terminated pod name stage-out-pod-ntekqdpz with id 197bc5a8-25d7-4ff6-8eb4-d6273a83416c
handling completion with 0
CALRISSIAN_DELETE_PODS=false(<class 'str'>)
returning False
DEBUG [step stage_out] produced output {}
INFO [step stage_out] completed success
INFO [workflow ] completed success
DEBUG [workflow ] outputs {
    "final_dataset_dir": {
        "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "nameext": "",
        "class": "Directory"
    }
}
shutil.rmtree(/tmp/m92_fax1, True)
DEBUG restore [ram: 256, cores: 1] to available [ram: 817.741824, cores: 0.0]
DEBUG wait_for_completion with 0 futures
DEBUG wait_for_completion with 0 futures
DEBUG Finishing ThreadPoolExecutor.run_jobs: total_resources=[ram: 1073.741824, cores: 1.0], available_resources=[ram: 1073.741824, cores: 1.0]
INFO Final process status is success
{
    "final_dataset_dir": {
        "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled",
        "class": "Directory",
        "listing": [
            {
                "class": "File",
                "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.met.json",
                "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.met.json",
                "path": "/calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.met.json"
            },
            {
                "class": "File",
                "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.browse.png",
                "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.browse.png",
                "path": "/calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.browse.png"
            },
            {
                "class": "File",
                "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.TIF",
                "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.TIF",
                "path": "/calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.TIF"
            }
        ],
        "path": "/calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled"
    }
}
CALRISSIAN_DELETE_PODS=false(<class 'str'>)
returning False
# BEGIN docker-usage.json
{
    "cores_allowed": 1.0,
    "ram_mb_allowed": 1073.741824,
    "children": [
        {
            "cpus": 1.0,
            "ram_megabytes": 268.435456,
            "disk_megabytes": 2.128405,
            "name": "stage_in",
            "start_time": "2022-05-10T20:53:28+00:00",
            "finish_time": "2022-05-10T20:53:32+00:00",
            "elapsed_hours": 0.0011111111111111111,
            "elapsed_seconds": 4.0,
            "ram_megabyte_hours": 0.29826161777777777,
            "cpu_hours": 0.0011111111111111111
        },
        {
            "cpus": 1.0,
            "ram_megabytes": 268.435456,
            "disk_megabytes": 0.012349,
            "name": "downsample_landsat",
            "start_time": "2022-05-10T20:53:35+00:00",
            "finish_time": "2022-05-10T20:54:04+00:00",
            "elapsed_hours": 0.008055555555555555,
            "elapsed_seconds": 29.0,
            "ram_megabyte_hours": 2.1623967288888886,
            "cpu_hours": 0.008055555555555555
        },
        {
            "cpus": 1.0,
            "ram_megabytes": 268.435456,
            "disk_megabytes": 0.0,
            "name": "stage_out",
            "start_time": "2022-05-10T20:54:08+00:00",
            "finish_time": "2022-05-10T20:54:09+00:00",
            "elapsed_hours": 0.0002777777777777778,
            "elapsed_seconds": 1.0,
            "ram_megabyte_hours": 0.07456540444444444,
            "cpu_hours": 0.0002777777777777778
        }
    ],
    "start_time": "2022-05-10T20:53:28+00:00",
    "finish_time": "2022-05-10T20:54:09+00:00",
    "elapsed_hours": 0.01138888888888889,
    "elapsed_seconds": 41.0,
    "total_cpu_hours": 0.009444444444444445,
    "total_ram_megabyte_hours": 2.535223751111111,
    "total_disk_megabytes": 2.140754,
    "total_tasks": 3,
    "max_parallel_cpus": 1.0,
    "max_parallel_ram_megabytes": 268.435456,
    "max_parallel_tasks": 1
}
# END docker-usage.json

[Tue May 10 14:07:15 gmanipon@MT-202634 calrissian]$ kc logs job/calrissian-job-h43neo2jeheg Defaulted container "calrissian-job" out of: calrissian-job, init-volumes (init) INFO calrissian 0.10.0 (cwltool 3.1.20220303001946) https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl:8:3: Warning: checking item Warning: Field `class` contains undefined reference to `http://commonwl.org/cwltool#Secrets` INFO https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl:8:3: Unknown hint http://commonwl.org/cwltool#Secrets https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/stage_out.cwl:8:3: Warning: checking item Warning: Field `class` contains undefined reference to `http://commonwl.org/cwltool#Secrets` INFO https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/stage_out.cwl:8:3: Unknown hint http://commonwl.org/cwltool#Secrets DEBUG Parsed job order from command line: { "id": "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl", "workflow_aws_access_key_id": "(secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)", "workflow_aws_secret_access_key": "(secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)", "workflow_base_dataset_url": "s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades", "workflow_input_url": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "workflow_max_spin_time": 30, "workflow_min_spin_time": 15 } DEBUG Starting ThreadPoolJobExecutor.run_jobs: total_resources=[ram: 1073.741824, cores: 1.0], max_workers=None DEBUG [workflow ] initialized from https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl DEBUG wait_for_completion with 0 futures DEBUG [workflow ] inputs { "workflow_aws_access_key_id": "(secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)", "workflow_aws_secret_access_key": "(secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)", "workflow_base_dataset_url": "s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades", "workflow_input_url": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "workflow_max_spin_time": 30, "workflow_min_spin_time": 15 } DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat not ready DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out not ready INFO [workflow ] starting step stage_in DEBUG [step stage_in] job input { "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_in/input_url": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF" } DEBUG [step stage_in] evaluated job input to { "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_in/input_url": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF" } INFO [step stage_in] start DEBUG [job stage_in] initializing from https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/stage_in.cwl as part of step stage_in DEBUG [job stage_in] { "input_url": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF" } DEBUG [job stage_in] path mappings is {} DEBUG [job stage_in] command line bindings is [ { "position": [ -1000000, 0 ], "datum": "papermill" }, { "position": [ -1000000, 1 ], "datum": "/home/jovyan/downsample-landsat/stage_in.ipynb" }, { "position": [ -1000000, 2 ], "datum": "output_nb.ipynb" }, { "position": [ 1, "input_url" ], "shellQuote": false, "prefix": "--parameters", "valueFrom": "input_url \"$(self)\"\n", "datum": "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF" } ] DEBUG wait_for_completion with 0 futures DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat not ready DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out not ready DEBUG allocate [ram: 0, cores: 0] from available [ram: 1073.741824, cores: 1.0] INFO [workflow ] start DEBUG restore [ram: 0, cores: 0] to available [ram: 1073.741824, cores: 1.0] DEBUG allocate [ram: 256, cores: 1] from available [ram: 1073.741824, cores: 1.0] DEBUG [job stage_in] initial work dir {} DEBUG wait_for_completion with 2 futures DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat not ready DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out not ready DEBUG wait_for_completion with 1 futures Building resources spec from {'cores': 1, 'ram': 256} -------------------------------------------------------------------------------- apiVersion: v1 kind: Pod metadata: labels: {} name: stage-in-pod-ksubablj spec: containers: - args: - /bin/sh -c 'papermill /home/jovyan/downsample-landsat/stage_in.ipynb output_nb.ipynb --parameters input_url "https://github.com/pymonger/downsample-landsat/releases/download/1.0.0/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF"' command: - /bin/sh - -c env: - name: HOME value: /wXQPqe - name: TMPDIR value: /tmp image: !!python/object/new:ruamel.yaml.scalarstring.SingleQuotedScalarString - pymonger/downsample-landsat:2.0.3 imagePullPolicy: Always name: stage-in-container resources: requests: cpu: '1' memory: 256Mi volumeMounts: - mountPath: /wXQPqe name: tmpout-h43neo2jeheg readOnly: false subPath: tmpout-h43neo2jeheg/pu2mkflz - mountPath: /tmp name: tmpdir workingDir: /wXQPqe initContainers: [] restartPolicy: Never securityContext: runAsGroup: 0 runAsUser: 1001 volumes: - name: tmpout-h43neo2jeheg persistentVolumeClaim: claimName: tmpout-h43neo2jeheg readOnly: false - name: output-data-h43neo2jeheg persistentVolumeClaim: claimName: output-data-h43neo2jeheg readOnly: false - emptyDir: {} name: tmpdir -------------------------------------------------------------------------------- Created k8s pod name stage-in-pod-ksubablj with id 6ecfb76e-33a8-4da9-a1d9-019b8b73ec74 PodMonitor adding stage-in-pod-ksubablj k8s pod 'stage-in-pod-ksubablj' started [stage-in-pod-ksubablj] follow_logs start [stage-in-pod-ksubablj] Input Notebook: /home/jovyan/downsample-landsat/stage_in.ipynb [stage-in-pod-ksubablj] Output Notebook: output_nb.ipynb [stage-in-pod-ksubablj] Generating grammar tables from /srv/conda/envs/notebook/lib/python3.7/site-packages/blib2to3/Grammar.txt [stage-in-pod-ksubablj] Writing grammar tables to /wXQPqe/.cache/black/22.1.0/Grammar3.7.12.final.0.pickle [stage-in-pod-ksubablj] Writing failed: [Errno 2] No such file or directory: '/wXQPqe/.cache/black/22.1.0/tmp2naxidzt' [stage-in-pod-ksubablj] Generating grammar tables from /srv/conda/envs/notebook/lib/python3.7/site-packages/blib2to3/PatternGrammar.txt [stage-in-pod-ksubablj] Writing grammar tables to /wXQPqe/.cache/black/22.1.0/PatternGrammar3.7.12.final.0.pickle [stage-in-pod-ksubablj] Writing failed: [Errno 2] No such file or directory: '/wXQPqe/.cache/black/22.1.0/tmpp1qmff2o' Executing: 0%| | 0/10 [00:00<?, ?cell/s]Executing notebook with kernel: python3 Executing: 100%|██████████| 10/10 [00:03<00:00, 2.83cell/s] [stage-in-pod-ksubablj] follow_logs end Handling terminated pod name stage-in-pod-ksubablj with id 6ecfb76e-33a8-4da9-a1d9-019b8b73ec74 handling completion with 0 CALRISSIAN_DELETE_PODS=false(<class 'str'>) returning False DEBUG [step stage_in] produced output { "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_in/output_nb_file": { "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/output_nb.ipynb", "basename": "output_nb.ipynb", "nameroot": "output_nb", "nameext": ".ipynb", "class": "File", "checksum": "sha1$3a517655d5bf8673ade15bd09c529b82566a798f", "size": 7374, "http://commonwl.org/cwltool#generation": 0 }, "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_in/image_file": { "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL", "nameext": ".TIF", "class": "File", "checksum": "sha1$8724b4a1707e93006cb6b5fc96171154bed1ff1c", "size": 2121031, "http://commonwl.org/cwltool#generation": 0 } } INFO [step stage_in] completed success shutil.rmtree(/tmp/y1a8z_dw, True) DEBUG restore [ram: 256, cores: 1] to available [ram: 817.741824, cores: 0.0] INFO [workflow ] starting step downsample_landsat DEBUG [step downsample_landsat] job input { "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/input_file": { "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL", "nameext": ".TIF", "class": "File", "checksum": "sha1$8724b4a1707e93006cb6b5fc96171154bed1ff1c", "size": 2121031, "http://commonwl.org/cwltool#generation": 0 }, "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/max_spin_time": 30, "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/min_spin_time": 15 } DEBUG [step downsample_landsat] evaluated job input to { "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/input_file": { "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL", "nameext": ".TIF", "class": "File", "checksum": "sha1$8724b4a1707e93006cb6b5fc96171154bed1ff1c", "size": 2121031, "http://commonwl.org/cwltool#generation": 0 }, "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/max_spin_time": 30, "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/min_spin_time": 15 } INFO [step downsample_landsat] start DEBUG [job downsample_landsat] initializing from https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/downsample_landsat.cwl as part of step downsample_landsat DEBUG [job downsample_landsat] { "input_file": { "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL", "nameext": ".TIF", "class": "File", "checksum": "sha1$8724b4a1707e93006cb6b5fc96171154bed1ff1c", "size": 2121031, "http://commonwl.org/cwltool#generation": 0 }, "max_spin_time": 30, "min_spin_time": 15 } DEBUG [job downsample_landsat] path mappings is { "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF": [ "/calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "/wXQPqe/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "File", false ] } DEBUG [job downsample_landsat] command line bindings is [ { "position": [ -1000000, 0 ], "datum": "papermill" }, { "position": [ -1000000, 1 ], "datum": "/home/jovyan/downsample-landsat/downsample_landsat.ipynb" }, { "position": [ -1000000, 2 ], "datum": "output_nb.ipynb" }, { "position": [ 1, "input_file" ], "shellQuote": false, "prefix": "--parameters", "valueFrom": "input_file \"$(self.basename)\"\n", "datum": { "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL", "nameext": ".TIF", "class": "File", "checksum": "sha1$8724b4a1707e93006cb6b5fc96171154bed1ff1c", "size": 2121031, "http://commonwl.org/cwltool#generation": 0, "path": "/wXQPqe/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "dirname": "/wXQPqe" } }, { "position": [ 2, "min_spin_time" ], "shellQuote": false, "prefix": "--parameters", "valueFrom": "min_spin_time \"$(self)\"\n", "datum": 15 }, { "position": [ 3, "max_spin_time" ], "shellQuote": false, "prefix": "--parameters", "valueFrom": "max_spin_time \"$(self)\"\n", "datum": 30 } ] DEBUG wait_for_completion with 0 futures DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out not ready DEBUG [workflow ] job step https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out not ready DEBUG allocate [ram: 256, cores: 1] from available [ram: 1073.741824, cores: 1.0] DEBUG wait_for_completion with 1 futures DEBUG [job downsample_landsat] initial work dir { "file:///calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF": [ "/calrissian/tmpout/tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "/wXQPqe/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF", "File", true ] } Building resources spec from {'cores': 1, 'ram': 256} -------------------------------------------------------------------------------- apiVersion: v1 kind: Pod metadata: labels: {} name: downsample-landsat-pod-klceaaah spec: containers: - args: - /bin/sh -c 'papermill /home/jovyan/downsample-landsat/downsample_landsat.ipynb output_nb.ipynb --parameters input_file "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF" --parameters min_spin_time "15" --parameters max_spin_time "30"' command: - /bin/sh - -c env: - name: HOME value: /wXQPqe - name: TMPDIR value: /tmp image: !!python/object/new:ruamel.yaml.scalarstring.SingleQuotedScalarString - pymonger/downsample-landsat:2.0.3 imagePullPolicy: Always name: downsample-landsat-container resources: requests: cpu: '1' memory: 256Mi volumeMounts: - mountPath: /wXQPqe name: tmpout-h43neo2jeheg readOnly: false subPath: tmpout-h43neo2jeheg/wlolfpk7 - mountPath: /tmp name: tmpdir - mountPath: /wXQPqe/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF name: tmpout-h43neo2jeheg readOnly: true subPath: tmpout-h43neo2jeheg/pu2mkflz/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL.TIF workingDir: /wXQPqe initContainers: [] restartPolicy: Never securityContext: runAsGroup: 0 runAsUser: 1001 volumes: - name: tmpout-h43neo2jeheg persistentVolumeClaim: claimName: tmpout-h43neo2jeheg readOnly: false - name: output-data-h43neo2jeheg persistentVolumeClaim: claimName: output-data-h43neo2jeheg readOnly: false - emptyDir: {} name: tmpdir -------------------------------------------------------------------------------- Created k8s pod name downsample-landsat-pod-klceaaah with id 57dba616-a26d-41fe-a644-b5f455979419 PodMonitor adding downsample-landsat-pod-klceaaah k8s pod 'downsample-landsat-pod-klceaaah' started [downsample-landsat-pod-klceaaah] follow_logs start [downsample-landsat-pod-klceaaah] Input Notebook: /home/jovyan/downsample-landsat/downsample_landsat.ipynb [downsample-landsat-pod-klceaaah] Output Notebook: output_nb.ipynb [downsample-landsat-pod-klceaaah] Generating grammar tables from /srv/conda/envs/notebook/lib/python3.7/site-packages/blib2to3/Grammar.txt [downsample-landsat-pod-klceaaah] Writing grammar tables to /wXQPqe/.cache/black/22.1.0/Grammar3.7.12.final.0.pickle [downsample-landsat-pod-klceaaah] Writing failed: [Errno 2] No such file or directory: '/wXQPqe/.cache/black/22.1.0/tmpii7o7abc' [downsample-landsat-pod-klceaaah] Generating grammar tables from /srv/conda/envs/notebook/lib/python3.7/site-packages/blib2to3/PatternGrammar.txt [downsample-landsat-pod-klceaaah] Writing grammar tables to /wXQPqe/.cache/black/22.1.0/PatternGrammar3.7.12.final.0.pickle [downsample-landsat-pod-klceaaah] Writing failed: [Errno 2] No such file or directory: '/wXQPqe/.cache/black/22.1.0/tmpsd7eqmcb' Executing: 0%| | 0/14 [00:00<?, ?cell/s]Executing notebook with kernel: python3 Executing: 100%|██████████| 14/14 [00:28<00:00, 2.01s/cell] [downsample-landsat-pod-klceaaah] follow_logs end Handling terminated pod name downsample-landsat-pod-klceaaah with id 57dba616-a26d-41fe-a644-b5f455979419 handling completion with 0 CALRISSIAN_DELETE_PODS=false(<class 'str'>) returning False DEBUG [step downsample_landsat] produced output { "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/output_nb_file": { "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/output_nb.ipynb", "basename": "output_nb.ipynb", "nameroot": "output_nb", "nameext": ".ipynb", "class": "File", "checksum": "sha1$c9c7fb003df4c0704c85638dbddd876cc956182e", "size": 12349, "http://commonwl.org/cwltool#generation": 0 }, "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#downsample_landsat/dataset_dir": { "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "nameext": "", "class": "Directory" } } INFO [step downsample_landsat] completed success shutil.rmtree(/tmp/ax5s3b_e, True) DEBUG restore [ram: 256, cores: 1] to available [ram: 817.741824, cores: 0.0] INFO [workflow ] starting step stage_out DEBUG [step stage_out] job input { "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/aws_access_key_id": "(secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)", "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/aws_secret_access_key": "(secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)", "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/base_dataset_url": "s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades", "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/dataset_dir": { "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "nameext": "", "class": "Directory" } } DEBUG [step stage_out] evaluated job input to { "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/aws_access_key_id": "(secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)", "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/aws_secret_access_key": "(secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)", "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/base_dataset_url": "s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades", "https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/workflow.cwl#stage_out/dataset_dir": { "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "nameext": "", "class": "Directory" } } INFO [step stage_out] start DEBUG [job stage_out] initializing from https://raw.githubusercontent.com/pymonger/downsample-landsat/2.0.3/stage_out.cwl as part of step stage_out DEBUG [job stage_out] { "aws_access_key_id": "(secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)", "aws_secret_access_key": "(secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)", "base_dataset_url": "s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades", "dataset_dir": { "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "nameext": "", "class": "Directory" } } DEBUG [job stage_out] path mappings is { "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled": [ "/calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "/var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "Directory", true ] } DEBUG [job stage_out] command line bindings is [ { "position": [ -1000000, 0 ], "datum": "sh" }, { "position": [ 0, 0 ], "datum": "-c" }, { "position": [ 0, 1 ], "valueFrom": "if [ ! -d $HOME/.aws ]; then cp -rp .aws $HOME/; fi && aws s3 cp --recursive $(inputs.dataset_dir.path) $(inputs.base_dataset_url)/$(inputs.dataset_dir.basename)" } ] DEBUG wait_for_completion with 0 futures DEBUG allocate [ram: 256, cores: 1] from available [ram: 1073.741824, cores: 1.0] DEBUG [job stage_out] initial work dir { "_:c2799287-b7c5-4bb5-abe3-55cf6b3d70a6": [ "[default]\noutput = json\nregion = us-west-2\naws_access_key_id = (secret-03d38492-42f0-4a78-a9cf-842ca203e9d6)\naws_secret_access_key = (secret-a0ceb54b-824d-4683-aed8-2b94e5fd6483)\n", "/wXQPqe/.aws/credentials", "CreateFile", true ] } DEBUG wait_for_completion with 1 futures Building resources spec from {'cores': 1, 'ram': 256} -------------------------------------------------------------------------------- apiVersion: v1 kind: Pod metadata: labels: {} name: stage-out-pod-ntekqdpz spec: containers: - args: - sh -c 'if [ ! -d $HOME/.aws ]; then cp -rp .aws $HOME/; fi && aws s3 cp --recursive /var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled' command: - /bin/sh - -c env: - name: HOME value: /wXQPqe - name: TMPDIR value: /tmp image: pymonger/aws-cli imagePullPolicy: Always name: stage-out-container resources: requests: cpu: '1' memory: 256Mi volumeMounts: - mountPath: /wXQPqe name: tmpout-h43neo2jeheg readOnly: false subPath: tmpout-h43neo2jeheg/m4h0_4nu - mountPath: /tmp name: tmpdir - mountPath: /var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled name: tmpout-h43neo2jeheg readOnly: true subPath: tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled workingDir: /wXQPqe initContainers: [] restartPolicy: Never securityContext: runAsGroup: 0 runAsUser: 1001 volumes: - name: tmpout-h43neo2jeheg persistentVolumeClaim: claimName: tmpout-h43neo2jeheg readOnly: false - name: output-data-h43neo2jeheg persistentVolumeClaim: claimName: output-data-h43neo2jeheg readOnly: false - emptyDir: {} name: tmpdir -------------------------------------------------------------------------------- Created k8s pod name stage-out-pod-ntekqdpz with id 197bc5a8-25d7-4ff6-8eb4-d6273a83416c PodMonitor adding stage-out-pod-ntekqdpz k8s pod 'stage-out-pod-ntekqdpz' started [stage-out-pod-ntekqdpz] follow_logs start upload: ../var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.met.json to s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.met.json upload: ../var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.TIF to s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.TIF upload: ../var/lib/cwl/stg0aa93776-fa8d-43b3-9820-4dcf0309eeff/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.browse.png to s3://hysds-dataset-bucket-gman-test/docker-on-desktop-k8s-via-ades/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.browse.png [stage-out-pod-ntekqdpz] follow_logs end Handling terminated pod name stage-out-pod-ntekqdpz with id 197bc5a8-25d7-4ff6-8eb4-d6273a83416c handling completion with 0 CALRISSIAN_DELETE_PODS=false(<class 'str'>) returning False DEBUG [step stage_out] produced output {} INFO [step stage_out] completed success INFO [workflow ] completed success DEBUG [workflow ] outputs { "final_dataset_dir": { "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "nameroot": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "nameext": "", "class": "Directory" } } shutil.rmtree(/tmp/m92_fax1, True) DEBUG restore [ram: 256, cores: 1] to available [ram: 817.741824, cores: 0.0] DEBUG wait_for_completion with 0 futures DEBUG wait_for_completion with 0 futures DEBUG Finishing ThreadPoolExecutor.run_jobs: total_resources=[ram: 1073.741824, cores: 1.0], available_resources=[ram: 1073.741824, cores: 1.0] INFO Final process status is success { "final_dataset_dir": { "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled", "class": "Directory", "listing": [ { "class": "File", "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.met.json", "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.met.json", "path": "/calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.met.json" }, { "class": "File", "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.browse.png", "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.browse.png", "path": "/calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.browse.png" }, { "class": "File", "location": "file:///calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.TIF", "basename": "LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.TIF", "path": "/calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled.TIF" } ], "path": "/calrissian/tmpout/tmpout-h43neo2jeheg/wlolfpk7/LC08_L1TP_065045_20211205_20211215_02_T1_QA_PIXEL_downsampled" } } CALRISSIAN_DELETE_PODS=false(<class 'str'>) returning False # BEGIN docker-usage.json { "cores_allowed": 1.0, "ram_mb_allowed": 1073.741824, "children": [ { "cpus": 1.0, "ram_megabytes": 268.435456, "disk_megabytes": 2.128405, "name": "stage_in", "start_time": "2022-05-10T20:53:28+00:00", "finish_time": "2022-05-10T20:53:32+00:00", "elapsed_hours": 0.0011111111111111111, "elapsed_seconds": 4.0, "ram_megabyte_hours": 0.29826161777777777, "cpu_hours": 0.0011111111111111111 }, { "cpus": 1.0, "ram_megabytes": 268.435456, "disk_megabytes": 0.012349, "name": "downsample_landsat", "start_time": "2022-05-10T20:53:35+00:00", "finish_time": "2022-05-10T20:54:04+00:00", "elapsed_hours": 0.008055555555555555, "elapsed_seconds": 29.0, "ram_megabyte_hours": 2.1623967288888886, "cpu_hours": 0.008055555555555555 }, { "cpus": 1.0, "ram_megabytes": 268.435456, "disk_megabytes": 0.0, "name": "stage_out", "start_time": "2022-05-10T20:54:08+00:00", "finish_time": "2022-05-10T20:54:09+00:00", "elapsed_hours": 0.0002777777777777778, "elapsed_seconds": 1.0, "ram_megabyte_hours": 0.07456540444444444, "cpu_hours": 0.0002777777777777778 } ], "start_time": "2022-05-10T20:53:28+00:00", "finish_time": "2022-05-10T20:54:09+00:00", "elapsed_hours": 0.01138888888888889, "elapsed_seconds": 41.0, "total_cpu_hours": 0.009444444444444445, "total_ram_megabyte_hours": 2.535223751111111, "total_disk_megabytes": 2.140754, "total_tasks": 3, "max_parallel_cpus": 1.0, "max_parallel_ram_megabytes": 268.435456, "max_parallel_tasks": 1 } # END docker-usage.json

MAAP-Project / maap-hec-aws

R1: Collect metrics on ADES/K8S #21