dstat: return all logfiles

DataBiosphere / dsub

Open-source command-line tool to run batch computing tasks and workflows on backend services such as Google Cloud.

Apache License 2.0

262 stars 45 forks source link

dstat: return all logfiles #75

Open calbach opened 6 years ago

calbach commented 6 years ago

Currently:

dsub --logging gs://bkt/logs/ ...
dstat --full ...
...
"logging": "gs://bkt/logs/generated-job-id.log"

Possible return value:

dstat --full ...
...
"logging": {
  "controller": "gs://bkt/logs/generated-job-id.log",
  "stderr": "gs://bkt/logs/generated-job-id-stderr.log",
  "stdout": "gs://bkt/logs/generated-job-id-stdout.log"
}

mbookman commented 6 years ago

Hi CH,

I filed a request for this against the Pipelines API last year. I would like for dstat to reflect what the provider returns. Other providers may provide different log files or API endpoints.

However, we can certainly fake this in dstat until direct support is available

-Matt

calbach commented 6 years ago

To expand on my proposal:

The provider contract would be to return a key-value string map of logical log names to log file URIs.
The above is the specific output from the Google provider, which needn't match other providers.

mbookman commented 6 years ago

SGTM