search

ga4gh / workflow-execution-service-schemas

The WES API is a standard way to run and manage portable workflows.
Apache License 2.0
82 stars 38 forks source link

In which field of the wes interface should the logs of the Cromwell engine and the tes server be reflected? #211

Open zhou-jianwen opened 1 year ago

zhou-jianwen commented 1 year ago

While attempting to implement a WES server, we founded that WES only provides stdout and stderr fields within the run and task dimensions. We would like to clarify whether, according to the standard design of WES, how should TES server's own log of a specific task be retained? Similarly, we seek to determine if the logs generated by the workflow execution engine, such as Cromwell, should be represented in a specific field within the WES interface.

patmagee commented 1 year ago

@zhou-jianwen at the moment this is not well supported. The WES 1.1.0 Release candidate (which I am hoping to merge and tag tomorrow) does address this by introducing a new field in the TaskLog schema: https://github.com/ga4gh/workflow-execution-service-schemas/pull/205/files#diff-1f171071387eb98ede480c7beb336dec6aa0a03e48a4c7a726fc19830121d804R907

uniqueg commented 1 year ago

To add on to what @patmagee said, we (and I think others) are storing the workflow engine (e.g., Cromwell) logs in stderr and stdout.

Are you willing to explain a bit about more about what WES/TES you are building? I'm curious. Perhaps it would be useful to schedule a call :)

zhou-jianwen commented 1 year ago

@zhou-jianwen at the moment this is not well supported. The WES 1.1.0 Release candidate (which I am hoping to merge and tag tomorrow) does address this by introducing a new field in the TaskLog schema: https://github.com/ga4gh/workflow-execution-service-schemas/pull/205/files#diff-1f171071387eb98ede480c7beb336dec6aa0a03e48a4c7a726fc19830121d804R907

Thank you for your enthusiastic help. I apologize for not keeping up with the those developments.
I'm considering whether it would be beneficial to record more properties , allowing wes instance to add additional information about TES task，like TaskState maybe .

patmagee commented 1 year ago

@zhou-jianwen for what it's worth, I view compliance with the spec as meaning you "must have what is in the spec, but not limited to that"

Imo if you have an api that has custom keys and properties, it would still be a valid WES api if it implements all Wes endpoints and dat models

zhou-jianwen commented 1 year ago

To add on to what @patmagee said, we (and I think others) are storing the workflow engine (e.g., Cromwell) logs in stderr and stdout.

Are you willing to explain a bit about more about what WES/TES you are building? I'm curious. Perhaps it would be useful to schedule a call :)

In line with your perspective, I also believe that the workflow execution engine log serves as the standard output within the WES dimension.

We are considering coupling AAI and accounting functions with WES and TES servers. It would be delighted to engage in a more in-depth conversation.

uniqueg commented 1 year ago

Great! Can you see the email address in my GitHub profile? Alternatively, you could also join the GA4GH Slack board with this invite link and contact me there. Note that there are dedicated #wes and #tes channels on that board that could be useful to you.

patmagee commented 1 year ago

@zhou-jianwen fyi Wes 1.1 is now released!