Getting standard error

[kokleong9406]

I would like to check whether cwl-airflow is able to show "what went wrong DURING a workflow step run"? Currently, I have a workflow step doing wget, which I purposely supply the wrong url

Workflow file: 3-wget-workflow-v2.txt CommandLineTool file: 3-wget-v2.txt Input file: 3-wget-input.txt

I was expecting the log files to contain information about "what went wrong DURING a workflow step run", like in the case when we do "cwltool 3-wget-workflow-v2.cwl 3-wget-input.yaml" in the Terminal, which will output a standard error file: wget-error-local.png.log. However, what I see in the log file (1.log) is something like below:

[2021-10-21 20:15:30,291] {load_tool.py:101} INFO - Resolved '/home/kokleong/projects/cwl-airflow/cwl_tmp_folder/_3-wget-workflow-v2_manual__2021-10-21T12_15_07.721579_00_00_rnx__vvc/wget/wget_step_workflow.cwl' to 'file:///home/kokleong/projects/cwl-airflow/cwl_tmp_folder/_3-wget-workflow-v2_manual2021-10-21T12_15_07.721579_00_00_rnx__vvc/wget/wget_step_workflow.cwl' [2021-10-21 20:15:37,769] {workflow_job.py:787} INFO - [workflow ] start [2021-10-21 20:15:37,769] {workflow_job.py:626} INFO - [workflow ] starting step wget [2021-10-21 20:15:37,770] {workflow_job.py:74} INFO - [step wget] start [2021-10-21 20:15:37,774] {job.py:259} INFO - [job wget] /home/kokleong/projects/cwl-airflow/cwl_tmp_folder/***_3-wget-workflow-v2_manual2021-10-21T12_15_07.721579_00_00_rnxvvc/wget/wget_step_cache/xoqkcufz$ wget \ https://s.yimgabc.com/rz/p/yahoo_homepage_en-US_s_f_p_bestfit_homepage.png \ -O \ local.png > /home/kokleong/projects/cwl-airflow/cwl_tmp_folder/***_3-wget-workflow-v2_manual__2021-10-21T12_15_07.721579_00_00_rnxvvc/wget/wget_step_cache/xoqkcufz/wget-local.png.log 2> /home/kokleong/projects/cwl-airflow/cwl_tmp_folder/_3-wget-workflow-v2_manual__2021-10-21T12_15_07.721579_00_00_rnx__vvc/wget/wget_step_cache/xoqkcufz/wget-error-local.png.log [2021-10-21 20:15:37,847] {job.py:401} WARNING - [job wget] completed permanentFail [2021-10-21 20:15:37,847] {workflow_job.py:585} WARNING - [step wget] completed permanentFail [2021-10-21 20:15:37,848] {workflow_job.py:549} INFO - [workflow ] completed permanentFail [2021-10-21 20:15:37,851] {taskinstance.py:1463} ERROR - Task failed with exception Traceback (most recent call last): File "/home/kokleong/projects/cwl-airflow/venv3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1165, in _run_raw_task self._prepare_and_execute_task_with_callbacks(context, task) File "/home/kokleong/projects/cwl-airflow/venv3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1283, in _prepare_and_execute_task_with_callbacks result = self._execute_task(context, task_copy) File "/home/kokleong/projects/cwl-airflow/venv3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1313, in _execute_task result = task_copy.execute(context=context) File "/home/kokleong/projects/cwl-airflow/venv3.8/lib/python3.8/site-packages/cwlairflow/extensions/operators/cwlstepoperator.py", line 40, in execute , step_report, skipped = execute_workflow_step( File "/home/kokleong/projects/cwl-airflow/venv3.8/lib/python3.8/site-packages/cwl_airflow/utilities/cwl.py", line 663, in execute_workflow_step raise ValueError("Failed to run workflow stepzzzzz") ValueError: Failed to run workflow stepzzzzz [2021-10-21 20:15:37,865] {taskinstance.py:1506} INFO - Marking task as FAILED. dag_id=_3-wget-workflow-v2, task_id=wget, execution_date=20211021T121507, start_date=20211021T121525, end_date=20211021T121537 [2021-10-21 20:15:38,002] {local_task_job.py:151} INFO - Task exited with return code 1 [2021-10-21 20:15:38,127] {local_task_job.py:261} INFO - 0 downstream tasks scheduled from follow-on schedule check

May I check wit you how to get the standard error?

My current use case is that, I develop a website for users to submit workflow attachments and input files, which will trigger the cwl-airflow. Apart from workflow run status and log files, I was hoping that users will have visibility to what went wrong during the course of the workflow run they submitted.

[michael-kotliar]

Hi @kokleong9406 Thank you for using CWL-Airflow. It looks like both of your stdout and stderr streams are redirected to files

/home/kokleong/projects/cwl-airflow/cwl_tmp_folder/***_3-wget-workflow-v2_manual__2021-10-21T12_15_07.721579_00_00_rnx__vvc/wget/wget_step_cache/xoqkcufz/wget-local.png.log 2> /home/kokleong/projects/cwl-airflow/cwl_tmp_folder/***_3-wget-workflow-v2_manual__2021-10-21T12_15_07.721579_00_00_rnx__vvc/wget/wget_step_cache/xoqkcufz/wget-error-local.png.log

So they might not be printed into the console even when you run the workflow with cwltool directly (please, correct me if I'm wrong). I would try update the workflow and tools to not redirect stderr and stdout logs into files. If you are using the latest CWL-Airflow from Pypi, you will probably have two files created in the folder where Airflow saves logs - one file for the CWL execution log, another one for the Airflow log. However, if you install CWL-Airflow from the latest commit from GitHub, you will have CWL execution log appended to the end of Airflow log, so you can get both of the logs from Airflow Webserver UI.

Let me know if you have any other questions, Michael

[kokleong9406]

Hi @michael-kotliar,

Yup, you're right. I tried different flags and arguments with cwltool itself, the errors will not be printed on the console. I am not sure is it the way it is, but seems like explicitly declaring it upfront in the cwl files is one of the possible solution to catch the stderr.

Oh ya, my cwl-airflow version is 1.2.11. In my .py dag file, my default_args actually look like below, which allows me to have the stdout and stderr saved to cwl_tmp_folder. I guess I would have to get the stderr and stdout from the cwl_tmp_folder then. :)

default_args={'cwl': {'inputs_folder': '/home/kokleong/projects/cwl-airflow/cwl_inputs_folder/admin', 'keep_tmp_data': True}}

[michael-kotliar]

That works too. Originally, I was planning to use the keep_tmp_data parameter for debugging purposes, but if you see it useful elsewhere, that's even better. I will close this issue, feel free to reopen if needed.