Closed kokleong9406 closed 2 years ago
Hi @kokleong9406 Thank you for using CWL-Airflow. It looks like both of your stdout and stderr streams are redirected to files
/home/kokleong/projects/cwl-airflow/cwl_tmp_folder/***_3-wget-workflow-v2_manual__2021-10-21T12_15_07.721579_00_00_rnx__vvc/wget/wget_step_cache/xoqkcufz/wget-local.png.log 2> /home/kokleong/projects/cwl-airflow/cwl_tmp_folder/***_3-wget-workflow-v2_manual__2021-10-21T12_15_07.721579_00_00_rnx__vvc/wget/wget_step_cache/xoqkcufz/wget-error-local.png.log
So they might not be printed into the console even when you run the workflow with cwltool
directly (please, correct me if I'm wrong). I would try update the workflow and tools to not redirect stderr and stdout logs into files. If you are using the latest CWL-Airflow from Pypi, you will probably have two files created in the folder where Airflow saves logs - one file for the CWL execution log, another one for the Airflow log. However, if you install CWL-Airflow from the latest commit from GitHub, you will have CWL execution log appended to the end of Airflow log, so you can get both of the logs from Airflow Webserver UI.
Let me know if you have any other questions, Michael
Hi @michael-kotliar,
Yup, you're right. I tried different flags and arguments with cwltool itself, the errors will not be printed on the console. I am not sure is it the way it is, but seems like explicitly declaring it upfront in the cwl files is one of the possible solution to catch the stderr.
Oh ya, my cwl-airflow version is 1.2.11. In my .py dag file, my default_args actually look like below, which allows me to have the stdout and stderr saved to cwl_tmp_folder. I guess I would have to get the stderr and stdout from the cwl_tmp_folder then. :)
default_args={'cwl': {'inputs_folder': '/home/kokleong/projects/cwl-airflow/cwl_inputs_folder/admin', 'keep_tmp_data': True}}
That works too. Originally, I was planning to use the keep_tmp_data
parameter for debugging purposes, but if you see it useful elsewhere, that's even better. I will close this issue, feel free to reopen if needed.
I would like to check whether cwl-airflow is able to show "what went wrong DURING a workflow step run"? Currently, I have a workflow step doing wget, which I purposely supply the wrong url
Workflow file: 3-wget-workflow-v2.txt CommandLineTool file: 3-wget-v2.txt Input file: 3-wget-input.txt
I was expecting the log files to contain information about "what went wrong DURING a workflow step run", like in the case when we do "cwltool 3-wget-workflow-v2.cwl 3-wget-input.yaml" in the Terminal, which will output a standard error file: wget-error-local.png.log. However, what I see in the log file (1.log) is something like below:
May I check wit you how to get the standard error?
My current use case is that, I develop a website for users to submit workflow attachments and input files, which will trigger the cwl-airflow. Apart from workflow run status and log files, I was hoping that users will have visibility to what went wrong during the course of the workflow run they submitted.