Open kokleong9406 opened 2 years ago
Hi @kokleong9406
Could you please provide more details on what kind of logs you would like to have? It looks like the reason why you want to have the scheduler logs is that there you can see the output from your 5_compute_prs.sh
script. However, you should be able the see the logs from your 5_compute_prs.sh
script in the task instance logs after the line CWL LOGS
. Please, let me know if you don't have this line in your task instance logs. Also, could you please provide a version of CWL-Airflow you use? This is the link on the code, that appends cwl related logs (a.k.a from your 5_compute_prs.sh
script) to the end of task instance logs https://github.com/Barski-lab/cwl-airflow/blob/ca14232bbb78df242c841047318bcb67c775379f/cwl_airflow/utilities/report.py#L52
Hi @michael-kotliar
Thank you for your reply. I see that I do not have "def append_cwl_log(content)" in report.py nor any of the files. The current version of CWL-Airflow I am using is 1.2.11. Which version do I need to use to have this function?
Btw, I guess I will try out bit by bit to integrate the "append_cwl_log" function into my existing cwl version as well to see how it goes.
Hi @kokleong9406
I think these changes are not in the release on Pypi yet, so you can install it from the latest master
branch commit on GitHub. Let me know if need any help with it.
Michael
Hi Michael,
Would like to ask that currently, is there anyway to get the standard output printed on airflow scheduler? Currently, I am able to get the log files via the Airflow REST API endpoint "/dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/logs/{task_try_number}". You may also refer to https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_task_instance. The log files will look something familiar like below:
I was thinking it would be great if I am also able to get the standard output that is printed on the airflow scheduler, which will look something like below:
Currently is this being supported in cwl-airflow? If no, could you point me to the source code that does this and hopefully I am able to do some tricks and modifications to get the standard output logged into the Airflow log files (which is saved to "1.log") :P