Open ttomasz opened 1 year ago
boring-cyborg[bot]
commented
1 year ago Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
hussein-awala
commented
1 year ago That's strange, there should be no difference between using .value and not using it.
If you tested it and it solved your problem, you can open a PR. I just assigned you!
Apache Airflow version
2.6.3
What happened
I'm observing that when using Microsoft Azure provider with Azure Batch service the AzureBatchHook fails to properly detect failed tasks which means that airflow task will finish as succeeded instead of failed.
As far as I can tell this comparison: https://github.com/apache/airflow/blob/main/airflow/providers/microsoft/azure/hooks/batch.py#L372 never returns True. I suspect that maybe the enum is generated dynamically or using other definition (microsoft's code is pretty weird) and since in Python enums are compared using identity instead of value it does not return True even for failed tasks.
I tried in my own method the following condition and it worked for me:
if task.executionInfo is not None and task.executionInfo.result.value == batch_models.TaskExecutionResult.failure.valueWhat you think should happen instead
When Azure Batch task fails then Python operator should be marked as failed. AzureBatchHook.wait_for_job_tasks_to_complete should return list of failed tasks.
How to reproduce
Well, that would require creating Azure Batch account in Azure then running some task using the AzureBatchOperator or PythonOperator with a function that utilizes AzureBatchHook
Operating System
not sure
Versions of Apache Airflow Providers
No response
Deployment
Other 3rd-party Helm chart
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
Code of Conduct