OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.
Describe the bug
When deploying a pipeline the API returns a 200 status code, indicating that the workflow has been created. However, attempting to trigger the pipeline immediately after results in a 400 error, as the DAG for the pipeline has not yet been created.
To Reproduce
Create an ingestion pipeline using the endpoint /v1/services/ingestionPipelines.
Deploy the just created pipeline using the endpoint /v1/services/ingestionPipelines/deploy/{ingestion_id} receiving the response {'code': 200, 'reason': '{"message": "Workflow [2372c559-a267-4365-83ad-c33e471cb05d] has been created"}', 'platform': 'Airflow'}.
Attempt to trigger the pipeline immediately after via endpoint /v1/services/ingestionPipelines/trigger/{ingestion_id}. The following error is encountered: {'code': 400, 'message': 'Failed to deploy pipeline [2372c559-a267-4365-83ad-c33e471cb05d] due to [Failed to trigger IngestionPipeline].'}.
Screenshots or steps to reproduce
Expected behavior
The API should only return a 200 response once the DAG has been created and it is ready to be triggered, preventing the subsequent 400 error.
Affected module Backend
Describe the bug When deploying a pipeline the API returns a 200 status code, indicating that the workflow has been created. However, attempting to trigger the pipeline immediately after results in a 400 error, as the DAG for the pipeline has not yet been created.
To Reproduce
/v1/services/ingestionPipelines./v1/services/ingestionPipelines/deploy/{ingestion_id}receiving the response{'code': 200, 'reason': '{"message": "Workflow [2372c559-a267-4365-83ad-c33e471cb05d] has been created"}', 'platform': 'Airflow'}./v1/services/ingestionPipelines/trigger/{ingestion_id}. The following error is encountered:{'code': 400, 'message': 'Failed to deploy pipeline [2372c559-a267-4365-83ad-c33e471cb05d] due to [Failed to trigger IngestionPipeline].'}.Screenshots or steps to reproduce
Expected behavior The API should only return a 200 response once the DAG has been created and it is ready to be triggered, preventing the subsequent 400 error.
Version:
openmetadata-ingestion[docker]==XYZ]Additional context The problem was discussed in this thread: https://openmetadata.slack.com/archives/C02B6955S4S/p1726231697820269