search

damavis / airflow-pentaho-plugin

Pentaho plugin for Apache Airflow - Orquestate pentaho transformations and jobs from Airflow
Apache License 2.0
38 stars 17 forks source link

CarteTransOperator works incorrect when transformation failed #30

Closed evgdol closed 1 year ago

evgdol commented 1 year ago

Scenario (simplified)

  1. We have the table in the database with constraints CREATE TABLE tst_constraints( id int PRIMARY KEY, val text )

  2. We have the simple transformation that inserts the data into this table image

  3. Call the transformation in Airflow trans = CarteTransOperator( task_id='tst_trans_fail', pdi_conn_id='pentaho', trans='/public/test/tr_tst_failed_tab_constraint' )

    Result

  4. If table is empty (no constraints inconsistency) all works as expected

  5. If the table contains inserted data, I have found two possible scenarios (the second is more often): 2.1. The task finished with SUCCESS flag image image

    2.2. The task freezing, but the transformation already failed. image

Expected result

CarteTransOperator works properly

Additions

After discussion in Issue #29 : If we wrap around the transformation in job, it works fine. But I want to know if it is possible to solve this issue without this.

Environment

Airflow 2.5.1 + Python 3.10 + Pentaho server CE 8.0

evgdol commented 1 year ago

Not solved! image

piffall commented 1 year ago

Sorry. Now it should work.

evgdol commented 1 year ago

Nothing changed. The result is the same as on previous my screenshot.

evgdol commented 1 year ago

I have found the issue. You should change the condition in WHILE: image There are FINISHED_STATUSES used in your code. I have changed on my side and it works.

piffall commented 1 year ago

You are right @evgdol . I need to add some tests for this. But I'm about to release a version with this fix.