Connector Test Failure reported as success on failure

[bnchrch]

Problem

source-firebase-realtime-database was reported as passing in the github comment when it had failed https://github.com/airbytehq/airbyte/pull/27420

[evantahler]

Grooming:

If there's a dagger engine error (like the above), this is a failure. The logic to check if the report was good or not is likely needs to be inverted.

[bnchrch]

Grooming:

• this is related to @marcosmarxm having a passing test report but failing unit tests: https://airbytehq-team.slack.com/archives/C03VDJ4FMJB/p1689628075772669
• the pytest success/failure reporting is a bit brittle because it relies on stdout parsing and not exit code: if "failed" in last_log_line or "errors" in last_log_line: If the stdout structure changes we can have unexpected results.
• This stdout parsing was originally introduced because Dagger raised errors without details when exit code > 0. We made pytest always exit with 0 and rely on stdout parsing to assess success or failure. This will be solved by upgrading to the latest dagger version, if a with_exec returns a non 0 exit code the error object now has stdout and stderr attribute.

Also likely a related issue: https://github.com/airbytehq/airbyte/issues/27770

[alafanechere]

For the false positive with test results: we need to change how we parse CAT output and transform it to StepResult: replace the use of pytest_logs_to_step_result with get_step_result . For the false positive without test result: to reproduce you should raise a an unhandled exception inside the pipeline, and I think the report wrongly states a success.

[bnchrch]

Will update reporting structure in https://app.zenhub.com/workspaces/connector-operations-6318ede5eda32e0010ef8566/issues/gh/airbytehq/airbyte/27729

[alafanechere]

For the false positive with test results: we need to change how we parse CAT output and transform it to StepResult: replace the use of pytest_logs_to_step_result with get_step_result . For the false positive without test result: to reproduce you should raise a an unhandled exception inside the pipeline, and I think the report wrongly states a success.

This is addressed in #28767