Missing `run_results.json` from failed runs (versionless)

Is this a new bug in dbt-core?

[X] I believe this is a new bug in dbt-core
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Raising this issue on behalf of customer - they are reporting no run_results.json were generated for failed job runs on versionless. Downgrading to dbt version 1.7 does the issue with the missing artifact. Their job is invoking dbt --fail-fast run and the issue started on Thu, 03 Oct 2024 11:43:48 UTC. They are using dbt API's to query for the artifacts/run_results.json file for dbt runs. This is causing broken pipelines for the customer.

This is also causing the following error from dbt Cloud CLI as well: Error: Cloud CLI detected incompatible schemas. This issue is likely occurring because your core version is lower than the production job that provides the deferred manifest. Please consider upgrading your core version to match or exceed the version of the production job.

This occurrence is related to deferral within dbt Cloud CLI. For Cloud CLI - in order to let dev job recognize defer schema, the dev core version must >= prod run core version.

However, removing the --no-deferral flag does not resolve the issue with missing artifact; failed jobs on versionless are still missing the run_results.json.

Worth noting that manifest and logs were both present for failed runs (and logs indicate that execution finished with one model having a database error).

Expected Behavior

run_results.json should be uploaded and reviewable in artifacts

Steps To Reproduce

This has not been reproduced yet but this is occurring on CI jobs with and without deferral on versionless

Relevant log output

No response

Environment

- dbt: 2024.10.258
- snowflake=1.9.0-post8+eea98443b21f6f3bdc3d297b727a0a9424d54d7b

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

dbt-labs / dbt-core