Return log results to inject to dagster ?

[slamer59]

Hello, I am using the rest API. tap_rest_api_msdk

Sometime this API returns a 404 error :

{
  "error": {
    "message": "Resource not found",
    "status_code": "404"
  }
}

The 404 is not a good response from the server but I cannot control it. If the dagster_op returns to logs we can imagine manipulate this for further dagster treatement.

Now, the pipeline fails.

I would return the _log_results of run_and_log so it becomes manageable.

Do you have another way to manage this use case ?

[slamer59]

https://github.com/quantile-development/dagster-meltano/blob/f9cfaebc2e4c046603eda65ba459b5004c2e69aa/dagster_meltano/ops.py#L70

[slamer59]

I look into dagster-shell. It returns the output of the command and errors.

https://github.com/dagster-io/dagster/blob/41d6c4d12c1b11fd140bb1749437f6ed71b971d2/python_modules/libraries/dagster-shell/dagster_shell/solids.py#L63

[JulesHuisman]

Yes, if you want you can return the log_results, this is currently a tuple with stdout and stderr.

In the future we can take a look at porting to using dagster-shell.