search

voxmedia / tap-dagster

Apache License 2.0
0 stars 1 forks source link

Fails when tap reaches the end of the stream #1

Open stkbailey opened 1 year ago

stkbailey commented 1 year ago

I'm running this tap via Meltano and getting an error when I reach the end of what I think is the run most recent set of records -- that is, the validate_response section is failing. code]

Traceback (most recent call last
  File \"/Users/stkbailey/Documents/github/dagster-workflows-meltano/.meltano/extractors/tap-dagster/venv/bin/tap-dagster\", line 8, in <module
    sys.exit(TapDagster.cli
  File \"/Users/stkbailey/Documents/github/dagster-workflows-meltano/.meltano/extractors/tap-dagster/venv/lib/python3.9/site-packages/click/core.py\", line 1130, in __call__
    return self.main(*args, **kwargs
  File \"/Users/stkbailey/Documents/github/dagster-workflows-meltano/.meltano/extractors/tap-dagster/venv/lib/python3.9/site-packages/click/core.py\", line 1055, in main
    rv = self.invoke(ctx
  File \"/Users/stkbailey/Documents/github/dagster-workflows-meltano/.meltano/extractors/tap-dagster/venv/lib/python3.9/site-packages/click/core.py\", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params
  File \"/Users/stkbailey/Documents/github/dagster-workflows-meltano/.meltano/extractors/tap-dagster/venv/lib/python3.9/site-packages/click/core.py\", line 760, in invoke
    return __callback(*args, **kwargs
  File \"/Users/stkbailey/Documents/github/dagster-workflows-meltano/.meltano/extractors/tap-dagster/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py\", line 501, in cli
    tap.sync_all
  File \"/Users/stkbailey/Documents/github/dagster-workflows-meltano/.meltano/extractors/tap-dagster/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py\", line 379, in sync_all
    stream.sync
  File \"/Users/stkbailey/Documents/github/dagster-workflows-meltano/.meltano/extractors/tap-dagster/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py\", line 1020, in sync
    self._sync_records(context
  File \"/Users/stkbailey/Documents/github/dagster-workflows-meltano/.meltano/extractors/tap-dagster/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py\", line 946, in _sync_records
    for record_result in self.get_records(current_context
  File \"/Users/stkbailey/Documents/github/dagster-workflows-meltano/.meltano/extractors/tap-dagster/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py\", line 424, in get_records
    for record in self.request_records(context
  File \"/Users/stkbailey/Documents/github/dagster-workflows-meltano/.meltano/extractors/tap-dagster/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py\", line 322, in request_records
    resp = decorated_request(prepared_request, context
  File \"/Users/stkbailey/Documents/github/dagster-workflows-meltano/.meltano/extractors/tap-dagster/venv/lib/python3.9/site-packages/backoff/_sync.py\", line 94, in retry
    ret = target(*args, **kwargs
  File \"/Users/stkbailey/Documents/github/dagster-workflows-meltano/.meltano/extractors/tap-dagster/venv/lib/python3.9/site-packages/singer_sdk/streams/rest.py\", line 235, in _request
    self.validate_response(response
  File \"/Users/stkbailey/Documents/github/dagster-workflows-meltano/.meltano/extractors/tap-dagster/venv/lib/python3.9/site-packages/tap_dagster/client.py\", line 55, in validate_response
    details = response.json().get(\"errors\", [])[0].get(\"message
IndexError: list index out of range

I wanted to flag this, and to check whether anyone is running this in production successfully now -- it could be that the Dagster Cloud graphql API is on a newer version than this is running against.

prratek commented 1 year ago

Hey @stkbailey, thanks for opening this issue! We haven't been running this tap in production for the last 2-3 months and may not have time to debug and fix ourselves but would love a PR if you're open to it. Seems like it would involve investigating what the error message looks like and returning something sensible in the error message. Possible causes: