search

AutoIDM / tap-clickup

tap-clickup , singer compliant tap for pulling clickup data
MIT License
12 stars 19 forks source link

Clarification of state capability #101

Closed JohannesRudolph closed 2 years ago

JohannesRudolph commented 2 years ago

The Readme lists the state capability as supported. However, when I'm trying to run meltano elt tap-clickup ... twice, the second invocation fails with an AttributeError: 'str' object has no attribute 'pop'.

The state file looks like this

{
  "bookmarks": {
    "folder": {
      "partitions": {
        "0": {
          "context": {
            "archived": "true",
            "space_id": "4520931"
          }
        },
        "1": {
          "context": {
            "archived": "false",
            "space_id": "4520931"
          },
          "starting_replication_value": null
        }
      }
    }

I've hacked in some log statements prefixed with XXXX to see what's going on here. It appears to me the incremental replication capability is actually broken, which is not a big problem as long as the tap does correctly document/declare that. Or am I getting something wrong here?

Here's the full log:


=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.876780Z [info     ] time=2021-12-29 21:10:47 name=tap-clickup level=INFO message=XXXX reset folder with state {'partitions': {'0': {'context': {'archived': 'true', 'space_id': '4520931'}}, '1': {'context': {'archived': 'false', 'space_id': '4520931'}, 'starting_replication_value': None}}} cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.876998Z [info     ] time=2021-12-29 21:10:47 name=tap-clickup level=INFO message=XXXX reset partition with state 0 cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.877235Z [info     ] Traceback (most recent call last): cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.877397Z [info     ]   File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/extractors/tap-clickup/venv/bin/tap-clickup", line 8, in <module> cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.877537Z [info     ]     sys.exit(TapClickUp.cli()) cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.877810Z [info     ]   File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/extractors/tap-clickup/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__ cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.877989Z [info     ]     return self.main(*args, **kwargs) cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.878123Z [info     ]   File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/extractors/tap-clickup/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.879121Z [info     ]     rv = self.invoke(ctx)      cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.879386Z [info     ]   File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/extractors/tap-clickup/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.879634Z [info     ]     return ctx.invoke(self.callback, **ctx.params) cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.879836Z [info     ]   File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/extractors/tap-clickup/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.880061Z [info     ]     return __callback(*args, **kwargs) cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.880931Z [info     ]   File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/extractors/tap-clickup/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 476, in cli cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.881243Z [info     ]     tap.sync_all()             cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.881493Z [info     ]   File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/extractors/tap-clickup/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 329, in sync_all cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.881706Z [info     ]     self._reset_state_progress_markers() cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.881846Z [info     ]   File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/extractors/tap-clickup/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 306, in _reset_state_progress_markers cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.881976Z [info     ]     _state.reset_state_progress_markers(partition_state) cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.882113Z [info     ]   File "/Users/jrudolph/dev/mc/meshbarn/tractor/.meltano/extractors/tap-clickup/venv/lib/python3.9/site-packages/singer_sdk/helpers/_state.py", line 170, in reset_state_progress_markers cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.883892Z [info     ]     progress_markers = stream_or_partition_state.pop(PROGRESS_MARKERS, {}) cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:47.884316Z [info     ] AttributeError: 'str' object has no attribute 'pop' cmd_type=extractor job_id=tap-clickup-to-bigquery name=tap-clickup run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:49.027603Z [info     ] INFO Pushing state: {}         cmd_type=loader job_id=tap-clickup-to-bigquery name=target-bigquery run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stderr
2021-12-29T20:10:49.028191Z [debug    ] {}                             cmd_type=loader job_id=tap-clickup-to-bigquery name=target-bigquery (out) run_id=23682466-c9ba-4ae6-bbfd-a1e5e76ed9cb stdio=stdout
2021-12-29T20:10:49.033728Z [info     ] Incremental state has been updated at 2021-12-29 20:10:49.033591.
2021-12-29T20:10:49.033940Z [debug    ] Incremental state: {}
visch commented 2 years ago

You're right, we originally did have State, but then removed it as the Partition with Child Streams fix we did here https://github.com/AutoIDM/tap-clickup/blob/main/tap_clickup/client.py#L114-L129 messed with context's in the different partitions.

Note that State only ever worked for Tasks as that's the only end point you can have state for anyway with clickup. We figured that due to how fast pulling all of the tasks are from the ClickUp api it isn't a huge deal, but we would love to add that back.

Yes we need to document this, this issue https://github.com/AutoIDM/tap-clickup/issues/95 tried to describe the problem but was a lack luster attempt (sorry!)