AutoIDM / tap-clickup

tap-clickup , singer compliant tap for pulling clickup data
MIT License
12 stars 19 forks source link

start_date implementation #118

Open visch opened 2 years ago

visch commented 2 years ago

Currently

    16     def _write_starting_replication_value(self, context: Optional[dict]) -> None:
    15         """Write the starting replication value, if available.
    14
    13         Args:
    12             context: Stream partition or context dictionary.
    11         """
    10         value = None
     9         state = self.get_context_state(context)
     8
     7         if self.replication_key:
     6             replication_key_value = state.get("replication_key_value")
     5             if replication_key_value and self.replication_key == state.get(
     4                 "replication_key"
     3             ):
     2                 value = replication_key_value
     1
  321              elif "start_date" in self.config:
     1                 value = self.config["start_date"]
     2
     3         write_starting_replication_value(state, value)

From the SDK brings in start_date, which means techincally if you pass in a unix timstamp for start_date it will work. We don't advertise this as the "standard" for start_date is to use an RFC3399 datetime type

Options

  1. We could just override start_date's value with the timestamp value. right at the initialization of the tap.
  2. We could add special handling of get_starting_timestamp like https://github.com/AutoIDM/tap-clickup/pull/117/files#diff-0b811eae9421519f5a8fc873c70bd3e821c57d9e584c88de748d635e4808d1c5R279 , but we'd have to figure out when the initial state is coming from start_date and when it's coming from state as start_date is RFC3399 and state is storing the value as a unix timestamp
  3. Convert state unix time stamps to use RFC3399 values, and convert these values to timestamps when needed like here https://github.com/AutoIDM/tap-clickup/pull/117/files#diff-0b811eae9421519f5a8fc873c70bd3e821c57d9e584c88de748d635e4808d1c5R252
  4. Probably others?

Instead of figuring this out right now we're just not going to officially support start_date.