search

MeltanoLabs / tap-gitlab

Singer.io Tap for extracting data from Gitlab's API
GNU Affero General Public License v3.0
11 stars 27 forks source link

Start Date format YYYY-MM-DD throwing TypeError: can't compare offset-naive and offset-aware datetimes #44

Closed pnadolny13 closed 2 years ago

pnadolny13 commented 2 years ago

In GitLab by @bwise on Dec 21, 2020, 18:42

A start date format of 'YYYY-MM-DD' throws errors on ELT runs with state. The first run will run fine, but follow on runs will error out with tap-gitlab | CRITICAL can't compare offset-naive and offset-aware datetimes

YYYY-MM-DD start_dates should either be supported fully, or explicitly raise an error saying they're not.

The errors are caused by a timezone mismatch.

Slack discussion on topic: https://meltano.slack.com/archives/C013EKWA2Q1/p1608586206289200

pnadolny13 commented 2 years ago

In GitLab by @bwise on Dec 21, 2020, 18:44

Work around: To fix this, you cannot just change the start date. You must change the start date and then do a full refresh to the get the time zone set properly on all the status.

  1. Set start date to '2019-07-01T00:00:00Z'
  2. Do a full refresh
  3. Do an incremental load
pnadolny13 commented 2 years ago

In GitLab by @DouweM on Dec 21, 2020, 19:13

changed title from Start Date format YYY-MM-DD throwing TypeError: can't compare offset-naive and offset-aware datetimes to Start Date format YYY{+Y+}-MM-DD throwing TypeError: can't compare offset-naive and offset-aware datetimes

pnadolny13 commented 2 years ago

In GitLab by @DouweM on Dec 21, 2020, 19:15

Alternatively:

  1. Add timezone to start date: meltano config tap-gitlab set start_date '<YYYY-MM-DD>T00:00:00Z'
  2. Dump state: meltano schedule run <job_id> --dump=state > state.json
  3. Manually modify state.json and add Z to the end of every timestamp that doesn't already end in Z or +HH:SS
  4. Run incremental load with fixed state: meltano schedule run <job_id> --state=state.json. This should work now!
  5. Dump new state to verify all timestamps have timezones now: meltano schedule run <job_id> --dump=state
pnadolny13 commented 2 years ago

In GitLab by @DouweM on Dec 23, 2020, 13:41

This is a regression introduced by https://gitlab.com/meltano/tap-gitlab/-/merge_requests/28.

pnadolny13 commented 2 years ago

In GitLab by @DouweM on Dec 23, 2020, 13:41

assigned to @DouweM

pnadolny13 commented 2 years ago

In GitLab by @DouweM on Dec 23, 2020, 17:08

mentioned in merge request !30

pnadolny13 commented 2 years ago

In GitLab by @DouweM on Dec 23, 2020, 17:09

mentioned in commit adffdc484b396f07e8299404639645315cb6a99f

pnadolny13 commented 2 years ago

In GitLab by @DouweM on Dec 23, 2020, 17:11

This has been fixed in v0.9.11! Run meltano install extractor tap-gitlab to install the latest version (assuming you didn't change pip_url to pin a specific version).