search

singer-io / tap-pipedrive

A Singer.io tap for extracting data from the Pipedrive API
GNU Affero General Public License v3.0
13 stars 34 forks source link

Sync fails if a new user has been created, but hasn't logged in yet #17

Closed cayleyh closed 6 years ago

cayleyh commented 6 years ago

When a new user is created in Pipedrive, their last_login property is set to an empty timestamp of 0000-00-00 00:00:00. This causes the current tap to fail on a schema error of singer.transform.SchemaMismatch: Errors during transform last_login: 0000-00-00 00:00:00 does not match {'type': ['string', 'null'], 'format': 'date-time'}.

Can be reproduced by creating a new Pipedrive users and setting a password, but not logging in.

Full error log (minus details of the user json):

2018-04-16 21:48:51,061Z    tap - CRITICAL Errors during transform
2018-04-16 21:48:51,061Z    tap -   last_login: 0000-00-00 00:00:00 does not match {'type': ['string', 'null'], 'format': 'date-time'}
2018-04-16 21:48:51,061Z    tap -   : { ... 'last_login': '0000-00-00 00:00:00', ... }
2018-04-16 21:48:51,061Z    tap - 
2018-04-16 21:48:51,061Z    tap - 
2018-04-16 21:48:51,062Z    tap - Errors during transform: [last_login: 0000-00-00 00:00:00 does not match {'type': ['string', 'null'], 'format': 'date-time'}, : { ... 'last_login': '0000-00-00 00:00:00', ... }]
2018-04-16 21:48:51,070Z    tap - Traceback (most recent call last):
2018-04-16 21:48:51,070Z    tap -   File "tap-env/bin/tap-pipedrive", line 11, in <module>
2018-04-16 21:48:51,070Z    tap -     sys.exit(main())
2018-04-16 21:48:51,070Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_pipedrive/cli.py", line 30, in main
2018-04-16 21:48:51,071Z    tap -     raise e
2018-04-16 21:48:51,071Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_pipedrive/cli.py", line 27, in main
2018-04-16 21:48:51,071Z    tap -     main_impl()
2018-04-16 21:48:51,071Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_pipedrive/cli.py", line 22, in main_impl
2018-04-16 21:48:51,071Z    tap -     pipedrive_tap.do_sync()
2018-04-16 21:48:51,071Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_pipedrive/tap.py", line 94, in do_sync
2018-04-16 21:48:51,071Z    tap -     row = optimus_prime.transform(row, stream.get_schema())
2018-04-16 21:48:51,071Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/transform.py", line 94, in transform
2018-04-16 21:48:51,071Z    tap -     raise SchemaMismatch(self.errors)
2018-04-16 21:48:51,071Z    tap - singer.transform.SchemaMismatch: Errors during transform
2018-04-16 21:48:51,071Z    tap -   last_login: 0000-00-00 00:00:00 does not match {'type': ['string', 'null'], 'format': 'date-time'}
2018-04-16 21:48:51,071Z    tap -   : { ... 'last_login': '0000-00-00 00:00:00', ... }
2018-04-16 21:48:51,072Z    tap - 
2018-04-16 21:48:51,072Z    tap - 
2018-04-16 21:48:51,072Z    tap - Errors during transform: [last_login: 0000-00-00 00:00:00 does not match {'type': ['string', 'null'], 'format': 'date-time'}, : { ... 'last_login': '0000-00-00 00:00:00', ... }]
2018-04-16 21:48:51,125Z   main - DEBUG Saving state: {'currently_syncing': 'notes', 'bookmarks': {'activities': {'update_time': '2018-04-11T22:45:19+00:00'}, 'organizations': {'update_time': '2018-04-11T22:40:20+00:00'}, 'filters': {'update_time': '2018-04-02T18:24:44+00:00'}, 'products': {'update_time': '2017-03-27T00:00:00+00:00'}, 'pipelines': {'update_time': '2018-03-15T19:58:00+00:00'}, 'files': {'update_time': '2018-04-10T23:15:13+00:00'}, 'deals': {'update_time': '2018-04-11T22:40:20+00:00'}, 'activity_types': {'update_time': '2018-03-16T15:49:24+00:00'}, 'notes': {'update_time': '2018-04-16T20:27:48+00:00'}, 'stages': {'update_time': '2018-03-15T19:49:47+00:00'}, 'persons': {'update_time': '2018-04-11T22:40:20+00:00'}}}
2018-04-16 21:48:51,128Z target - INFO Sending batch with 14 messages for table users to https://api.stitchdata.com/v2/import/batch
2018-04-16 21:48:51,279Z target - INFO Exiting normally
2018-04-16 21:48:51,318Z   main - INFO Target exited normally with status 0
2018-04-16 21:48:51,319Z   main - INFO Exit status is: Tap failed with code 1 and error message: "Errors during transform". Target succeeded.
dmosorast commented 6 years ago

This should be fixed in version 0.0.11. Thanks!