Closed TomaszE closed 2 years ago
luandy64
commented
3 years ago 2021-05-12 11:15:39,137Z tap - : {'user_email': '07vmail@gmail.com', 'email_type_id': 15, 'unsubscirbed_by_name': 'USER', 'id': ''}The API is handing back null "id" fields and the tap marks "id" as the primary key for the Unsubscribers stream
TomaszE
commented
3 years ago thanks for your reply. Unfortunatelly i have no technicall knowledge here , i am just an user who use the Stitch integration. Is that something that need to be handled in Python? Another words are there any changes required in currect integration code? Or something is that something that end user can do?
dave-connors-3
commented
3 years ago @luandy64 - Any advice on handling this? will the tap need to be adjusted to handle null ids? I have a client who is also experiencing this issue
bshichman
commented
3 years ago @TomaszE @luandy64 @dave-connors-3 We are experiencing the exact same issue over here with this. Does someone on this thread know how to update the tap?
elkelk
commented
3 years ago This is breaking the extraction for me as well.
JDW818
commented
3 years ago @cosimon @KAllan357 can either of you help here? I saw that y'all helped with another Singer PR that had been sitting for a while. This is a client-facing issue for myself and @dave-connors-3, so any help would be much appreciated!
lluisgassovillarejo
commented
3 years ago Hi all,
after downloading the repo and trying to solved the issue, I found the code added to line 90 on the streams.py file.
Looks like this was introduced in the Tdl 13499 fix transform error in unsubscribers stream (#15) commit.
Should this issue be closed? The code is working on our Stitch instance.
Thanks
sgandhi1311
commented
2 years ago
Hi Guys,
We got a message from Yotpo that due to incident they had to update all the API secrets - after the update we are able to connect correctly bot we are receiving error :
Errors during transform
full log:
2021-05-12 11:15:35,116Z main - INFO Running tap-yotpo version 1.2.0 and target-stitch version 3.2.0 2021-05-12 11:15:35,270Z main - INFO [smart-services] building smart service producer: topic( com.stitchdata.extractionJobStarted ), use_ssl( true ) 2021-05-12 11:15:35,274Z main - INFO [smart-services] building smart service producer: topic( com.stitchdata.extractionJobFinished ), use_ssl( true ) 2021-05-12 11:15:35,276Z main - INFO [smart-services] building smart service producer: topic( com.stitchdata.streamRecordCount ), use_ssl( true ) 2021-05-12 11:15:36,289Z main - INFO [smart-services] event successfully sent to kafka: com.stitchdata.extractionJobStarted [18] at offset None 2021-05-12 11:15:36,292Z main - INFO Starting tap to discover schemas: tap-env/bin/tap-yotpo --config /tmp/tap_discover_config.json --discover 2021-05-12 11:15:36,684Z main - INFO Tap exited normally. 2021-05-12 11:15:36,685Z main - INFO Saving list of discovered streams 2021-05-12 11:15:36,707Z main - INFO Saving structure of stream products (tap_stream_id: products) 2021-05-12 11:15:36,724Z main - INFO Saving structure of stream unsubscribers (tap_stream_id: unsubscribers) 2021-05-12 11:15:36,743Z main - INFO Saving structure of stream reviews (tap_stream_id: reviews) 2021-05-12 11:15:36,764Z main - INFO Saving structure of stream emails (tap_stream_id: emails) 2021-05-12 11:15:36,786Z main - INFO Saving structure of stream product_reviews (tap_stream_id: product_reviews) 2021-05-12 11:15:36,804Z main - INFO Writing catalog to file 2021-05-12 11:15:36,810Z main - INFO Selected fields for stream unsubscribers: unsubscirbed_by_name, email_type_id, id, user_email 2021-05-12 11:15:36,810Z main - INFO Selected fields for stream reviews: id, user_reference, title, updated_at, sentiment, deleted, name, reviewer_type, email, votes_down, created_at, score, content, sku, votes_up 2021-05-12 11:15:36,810Z main - INFO Selected fields for stream products: total_reviews, url, id, product_specs, updated_at, images, name, category, average_score, created_at, external_product_id, description 2021-05-12 11:15:36,810Z main - INFO Selected fields for stream emails: failed_timestamp, unsubscribed_timestamp, invalid_address_timestamp, order_id, opened_timestamp, platform, review_form, arrived_early_timestamp, coupon_code, trr_bundle_id, marked_spam_timestamp, reminder_num, content_creation_timestamp, trr_bundle_subject, product_id, email_sent_timestamp, order_timestamp, clicked_through_timestamp, email_type, sku, review_type, email_address 2021-05-12 11:15:36,815Z main - INFO Current state: {"bookmarks": {"unsubscribers": {"since_date": "2016-01-01T00:00:00Z"}, "reviews": {"since_date": "2021-05-07"}, "products": {"since_date": "2016-01-01T00:00:00Z"}, "emails": {"since_date": "2021-05-08"}}, "currently_syncing": "unsubscribers"} 2021-05-12 11:15:36,815Z main - INFO Starting tap: tap-env/bin/tap-yotpo --config /tmp/tap_config.json --properties /tmp/properties.json --catalog /tmp/catalog.json --state /tmp/tap_state.json 2021-05-12 11:15:36,819Z main - INFO Starting target: target-env/bin/target-stitch --config /tmp/target_config.json 2021-05-12 11:15:37,273Z target - INFO Using batch_size_prefernces of {'full_table_streams': [], 'user_batch_size_preference': None, 'batch_size_preference': None} 2021-05-12 11:15:38,559Z tap - INFO HTTP request to "products" endpoint took 1.399s, returned status code 200 2021-05-12 11:15:38,995Z tap - INFO HTTP request to "products" endpoint took 0.385s, returned status code 200 2021-05-12 11:15:39,071Z tap - INFO HTTP request to "products" endpoint took 0.064s, returned status code 200 2021-05-12 11:15:39,072Z main - INFO State update: adding bookmarks.emails.since_date = "2021-05-08" 2021-05-12 11:15:39,072Z main - INFO State update: adding bookmarks.products.since_date = "2016-01-01T00:00:00Z" 2021-05-12 11:15:39,072Z main - INFO State update: adding bookmarks.reviews.since_date = "2021-05-07" 2021-05-12 11:15:39,072Z main - INFO State update: adding bookmarks.unsubscribers.since_date = "2016-01-01T00:00:00Z" 2021-05-12 11:15:39,072Z main - INFO State update: adding currently_syncing = "unsubscribers" 2021-05-12 11:15:39,135Z tap - INFO HTTP request to "unsubscribers" endpoint took 0.063s, returned status code 200 2021-05-12 11:15:39,135Z tap - CRITICAL Errors during transform 2021-05-12 11:15:39,135Z tap - id: does not match {'inclusion': 'automatic', 'type': 'integer'} 2021-05-12 11:15:39,135Z tap - : {'user_email': '07vmail@gmail.com', 'email_type_id': 15, 'unsubscirbed_by_name': 'USER', 'id': ''} does not match {'additionalProperties': False, 'properties': {'user_email': {'inclusion': 'automatic', 'type': 'string'}, 'email_type_id': {'inclusion': 'automatic', 'type': 'number'}, 'unsubscirbed_by_name': {'description': 'This field is misspelled in the api response!', 'inclusion': 'automatic', 'type': 'string'}, 'id': {'inclusion': 'automatic', 'type': 'integer'}}, 'type': ['object'], 'inclusion': 'available', 'selected': True} 2021-05-12 11:15:39,135Z tap - 2021-05-12 11:15:39,135Z tap - 2021-05-12 11:15:39,136Z tap - Errors during transform: [id: does not match {'inclusion': 'automatic', 'type': 'integer'}, : {'user_email': '07vmail@gmail.com', 'email_type_id': 15, 'unsubscirbed_by_name': 'USER', 'id': ''} does not match {'additionalProperties': False, 'properties': {'user_email': {'inclusion': 'automatic', 'type': 'string'}, 'email_type_id': {'inclusion': 'automatic', 'type': 'number'}, 'unsubscirbed_by_name': {'description': 'This field is misspelled in the api response!', 'inclusion': 'automatic', 'type': 'string'}, 'id': {'inclusion': 'automatic', 'type': 'integer'}}, 'type': ['object'], 'inclusion': 'available', 'selected': True}] 2021-05-12 11:15:39,136Z tap - Traceback (most recent call last): 2021-05-12 11:15:39,136Z tap - File "tap-env/bin/tap-yotpo", line 33, in
2021-05-12 11:15:39,136Z tap - sys.exit(load_entry_point('tap-yotpo==1.2.0', 'console_scripts', 'tap-yotpo')())
2021-05-12 11:15:39,136Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_yotpo/init.py", line 91, in main
2021-05-12 11:15:39,136Z tap - main_impl()
2021-05-12 11:15:39,136Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_yotpo/init.py", line 86, in main_impl
2021-05-12 11:15:39,136Z tap - sync(ctx)
2021-05-12 11:15:39,136Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_yotpo/init.py", line 71, in sync
2021-05-12 11:15:39,136Z tap - stream.sync(ctx)
2021-05-12 11:15:39,136Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_yotpo/streams.py", line 96, in sync
2021-05-12 11:15:39,137Z tap - self._sync(ctx)
2021-05-12 11:15:39,137Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_yotpo/streams.py", line 86, in _sync
2021-05-12 11:15:39,137Z tap - records = [transform(record, schema) for record in raw_records]
2021-05-12 11:15:39,137Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_yotpo/streams.py", line 86, in
2021-05-12 11:15:39,137Z tap - records = [transform(record, schema) for record in raw_records]
2021-05-12 11:15:39,137Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/transform.py", line 324, in transform
2021-05-12 11:15:39,137Z tap - return transformer.transform(data, schema, metadata=metadata)
2021-05-12 11:15:39,137Z tap - File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/transform.py", line 131, in transform
2021-05-12 11:15:39,137Z tap - raise SchemaMismatch(self.errors)
2021-05-12 11:15:39,137Z tap - singer.transform.SchemaMismatch: Errors during transform
2021-05-12 11:15:39,137Z tap - id: does not match {'inclusion': 'automatic', 'type': 'integer'}
2021-05-12 11:15:39,137Z tap - : {'user_email': '07vmail@gmail.com', 'email_type_id': 15, 'unsubscirbed_by_name': 'USER', 'id': ''} does not match {'additionalProperties': False, 'properties': {'user_email': {'inclusion': 'automatic', 'type': 'string'}, 'email_type_id': {'inclusion': 'automatic', 'type': 'number'}, 'unsubscirbed_by_name': {'description': 'This field is misspelled in the api response!', 'inclusion': 'automatic', 'type': 'string'}, 'id': {'inclusion': 'automatic', 'type': 'integer'}}, 'type': ['object'], 'inclusion': 'available', 'selected': True}
2021-05-12 11:15:39,137Z tap -
2021-05-12 11:15:39,137Z tap -
2021-05-12 11:15:39,137Z tap - Errors during transform: [id: does not match {'inclusion': 'automatic', 'type': 'integer'}, : {'user_email': '07vmail@gmail.com', 'email_type_id': 15, 'unsubscirbed_by_name': 'USER', 'id': ''} does not match {'additionalProperties': False, 'properties': {'user_email': {'inclusion': 'automatic', 'type': 'string'}, 'email_type_id': {'inclusion': 'automatic', 'type': 'number'}, 'unsubscirbed_by_name': {'description': 'This field is misspelled in the api response!', 'inclusion': 'automatic', 'type': 'string'}, 'id': {'inclusion': 'automatic', 'type': 'integer'}}, 'type': ['object'], 'inclusion': 'available', 'selected': True}]
2021-05-12 11:15:39,179Z target - INFO Requests complete, stopping loop
2021-05-12 11:15:39,222Z main - INFO Target exited normally with status 0
2021-05-12 11:15:39,283Z main - INFO [smart-services] event successfully sent to kafka: com.stitchdata.extractionJobFinished [35] at offset None
2021-05-12 11:15:39,284Z main - INFO No tunnel subprocess to tear down
2021-05-12 11:15:39,284Z main - INFO Exit status is: Discovery succeeded. Tap failed with code 1 and error message: "Errors during transform". Target succeeded.