search

singer-io / tap-yotpo

GNU Affero General Public License v3.0
0 stars 7 forks source link

Unable to sync data #34

Closed TomaszE closed 2 years ago

TomaszE commented 2 years ago

Hi Folks,

for past few days we are not getting any data on this tap.

Error:

Errors during transform: [category.id: data does not match {'type': 'number'}, category.name: data does not match {'type': 'string'}, category: data does not match {'type': 'object', 'properties': {'id': {'type': 'number'}, 'name': {'type': 'string'}}, 'inclusion': 'available', 'selected': False}, : data does not match {'additionalProperties': False, 'type': ['object'], 'properties': {'url': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'product_specs': {'type': 'array', 'items': {'type': 'object', 'properties': {'key': {'type': 'string'}, 'value': {'type': 'string'}}}, 'inclusion': 'available', 'selected': False}, 'external_product_id': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'average_score': {'type': 'number', 'inclusion': 'available', 'selected': False}, 'category': {'type': 'object', 'properties': {'id': {'type': 'number'}, 'name': {'type': 'string'}}, 'inclusion': 'available', 'selected': False}, 'total_reviews': {'type': 'number', 'inclusion': 'available', 'selected': False}, 'id': {'type': 'integer', 'inclusion': 'automatic', 'selected': False}, 'created_at': {'format': 'date-time', 'type': 'string', 'inclusion': 'available', 'selected': False}, 'name': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'description': {'type': ['string', 'null'], 'inclusion': 'available', 'selected': False}, 'images': {'type': 'array', 'items': {'type': 'object', 'properties': {'facebook': {'type': 'string'}, 'original': {'type': 'string'}, 'facebook_square': {'type': 'string'}, 'kind': {'type': 'string'}, 'square': {'type': 'string'}}}, 'inclusion': 'available', 'selected': False}, 'updated_at': {'format': 'date-time', 'type': 'string', 'inclusion': 'available', 'selected': False}}, 'inclusion': 'available', 'selected': True}]". Target succeeded.

full log: 


2022-06-21 23:24:58,000Z   main - INFO Running tap-yotpo version 1.3.4 and target-stitch version 3.2.1
2022-06-21 23:24:59,123Z   main - INFO [smart-services] building smart service producer: topic( com.stitchdata.extractionJobStarted ), use_ssl( true )
2022-06-21 23:24:59,130Z   main - INFO [smart-services] building smart service producer: topic( com.stitchdata.extractionJobFinished ), use_ssl( true )
2022-06-21 23:24:59,136Z   main - INFO [smart-services] building smart service producer: topic( com.stitchdata.streamRecordCount ), use_ssl( true )
2022-06-21 23:25:00,208Z   main - INFO [smart-services] event successfully sent to kafka: com.stitchdata.extractionJobStarted [40] at offset None
2022-06-21 23:25:00,210Z   main - INFO Starting tap to discover schemas: tap-env/bin/tap-yotpo --config /tmp/tap_discover_config.json --discover
2022-06-21 23:25:00,586Z   main - INFO Tap exited normally.
2022-06-21 23:25:00,587Z   main - INFO Saving list of discovered streams
2022-06-21 23:25:00,617Z   main - INFO Saving structure of stream products (tap_stream_id: products)
2022-06-21 23:25:00,664Z   main - INFO Saving structure of stream unsubscribers (tap_stream_id: unsubscribers)
2022-06-21 23:25:00,679Z   main - INFO Saving structure of stream reviews (tap_stream_id: reviews)
2022-06-21 23:25:00,715Z   main - INFO Saving structure of stream emails (tap_stream_id: emails)
2022-06-21 23:25:00,736Z   main - INFO Saving structure of stream product_reviews (tap_stream_id: product_reviews)
2022-06-21 23:25:00,770Z   main - INFO Writing catalog to file
2022-06-21 23:25:00,775Z   main - INFO Selected fields for stream products: id
2022-06-21 23:25:00,775Z   main - INFO Selected fields for stream unsubscribers: id
2022-06-21 23:25:00,775Z   main - INFO Selected fields for stream reviews: id, created_at
2022-06-21 23:25:00,775Z   main - INFO Selected fields for stream emails: email_sent_timestamp, email_address
2022-06-21 23:25:00,780Z   main - INFO Current state: {"bookmarks": {"emails": {"since_date": "2022-06-13"}, "reviews": {"since_date": "2022-06-13"}, "products": {"since_date": "2016-01-01T00:00:00Z"}, "unsubscribers": {"since_date": "2016-01-01T00:00:00Z"}}, "currently_syncing": null}
2022-06-21 23:25:00,780Z   main - INFO Starting tap: tap-env/bin/tap-yotpo --config /tmp/tap_config.json --properties /tmp/properties.json --catalog /tmp/catalog.json --state /tmp/tap_state.json
2022-06-21 23:25:00,783Z   main - INFO Starting target: target-env/bin/target-stitch --config /tmp/target_config.json
2022-06-21 23:25:01,225Z target - INFO Using batch_size_prefernces of {'full_table_streams': [], 'batch_size_preference': None, 'user_batch_size_preference': None}
2022-06-21 23:25:02,506Z    tap - INFO HTTP request to "products" endpoint took 1.406s, returned status code 200
2022-06-21 23:25:02,508Z    tap - CRITICAL Errors during transform
2022-06-21 23:25:02,508Z    tap - CRITICAL  category.id: data does not match {'type': 'number'}
2022-06-21 23:25:02,508Z    tap - CRITICAL  category.name: data does not match {'type': 'string'}
2022-06-21 23:25:02,508Z    tap - CRITICAL  category: data does not match {'type': 'object', 'properties': {'id': {'type': 'number'}, 'name': {'type': 'string'}}, 'inclusion': 'available', 'selected': False}
2022-06-21 23:25:02,508Z    tap - CRITICAL  : data does not match {'additionalProperties': False, 'type': ['object'], 'properties': {'url': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'product_specs': {'type': 'array', 'items': {'type': 'object', 'properties': {'key': {'type': 'string'}, 'value': {'type': 'string'}}}, 'inclusion': 'available', 'selected': False}, 'external_product_id': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'average_score': {'type': 'number', 'inclusion': 'available', 'selected': False}, 'category': {'type': 'object', 'properties': {'id': {'type': 'number'}, 'name': {'type': 'string'}}, 'inclusion': 'available', 'selected': False}, 'total_reviews': {'type': 'number', 'inclusion': 'available', 'selected': False}, 'id': {'type': 'integer', 'inclusion': 'automatic', 'selected': False}, 'created_at': {'format': 'date-time', 'type': 'string', 'inclusion': 'available', 'selected': False}, 'name': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'description': {'type': ['string', 'null'], 'inclusion': 'available', 'selected': False}, 'images': {'type': 'array', 'items': {'type': 'object', 'properties': {'facebook': {'type': 'string'}, 'original': {'type': 'string'}, 'facebook_square': {'type': 'string'}, 'kind': {'type': 'string'}, 'square': {'type': 'string'}}}, 'inclusion': 'available', 'selected': False}, 'updated_at': {'format': 'date-time', 'type': 'string', 'inclusion': 'available', 'selected': False}}, 'inclusion': 'available', 'selected': True}
2022-06-21 23:25:02,508Z    tap - CRITICAL 
2022-06-21 23:25:02,508Z    tap - CRITICAL 
2022-06-21 23:25:02,508Z    tap - CRITICAL Errors during transform: [category.id: data does not match {'type': 'number'}, category.name: data does not match {'type': 'string'}, category: data does not match {'type': 'object', 'properties': {'id': {'type': 'number'}, 'name': {'type': 'string'}}, 'inclusion': 'available', 'selected': False}, : data does not match {'additionalProperties': False, 'type': ['object'], 'properties': {'url': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'product_specs': {'type': 'array', 'items': {'type': 'object', 'properties': {'key': {'type': 'string'}, 'value': {'type': 'string'}}}, 'inclusion': 'available', 'selected': False}, 'external_product_id': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'average_score': {'type': 'number', 'inclusion': 'available', 'selected': False}, 'category': {'type': 'object', 'properties': {'id': {'type': 'number'}, 'name': {'type': 'string'}}, 'inclusion': 'available', 'selected': False}, 'total_reviews': {'type': 'number', 'inclusion': 'available', 'selected': False}, 'id': {'type': 'integer', 'inclusion': 'automatic', 'selected': False}, 'created_at': {'format': 'date-time', 'type': 'string', 'inclusion': 'available', 'selected': False}, 'name': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'description': {'type': ['string', 'null'], 'inclusion': 'available', 'selected': False}, 'images': {'type': 'array', 'items': {'type': 'object', 'properties': {'facebook': {'type': 'string'}, 'original': {'type': 'string'}, 'facebook_square': {'type': 'string'}, 'kind': {'type': 'string'}, 'square': {'type': 'string'}}}, 'inclusion': 'available', 'selected': False}, 'updated_at': {'format': 'date-time', 'type': 'string', 'inclusion': 'available', 'selected': False}}, 'inclusion': 'available', 'selected': True}]
2022-06-21 23:25:02,509Z    tap - Traceback (most recent call last):
2022-06-21 23:25:02,509Z    tap -   File "tap-env/bin/tap-yotpo", line 33, in <module>
2022-06-21 23:25:02,509Z    tap -     sys.exit(load_entry_point('tap-yotpo==1.3.4', 'console_scripts', 'tap-yotpo')())
2022-06-21 23:25:02,509Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/utils.py", line 229, in wrapped
2022-06-21 23:25:02,509Z    tap -     return fnc(*args, **kwargs)
2022-06-21 23:25:02,509Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_yotpo/__init__.py", line 89, in main
2022-06-21 23:25:02,509Z    tap -     sync(ctx)
2022-06-21 23:25:02,509Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_yotpo/__init__.py", line 62, in sync
2022-06-21 23:25:02,509Z    tap -     streams_.products.fetch_into_cache(ctx)
2022-06-21 23:25:02,509Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_yotpo/streams.py", line 132, in fetch_into_cache
2022-06-21 23:25:02,509Z    tap -     self._sync(ctx)
2022-06-21 23:25:02,509Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/tap_yotpo/streams.py", line 95, in _sync
2022-06-21 23:25:02,509Z    tap -     transformed_record = transform(record, schema)
2022-06-21 23:25:02,509Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/transform.py", line 364, in transform
2022-06-21 23:25:02,509Z    tap -     return transformer.transform(data, schema, metadata=metadata)
2022-06-21 23:25:02,509Z    tap -   File "/code/orchestrator/tap-env/lib/python3.5/site-packages/singer/transform.py", line 153, in transform
2022-06-21 23:25:02,509Z    tap -     raise SchemaMismatch(self.errors)
2022-06-21 23:25:02,509Z    tap - singer.transform.SchemaMismatch: Errors during transform
2022-06-21 23:25:02,510Z    tap -   category.id: data does not match {'type': 'number'}
2022-06-21 23:25:02,510Z    tap -   category.name: data does not match {'type': 'string'}
2022-06-21 23:25:02,510Z    tap -   category: data does not match {'type': 'object', 'properties': {'id': {'type': 'number'}, 'name': {'type': 'string'}}, 'inclusion': 'available', 'selected': False}
2022-06-21 23:25:02,510Z    tap -   : data does not match {'additionalProperties': False, 'type': ['object'], 'properties': {'url': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'product_specs': {'type': 'array', 'items': {'type': 'object', 'properties': {'key': {'type': 'string'}, 'value': {'type': 'string'}}}, 'inclusion': 'available', 'selected': False}, 'external_product_id': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'average_score': {'type': 'number', 'inclusion': 'available', 'selected': False}, 'category': {'type': 'object', 'properties': {'id': {'type': 'number'}, 'name': {'type': 'string'}}, 'inclusion': 'available', 'selected': False}, 'total_reviews': {'type': 'number', 'inclusion': 'available', 'selected': False}, 'id': {'type': 'integer', 'inclusion': 'automatic', 'selected': False}, 'created_at': {'format': 'date-time', 'type': 'string', 'inclusion': 'available', 'selected': False}, 'name': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'description': {'type': ['string', 'null'], 'inclusion': 'available', 'selected': False}, 'images': {'type': 'array', 'items': {'type': 'object', 'properties': {'facebook': {'type': 'string'}, 'original': {'type': 'string'}, 'facebook_square': {'type': 'string'}, 'kind': {'type': 'string'}, 'square': {'type': 'string'}}}, 'inclusion': 'available', 'selected': False}, 'updated_at': {'format': 'date-time', 'type': 'string', 'inclusion': 'available', 'selected': False}}, 'inclusion': 'available', 'selected': True}
2022-06-21 23:25:02,510Z    tap - 
2022-06-21 23:25:02,510Z    tap - 
2022-06-21 23:25:02,510Z    tap - Errors during transform: [category.id: data does not match {'type': 'number'}, category.name: data does not match {'type': 'string'}, category: data does not match {'type': 'object', 'properties': {'id': {'type': 'number'}, 'name': {'type': 'string'}}, 'inclusion': 'available', 'selected': False}, : data does not match {'additionalProperties': False, 'type': ['object'], 'properties': {'url': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'product_specs': {'type': 'array', 'items': {'type': 'object', 'properties': {'key': {'type': 'string'}, 'value': {'type': 'string'}}}, 'inclusion': 'available', 'selected': False}, 'external_product_id': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'average_score': {'type': 'number', 'inclusion': 'available', 'selected': False}, 'category': {'type': 'object', 'properties': {'id': {'type': 'number'}, 'name': {'type': 'string'}}, 'inclusion': 'available', 'selected': False}, 'total_reviews': {'type': 'number', 'inclusion': 'available', 'selected': False}, 'id': {'type': 'integer', 'inclusion': 'automatic', 'selected': False}, 'created_at': {'format': 'date-time', 'type': 'string', 'inclusion': 'available', 'selected': False}, 'name': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'description': {'type': ['string', 'null'], 'inclusion': 'available', 'selected': False}, 'images': {'type': 'array', 'items': {'type': 'object', 'properties': {'facebook': {'type': 'string'}, 'original': {'type': 'string'}, 'facebook_square': {'type': 'string'}, 'kind': {'type': 'string'}, 'square': {'type': 'string'}}}, 'inclusion': 'available', 'selected': False}, 'updated_at': {'format': 'date-time', 'type': 'string', 'inclusion': 'available', 'selected': False}}, 'inclusion': 'available', 'selected': True}]
2022-06-21 23:25:02,543Z target - INFO Requests complete, stopping loop
2022-06-21 23:25:02,580Z   main - INFO Target exited normally with status 0
2022-06-21 23:25:03,705Z   main - INFO [smart-services] event successfully sent to kafka: com.stitchdata.extractionJobFinished [21] at offset None
2022-06-21 23:25:03,709Z   main - INFO No tunnel subprocess to tear down
2022-06-21 23:25:03,710Z   main - INFO Exit status is: Discovery succeeded. Tap failed with code 1 and error message: "Errors during transform
    category.id: data does not match {'type': 'number'}
    category.name: data does not match {'type': 'string'}
    category: data does not match {'type': 'object', 'properties': {'id': {'type': 'number'}, 'name': {'type': 'string'}}, 'inclusion': 'available', 'selected': False}
    : data does not match {'additionalProperties': False, 'type': ['object'], 'properties': {'url': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'product_specs': {'type': 'array', 'items': {'type': 'object', 'properties': {'key': {'type': 'string'}, 'value': {'type': 'string'}}}, 'inclusion': 'available', 'selected': False}, 'external_product_id': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'average_score': {'type': 'number', 'inclusion': 'available', 'selected': False}, 'category': {'type': 'object', 'properties': {'id': {'type': 'number'}, 'name': {'type': 'string'}}, 'inclusion': 'available', 'selected': False}, 'total_reviews': {'type': 'number', 'inclusion': 'available', 'selected': False}, 'id': {'type': 'integer', 'inclusion': 'automatic', 'selected': False}, 'created_at': {'format': 'date-time', 'type': 'string', 'inclusion': 'available', 'selected': False}, 'name': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'description': {'type': ['string', 'null'], 'inclusion': 'available', 'selected': False}, 'images': {'type': 'array', 'items': {'type': 'object', 'properties': {'facebook': {'type': 'string'}, 'original': {'type': 'string'}, 'facebook_square': {'type': 'string'}, 'kind': {'type': 'string'}, 'square': {'type': 'string'}}}, 'inclusion': 'available', 'selected': False}, 'updated_at': {'format': 'date-time', 'type': 'string', 'inclusion': 'available', 'selected': False}}, 'inclusion': 'available', 'selected': True}

Errors during transform: [category.id: data does not match {'type': 'number'}, category.name: data does not match {'type': 'string'}, category: data does not match {'type': 'object', 'properties': {'id': {'type': 'number'}, 'name': {'type': 'string'}}, 'inclusion': 'available', 'selected': False}, : data does not match {'additionalProperties': False, 'type': ['object'], 'properties': {'url': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'product_specs': {'type': 'array', 'items': {'type': 'object', 'properties': {'key': {'type': 'string'}, 'value': {'type': 'string'}}}, 'inclusion': 'available', 'selected': False}, 'external_product_id': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'average_score': {'type': 'number', 'inclusion': 'available', 'selected': False}, 'category': {'type': 'object', 'properties': {'id': {'type': 'number'}, 'name': {'type': 'string'}}, 'inclusion': 'available', 'selected': False}, 'total_reviews': {'type': 'number', 'inclusion': 'available', 'selected': False}, 'id': {'type': 'integer', 'inclusion': 'automatic', 'selected': False}, 'created_at': {'format': 'date-time', 'type': 'string', 'inclusion': 'available', 'selected': False}, 'name': {'type': 'string', 'inclusion': 'available', 'selected': False}, 'description': {'type': ['string', 'null'], 'inclusion': 'available', 'selected': False}, 'images': {'type': 'array', 'items': {'type': 'object', 'properties': {'facebook': {'type': 'string'}, 'original': {'type': 'string'}, 'facebook_square': {'type': 'string'}, 'kind': {'type': 'string'}, 'square': {'type': 'string'}}}, 'inclusion': 'available', 'selected': False}, 'updated_at': {'format': 'date-time', 'type': 'string', 'inclusion': 'available', 'selected': False}}, 'inclusion': 'available', 'selected': True}]". Target succeeded.
toandm commented 2 years ago

I have been getting the same error too. There's a PR related to this, but is blocked merging: https://github.com/singer-io/tap-yotpo/pull/33

sgandhi1311 commented 2 years ago

I have been getting the same error too. There's a PR related to this, but is blocked merging: #33

Merged the changes and bumped the version - https://github.com/singer-io/tap-yotpo/pull/35