search

sehnem / tap-shopify

Apache License 2.0
5 stars 3 forks source link

Possibly missing some configuration option #9

Open konung opened 10 months ago

konung commented 10 months ago

Hi .

I've been testing meltano ( a total meltano noob here) with this tap-shopify variant, and keep hitting all kinds of errors when trying to run actual el/run .

I know it's not a credentials issue:

  1. I'm able to run the sync job using old stitch tap-shopify & my target-postgres.
  2. I'm also able to pass data from tap-spreadsheets-anywhere to target-postgres for other data.
  3. meltano config tap-shopify test runs green
  4. I can run meltano invoke tap-shopify --discover > ./shopify_catalog.json
  5. meltano invoke tap-shopify --catalog=./shopify_catalog.json - I can see actual live data pulled form my store.

I seem to be missing some crucial step in tap-shopify setup, that's probably obvious to a more experienced Meltano user. For example: do I need to somehow point my meltano el tap-shopify target-postgres to use --catalog=shopify_catalog.json or possibly do something else?

konung commented 10 months ago

Here is my log when running meltano el tap-shopify target-jsonl

2023-09-26T08:27:50.970629Z [info     ] Running extract & load...      name=meltano run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl
2023-09-26T08:27:51.584625Z [warning  ] No state was found, complete import.
2023-09-26T08:28:18.406789Z [info     ] 2023-09-26 03:28:18,406 | INFO     | tap-shopify          | Skipping deselected stream 'automatic_discount_nodes'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.407255Z [info     ] 2023-09-26 03:28:18,406 | INFO     | tap-shopify          | Skipping deselected stream 'catalogs'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.407624Z [info     ] 2023-09-26 03:28:18,406 | INFO     | tap-shopify          | Skipping deselected stream 'checkout_profiles'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.407851Z [info     ] 2023-09-26 03:28:18,406 | INFO     | tap-shopify          | Skipping deselected stream 'code_discount_nodes'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.408037Z [info     ] 2023-09-26 03:28:18,406 | INFO     | tap-shopify          | Skipping deselected stream 'collections'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.408140Z [info     ] 2023-09-26 03:28:18,407 | INFO     | tap-shopify          | Skipping deselected stream 'companies'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.408355Z [info     ] 2023-09-26 03:28:18,407 | INFO     | tap-shopify          | Skipping deselected stream 'company_locations'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.408462Z [info     ] 2023-09-26 03:28:18,408 | INFO     | tap-shopify          | Skipping deselected stream 'customers'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.408558Z [info     ] 2023-09-26 03:28:18,408 | INFO     | tap-shopify          | Skipping deselected stream 'deletion_events'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.408732Z [info     ] 2023-09-26 03:28:18,408 | INFO     | tap-shopify          | Skipping deselected stream 'delivery_customizations'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.408830Z [info     ] 2023-09-26 03:28:18,408 | INFO     | tap-shopify          | Skipping deselected stream 'discount_nodes'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.408922Z [info     ] 2023-09-26 03:28:18,408 | INFO     | tap-shopify          | Skipping deselected stream 'discount_redeem_code_saved_searches'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.409073Z [info     ] 2023-09-26 03:28:18,408 | INFO     | tap-shopify          | Skipping deselected stream 'draft_orders'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.409171Z [info     ] 2023-09-26 03:28:18,408 | INFO     | tap-shopify          | Skipping deselected stream 'files'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.409408Z [info     ] 2023-09-26 03:28:18,409 | INFO     | tap-shopify          | Skipping deselected stream 'fulfillment_orders'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.409562Z [info     ] 2023-09-26 03:28:18,409 | INFO     | tap-shopify          | Skipping deselected stream 'gift_cards'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.409659Z [info     ] 2023-09-26 03:28:18,409 | INFO     | tap-shopify          | Skipping deselected stream 'inventory_items'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.409749Z [info     ] 2023-09-26 03:28:18,409 | INFO     | tap-shopify          | Skipping deselected stream 'locations'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.409855Z [info     ] 2023-09-26 03:28:18,409 | INFO     | tap-shopify          | Skipping deselected stream 'manual_holds_fulfillment_orders'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.409994Z [info     ] 2023-09-26 03:28:18,409 | INFO     | tap-shopify          | Skipping deselected stream 'marketing_activities'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.410125Z [info     ] 2023-09-26 03:28:18,409 | INFO     | tap-shopify          | Skipping deselected stream 'marketing_events'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.410265Z [info     ] 2023-09-26 03:28:18,410 | INFO     | tap-shopify          | Skipping deselected stream 'metafield_definitions'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.410433Z [info     ] 2023-09-26 03:28:18,410 | INFO     | tap-shopify          | Skipping deselected stream 'metaobjects'. cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.412148Z [info     ] 2023-09-26 03:28:18,412 | INFO     | tap-shopify          | Beginning incremental sync of 'orders'... cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:18.412303Z [info     ] 2023-09-26 03:28:18,412 | INFO     | tap-shopify          | Tap has custom mapper. Using 1 provided map(s). cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.155387Z [info     ] 2023-09-26 03:28:19,155 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "http_request_count", "value": 0, "tags": {"stream": "orders", "endpoint": ""}} cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.155659Z [info     ] 2023-09-26 03:28:19,155 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "sync_duration", "value": 0.40765810012817383, "tags": {"stream": "orders", "context": {}, "status": "failed"}} cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.155922Z [info     ] 2023-09-26 03:28:19,155 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 0, "tags": {"stream": "orders", "context": {}}} cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.156254Z [info     ] 2023-09-26 03:28:19,155 | ERROR    | tap-shopify          | An unhandled error occurred while syncing 'orders' cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.156431Z [info     ] Traceback (most recent call last): cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.156585Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 1175, in sync cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.156743Z [info     ]     for _ in self._sync_records(context=context): cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.156877Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 1070, in _sync_records cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.157016Z [info     ]     for record_result in self.get_records(current_context): cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.157147Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/rest.py", line 576, in get_records cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.157279Z [info     ]     for record in self.request_records(context): cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.157407Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/rest.py", line 393, in request_records cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.157543Z [info     ]     prepared_request = self.prepare_request( cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.157672Z [info     ]                        ^^^^^^^^^^^^^^^^^^^^^ cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.157800Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/rest.py", line 364, in prepare_request cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.157951Z [info     ]     request_data = self.prepare_request_payload(context, next_page_token) cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.158086Z [info     ]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.158213Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/tap_shopify/client.py", line 365, in prepare_request_payload cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.158342Z [info     ]     query = self.query.lstrip() cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.158470Z [info     ]             ^^^^^^^^^^         cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.158597Z [info     ]   File "/usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/functools.py", line 1001, in __get__ cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.158724Z [info     ]     val = self.func(instance)  cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.158851Z [info     ]           ^^^^^^^^^^^^^^^^^^^  cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.158977Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/tap_shopify/client.py", line 497, in query cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.159106Z [info     ]     self.evaluate_query()      cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.159235Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/tap_shopify/client.py", line 521, in evaluate_query cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.159366Z [info     ]     error_code = error.get("extensions", {}).get("code") cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.159500Z [info     ]                  ^^^^^^^^^     cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.159632Z [info     ] AttributeError: 'str' object has no attribute 'get' cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.159839Z [info     ] Traceback (most recent call last): cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.160145Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/bin/tap-shopify", line 8, in <module> cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.160287Z [info     ]     sys.exit(TapShopify.cli()) cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.160426Z [info     ]              ^^^^^^^^^^^^^^^^  cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.160555Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.160688Z [info     ]     return self.main(*args, **kwargs) cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.160819Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.160945Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/click/core.py", line 1078, in main cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.161079Z [info     ]     rv = self.invoke(ctx)      cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.161207Z [info     ]          ^^^^^^^^^^^^^^^^      cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.161334Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.161465Z [info     ]     return ctx.invoke(self.callback, **ctx.params) cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.161592Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.161718Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.161849Z [info     ]     return __callback(*args, **kwargs) cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.161978Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.162124Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 498, in invoke cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.162258Z [info     ]     tap.sync_all()             cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.162385Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 456, in sync_all cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.162515Z [info     ]     stream.sync()              cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.162641Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 1182, in sync cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.162770Z [info     ]     raise ex                   cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.162898Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 1175, in sync cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.163109Z [info     ]     for _ in self._sync_records(context=context): cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.163277Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 1070, in _sync_records cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.163472Z [info     ]     for record_result in self.get_records(current_context): cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.163612Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/rest.py", line 576, in get_records cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.163750Z [info     ]     for record in self.request_records(context): cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.163884Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/rest.py", line 393, in request_records cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.164043Z [info     ]     prepared_request = self.prepare_request( cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.164177Z [info     ]                        ^^^^^^^^^^^^^^^^^^^^^ cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.164308Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/rest.py", line 364, in prepare_request cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.165142Z [info     ]     request_data = self.prepare_request_payload(context, next_page_token) cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.165310Z [info     ]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.165454Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/tap_shopify/client.py", line 365, in prepare_request_payload cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.165594Z [info     ]     query = self.query.lstrip() cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.165730Z [info     ]             ^^^^^^^^^^         cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.165864Z [info     ]   File "/usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/functools.py", line 1001, in __get__ cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.165996Z [info     ]     val = self.func(instance)  cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.166125Z [info     ]           ^^^^^^^^^^^^^^^^^^^  cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.166260Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/tap_shopify/client.py", line 497, in query cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.166508Z [info     ]     self.evaluate_query()      cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.166652Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/tap_shopify/client.py", line 521, in evaluate_query cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.166788Z [info     ]     error_code = error.get("extensions", {}).get("code") cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.166946Z [info     ]                  ^^^^^^^^^     cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:19.167079Z [info     ] AttributeError: 'str' object has no attribute 'get' cmd_type=extractor name=tap-shopify run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:20.086857Z [info     ] Incremental state has been updated at 2023-09-26 08:28:20.086689.
2023-09-26T08:28:20.095897Z [error    ] Extraction failed              code=1 message=AttributeError: 'str' object has no attribute 'get' name=meltano run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl
2023-09-26T08:28:20.096407Z [info     ] ELT could not be completed: Extractor failed. cmd_type=elt name=meltano run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:20.096562Z [info     ] For more detailed log messages re-run the command using 'meltano --log-level=debug ...' CLI flag. cmd_type=elt name=meltano run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:20.096698Z [info     ] Note that you can also check the generated log file at '/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/logs/elt/2023-09-26T082750--tap-shopify--target-jsonl/e9709048-99fb-4aa8-8c6c-6ae1bc2d011a/elt.log'. cmd_type=elt name=meltano run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
2023-09-26T08:28:20.096829Z [info     ] For more information on debugging and logging: https://docs.meltano.com/reference/command-line-interface#debugging cmd_type=elt name=meltano run_id=e9709048-99fb-4aa8-8c6c-6ae1bc2d011a state_id=2023-09-26T082750--tap-shopify--target-jsonl stdio=stderr
konung commented 10 months ago

And here is me trying to run meltano el tap-shopify target-postgres

2023-09-26T08:37:13.329392Z [info     ] Running extract & load...      name=meltano run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres
2023-09-26T08:37:14.090548Z [warning  ] No state was found, complete import.
2023-09-26T08:37:32.122614Z [info     ] 2023-09-26 03:37:32,122 | INFO     | target-postgres      | Target 'target-postgres' is listening for input from tap. cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.920422Z [info     ] 2023-09-26 03:37:42,920 | INFO     | tap-shopify          | Skipping deselected stream 'automatic_discount_nodes'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.920817Z [info     ] 2023-09-26 03:37:42,920 | INFO     | tap-shopify          | Skipping deselected stream 'catalogs'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.921016Z [info     ] 2023-09-26 03:37:42,920 | INFO     | tap-shopify          | Skipping deselected stream 'checkout_profiles'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.921172Z [info     ] 2023-09-26 03:37:42,920 | INFO     | tap-shopify          | Skipping deselected stream 'code_discount_nodes'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.921456Z [info     ] 2023-09-26 03:37:42,920 | INFO     | tap-shopify          | Skipping deselected stream 'collections'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.921672Z [info     ] 2023-09-26 03:37:42,920 | INFO     | tap-shopify          | Skipping deselected stream 'companies'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.922286Z [info     ] 2023-09-26 03:37:42,921 | INFO     | tap-shopify          | Skipping deselected stream 'company_locations'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.922427Z [info     ] 2023-09-26 03:37:42,921 | INFO     | tap-shopify          | Skipping deselected stream 'customers'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.922583Z [info     ] 2023-09-26 03:37:42,921 | INFO     | tap-shopify          | Skipping deselected stream 'deletion_events'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.922813Z [info     ] 2023-09-26 03:37:42,921 | INFO     | tap-shopify          | Skipping deselected stream 'delivery_customizations'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.922969Z [info     ] 2023-09-26 03:37:42,921 | INFO     | tap-shopify          | Skipping deselected stream 'discount_nodes'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.923121Z [info     ] 2023-09-26 03:37:42,922 | INFO     | tap-shopify          | Skipping deselected stream 'discount_redeem_code_saved_searches'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.923297Z [info     ] 2023-09-26 03:37:42,922 | INFO     | tap-shopify          | Skipping deselected stream 'draft_orders'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.923425Z [info     ] 2023-09-26 03:37:42,922 | INFO     | tap-shopify          | Skipping deselected stream 'files'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.923546Z [info     ] 2023-09-26 03:37:42,922 | INFO     | tap-shopify          | Skipping deselected stream 'fulfillment_orders'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.923743Z [info     ] 2023-09-26 03:37:42,922 | INFO     | tap-shopify          | Skipping deselected stream 'gift_cards'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.924104Z [info     ] 2023-09-26 03:37:42,922 | INFO     | tap-shopify          | Skipping deselected stream 'inventory_items'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.924245Z [info     ] 2023-09-26 03:37:42,923 | INFO     | tap-shopify          | Skipping deselected stream 'locations'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.924715Z [info     ] 2023-09-26 03:37:42,923 | INFO     | tap-shopify          | Skipping deselected stream 'manual_holds_fulfillment_orders'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.924888Z [info     ] 2023-09-26 03:37:42,923 | INFO     | tap-shopify          | Skipping deselected stream 'marketing_activities'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.925125Z [info     ] 2023-09-26 03:37:42,923 | INFO     | tap-shopify          | Skipping deselected stream 'marketing_events'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.925266Z [info     ] 2023-09-26 03:37:42,923 | INFO     | tap-shopify          | Skipping deselected stream 'metafield_definitions'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.925401Z [info     ] 2023-09-26 03:37:42,924 | INFO     | tap-shopify          | Skipping deselected stream 'metaobjects'. cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.926801Z [info     ] 2023-09-26 03:37:42,926 | INFO     | tap-shopify          | Beginning incremental sync of 'orders'... cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:42.927054Z [info     ] 2023-09-26 03:37:42,926 | INFO     | tap-shopify          | Tap has custom mapper. Using 1 provided map(s). cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.345970Z [info     ] 2023-09-26 03:37:43,345 | INFO     | target-postgres      | Initializing 'target-postgres' target sink... cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.747843Z [info     ] 2023-09-26 03:37:43,747 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "http_request_count", "value": 0, "tags": {"stream": "orders", "endpoint": ""}} cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.748230Z [info     ] 2023-09-26 03:37:43,747 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "sync_duration", "value": 0.43436479568481445, "tags": {"stream": "orders", "context": {}, "status": "failed"}} cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.748782Z [info     ] 2023-09-26 03:37:43,747 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 0, "tags": {"stream": "orders", "context": {}}} cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.749224Z [info     ] 2023-09-26 03:37:43,747 | ERROR    | tap-shopify          | An unhandled error occurred while syncing 'orders' cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.749431Z [info     ] Traceback (most recent call last): cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.749674Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 1175, in sync cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.749841Z [info     ]     for _ in self._sync_records(context=context): cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.750045Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 1070, in _sync_records cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.750237Z [info     ]     for record_result in self.get_records(current_context): cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.750425Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/rest.py", line 576, in get_records cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.750592Z [info     ]     for record in self.request_records(context): cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.750713Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/rest.py", line 393, in request_records cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.750873Z [info     ]     prepared_request = self.prepare_request( cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.751072Z [info     ]                        ^^^^^^^^^^^^^^^^^^^^^ cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.751211Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/rest.py", line 364, in prepare_request cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.751331Z [info     ]     request_data = self.prepare_request_payload(context, next_page_token) cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.751481Z [info     ]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.751577Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/tap_shopify/client.py", line 365, in prepare_request_payload cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.751659Z [info     ]     query = self.query.lstrip() cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.752237Z [info     ]             ^^^^^^^^^^         cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.752333Z [info     ]   File "/usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/functools.py", line 1001, in __get__ cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.752488Z [info     ]     val = self.func(instance)  cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.753216Z [info     ]           ^^^^^^^^^^^^^^^^^^^  cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.753334Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/tap_shopify/client.py", line 497, in query cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.753447Z [info     ]     self.evaluate_query()      cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.753571Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/tap_shopify/client.py", line 521, in evaluate_query cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.753771Z [info     ]     error_code = error.get("extensions", {}).get("code") cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.754028Z [info     ]                  ^^^^^^^^^     cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.754275Z [info     ] AttributeError: 'str' object has no attribute 'get' cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.754408Z [info     ] Traceback (most recent call last): cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.754525Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/bin/tap-shopify", line 8, in <module> cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.754682Z [info     ]     sys.exit(TapShopify.cli()) cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.754773Z [info     ]              ^^^^^^^^^^^^^^^^  cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.754858Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.754953Z [info     ]     return self.main(*args, **kwargs) cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.755077Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.755185Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/click/core.py", line 1078, in main cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.755277Z [info     ]     rv = self.invoke(ctx)      cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.755363Z [info     ]          ^^^^^^^^^^^^^^^^      cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.755448Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.755539Z [info     ]     return ctx.invoke(self.callback, **ctx.params) cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.755627Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.755725Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.755815Z [info     ]     return __callback(*args, **kwargs) cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.755900Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.755985Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 498, in invoke cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.756071Z [info     ]     tap.sync_all()             cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.756155Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 456, in sync_all cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.756240Z [info     ]     stream.sync()              cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.756325Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 1182, in sync cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.756413Z [info     ]     raise ex                   cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.756497Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 1175, in sync cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.756625Z [info     ]     for _ in self._sync_records(context=context): cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.756711Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/core.py", line 1070, in _sync_records cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.756798Z [info     ]     for record_result in self.get_records(current_context): cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.756884Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/rest.py", line 576, in get_records cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.756983Z [info     ]     for record in self.request_records(context): cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.757069Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/rest.py", line 393, in request_records cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.757157Z [info     ]     prepared_request = self.prepare_request( cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.757242Z [info     ]                        ^^^^^^^^^^^^^^^^^^^^^ cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.757326Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/singer_sdk/streams/rest.py", line 364, in prepare_request cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.757413Z [info     ]     request_data = self.prepare_request_payload(context, next_page_token) cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.757497Z [info     ]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.757583Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/tap_shopify/client.py", line 365, in prepare_request_payload cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.757668Z [info     ]     query = self.query.lstrip() cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.757751Z [info     ]             ^^^^^^^^^^         cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.758102Z [info     ]   File "/usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/functools.py", line 1001, in __get__ cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.758209Z [info     ]     val = self.func(instance)  cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.758300Z [info     ]           ^^^^^^^^^^^^^^^^^^^  cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.759214Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/tap_shopify/client.py", line 497, in query cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.759335Z [info     ]     self.evaluate_query()      cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.759432Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/extractors/tap-shopify/venv/lib/python3.11/site-packages/tap_shopify/client.py", line 521, in evaluate_query cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.759530Z [info     ]     error_code = error.get("extensions", {}).get("code") cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.759619Z [info     ]                  ^^^^^^^^^     cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.759705Z [info     ] AttributeError: 'str' object has no attribute 'get' cmd_type=extractor name=tap-shopify run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:43.824886Z [info     ] 2023-09-26 03:37:43,824 | INFO     | paramiko.transport   | Connected (version 2.0, client OpenSSH_8.9p1) cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:44.147057Z [info     ] 2023-09-26 03:37:44,146 | INFO     | paramiko.transport   | Authentication (publickey) successful! cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:37:44.147782Z [info     ] 2023-09-26 03:37:44,147 | INFO     | target-postgres      | Initializing target sink for stream 'orders'... cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.093174Z [info     ] Traceback (most recent call last): cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.093428Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-postgres/venv/bin/target-postgres", line 8, in <module> cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.094068Z [info     ]     sys.exit(TargetPostgres.cli()) cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.094268Z [info     ]              ^^^^^^^^^^^^^^^^^^^^ cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.094420Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.095533Z [info     ]     return self.main(*args, **kwargs) cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.095687Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.095921Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/click/core.py", line 1078, in main cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.096179Z [info     ]     rv = self.invoke(ctx)      cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.096326Z [info     ]          ^^^^^^^^^^^^^^^^      cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.096459Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.096618Z [info     ]     return ctx.invoke(self.callback, **ctx.params) cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.096825Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.096960Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.097092Z [info     ]     return __callback(*args, **kwargs) cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.097218Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.097343Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/singer_sdk/target_base.py", line 553, in invoke cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.097556Z [info     ]     target.listen(file_input)  cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.097693Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/singer_sdk/io_base.py", line 34, in listen cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.097958Z [info     ]     self._process_lines(file_input) cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.098134Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/singer_sdk/target_base.py", line 294, in _process_lines cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.098407Z [info     ]     counter = super()._process_lines(file_input) cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.098563Z [info     ]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.098701Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/singer_sdk/io_base.py", line 90, in _process_lines cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.098842Z [info     ]     self._process_schema_message(line_dict) cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.098982Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/singer_sdk/target_base.py", line 406, in _process_schema_message cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.099117Z [info     ]     _ = self.get_sink(         cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.099250Z [info     ]         ^^^^^^^^^^^^^^         cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.099383Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/singer_sdk/target_base.py", line 769, in get_sink cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.099612Z [info     ]     return self.add_sqlsink(stream_name, schema, key_properties) cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.099757Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.099890Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/singer_sdk/target_base.py", line 703, in add_sqlsink cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.100023Z [info     ]     sink.setup()               cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.100153Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/target_postgres/sinks.py", line 49, in setup cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.100311Z [info     ]     self.connector.prepare_table( cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.100440Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/target_postgres/connector.py", line 117, in prepare_table cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.100572Z [info     ]     self.prepare_column(       cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.100874Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/target_postgres/connector.py", line 378, in prepare_column cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.101136Z [info     ]     self._adapt_column_type(   cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.101789Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-postgres/venv/lib/python3.11/site-packages/target_postgres/connector.py", line 500, in _adapt_column_type cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.101950Z [info     ]     raise NotImplementedError(msg) cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.102089Z [info     ] NotImplementedError: Altering columns is not supported. Could not convert column 'meltano_tap_shopify.orders.customer__id' from 'BIGINT' to 'VARCHAR'. cmd_type=loader name=target-postgres run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.280321Z [error    ] Extraction failed              code=1 message=AttributeError: 'str' object has no attribute 'get' name=meltano run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres
2023-09-26T08:39:07.280560Z [error    ] Loading failed                 code=1 message=NotImplementedError: Altering columns is not supported. Could not convert column 'meltano_tap_shopify.orders.customer__id' from 'BIGINT' to 'VARCHAR'. name=meltano run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres
2023-09-26T08:39:07.281078Z [info     ] ELT could not be completed: Extractor and loader failed. cmd_type=elt name=meltano run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.295006Z [info     ] For more detailed log messages re-run the command using 'meltano --log-level=debug ...' CLI flag. cmd_type=elt name=meltano run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.295169Z [info     ] Note that you can also check the generated log file at '/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/logs/elt/2023-09-26T083712--tap-shopify--target-postgres/445e675a-a305-4742-9599-0584ca67a4a3/elt.log'. cmd_type=elt name=meltano run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
2023-09-26T08:39:07.295315Z [info     ] For more information on debugging and logging: https://docs.meltano.com/reference/command-line-interface#debugging cmd_type=elt name=meltano run_id=445e675a-a305-4742-9599-0584ca67a4a3 state_id=2023-09-26T083712--tap-shopify--target-postgres stdio=stderr
sehnem commented 10 months ago

Hi @konung, I am currently working on refactoring the main logic of this tap, there are a lot of constrains with the GraphQL API, and the logs are not great. This erros usually happens when the query is too long, try to select just a few fields to test and see if it woks, if not, I can try to help you to debug the error.

I hope I have the refactoring of the tap done in a week or two, but feedback on this implementation is welcome.

konung commented 10 months ago

@sehnem Thank you so much for looking into this.

I tried to transfer a bare minimum of attributes for an order and still hit errors. Also if I'm reading Meltano docs correctly - even if I select only 2 attributes - the ones that are marked as "automatic" will still be selected even for those entities that I don't want to transfer ? They all seem related to schema ( doesn't matter if I use "generated/saved schema file" or the let meltano build on on the fly) I think I'm missing something with my config / schema

Here is my meltano.yml


version: 1
default_environment: dev
project_id: 5ce76c58-daa5-410c-9e9f-3893d658b5cc
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-spreadsheets-anywhere
    variant: ets
    pip_url: git+https://github.com/ets/tap-spreadsheets-anywhere.git
    config:
      tables:
      - path: https://ip-ranges.amazonaws.com
        format: json
        start_date: '2022-12-01T00:00:00Z'
        key_properties: [ip_prefix]
        name: aws_ips
        pattern: ip-ranges.json
        json_path: prefixes
  - name: tap-shopify
    variant: sehnem
    pip_url: git+https://github.com/sehnem/tap-shopify.git
    config:
      api_version: 2023-10
      bulk: false
      flattening_enabled: true
      flattening_max_depth: 3
      start_date: '2023-09-25'
      use_numeric_ids: false
    select:
    - orders.id
    - orders.legacyResourceId
    - orders.createdAt
    - orders.email
    - orders.updatedAt
    catalog: extract/shopify.sehnem.catalog.json
  loaders:
  - name: target-postgres
    variant: meltanolabs
    pip_url: meltanolabs-target-postgres~=0.0.7
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
    config:
      do_timestamp_file: true
      custom_name: slshopify
  - name: target-csv
    variant: hotgluexyz
    pip_url: git+https://github.com/hotgluexyz/target-csv.git
  utilities:
  - name: airflow
    variant: apache
    pip_url: git+https://github.com/meltano/airflow-ext.git@main apache-airflow==2.3.3
      --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.3.3/constraints-no-providers-${MELTANO__PYTHON_VERSION}.txt

When I try to run

meltano el tap-shopify target-jsonl


2023-09-26T15:42:13.361009Z [info     ] Running extract & load...      name=meltano run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl
2023-09-26T15:42:13.961573Z [warning  ] No state was found, complete import.
2023-09-26T15:42:13.966160Z [info     ] Found catalog in /Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/extract/shopify.sehnem.catalog.json
2023-09-26T15:42:27.186884Z [info     ] 2023-09-26 10:42:27,186 | INFO     | tap-shopify          | Beginning full_table sync of 'automatic_discount_nodes'... cmd_type=extractor name=tap-shopify run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.188026Z [info     ] 2023-09-26 10:42:27,186 | INFO     | tap-shopify          | Tap has custom mapper. Using 1 provided map(s). cmd_type=extractor name=tap-shopify run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.567558Z [info     ] 2023-09-26 10:42:27,567 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.159894, "tags": {"stream": "automatic_discount_nodes", "endpoint": "", "http_status_code": 200, "status": "succeeded"}} cmd_type=extractor name=tap-shopify run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.585098Z [info     ] 2023-09-26 10:42:27,584 | WARNING  | tap-shopify          | Properties ('edges', 'pageInfo') were present in the 'automatic_discount_nodes' stream but not found in catalog schema. Ignoring. cmd_type=extractor name=tap-shopify run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.585424Z [info     ] 2023-09-26 10:42:27,585 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "http_request_count", "value": 1, "tags": {"stream": "automatic_discount_nodes", "endpoint": ""}} cmd_type=extractor name=tap-shopify run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.585578Z [info     ] 2023-09-26 10:42:27,585 | INFO     | singer_sdk.metrics   | METRIC: {"type": "timer", "metric": "sync_duration", "value": 0.39823007583618164, "tags": {"stream": "automatic_discount_nodes", "context": {}, "status": "succeeded"}} cmd_type=extractor name=tap-shopify run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.585707Z [info     ] 2023-09-26 10:42:27,585 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 1, "tags": {"stream": "automatic_discount_nodes", "context": {}}} cmd_type=extractor name=tap-shopify run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.585930Z [info     ] 2023-09-26 10:42:27,585 | INFO     | tap-shopify          | Beginning full_table sync of 'catalogs'... cmd_type=extractor name=tap-shopify run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.586061Z [info     ] 2023-09-26 10:42:27,585 | INFO     | tap-shopify          | Tap has custom mapper. Using 1 provided map(s). cmd_type=extractor name=tap-shopify run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.586490Z [info     ] Traceback (most recent call last): cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.586644Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-jsonl/venv/bin/target-jsonl", line 8, in <module> cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.621001Z [info     ]     sys.exit(main())           cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.621178Z [info     ]              ^^^^^^            cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.621321Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-jsonl/venv/lib/python3.11/site-packages/target_jsonl.py", line 92, in main cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.621476Z [info     ]     state = persist_messages(  cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.621609Z [info     ]             ^^^^^^^^^^^^^^^^^  cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.621737Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-jsonl/venv/lib/python3.11/site-packages/target_jsonl.py", line 54, in persist_messages cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.621876Z [info     ]     validators[o['stream']].validate((o['record'])) cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.622006Z [info     ]   File "/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/loaders/target-jsonl/venv/lib/python3.11/site-packages/jsonschema/validators.py", line 130, in validate cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.622135Z [info     ]     raise error                cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.622263Z [info     ] jsonschema.exceptions.ValidationError: 'id' is a required property cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.622392Z [info     ]                                cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.622519Z [info     ] Failed validating 'required' in schema: cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.622713Z [info     ]     {'properties': {'id': {'type': ['string']}, cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.622938Z [info     ]                     'metafieldDefinitions__edges': {'items': {'properties': {'node': {'properties': {'id': {'type': ['string']}}, cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.623097Z [info     ]                                                                                       'required': ['id'], cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.623239Z [info     ]                                                                                       'type': 'object'}}, cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.623390Z [info     ]                                                               'required': ['node'], cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.623521Z [info     ]                                                               'type': 'object'}, cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.623863Z [info     ]                                                     'type': 'array'}}, cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.623999Z [info     ]      'required': ['id', 'metafieldDefinitions'], cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.624128Z [info     ]      'type': 'object'}         cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.624276Z [info     ]                                cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.624465Z [info     ] On instance:                   cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.624618Z [info     ]     {}                         cmd_type=loader name=target-jsonl run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.651955Z [error    ] Loading failed                 code=1 message=    {} name=meltano run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl
2023-09-26T15:42:27.652464Z [info     ] ELT could not be completed: Loader failed. cmd_type=elt name=meltano run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.652624Z [info     ] For more detailed log messages re-run the command using 'meltano --log-level=debug ...' CLI flag. cmd_type=elt name=meltano run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.652763Z [info     ] Note that you can also check the generated log file at '/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/logs/elt/2023-09-26T154212--tap-shopify--target-jsonl/7f0da754-0c3f-480d-b780-2328ed721108/elt.log'. cmd_type=elt name=meltano run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
2023-09-26T15:42:27.652903Z [info     ] For more information on debugging and logging: https://docs.meltano.com/reference/command-line-interface#debugging cmd_type=elt name=meltano run_id=7f0da754-0c3f-480d-b780-2328ed721108 state_id=2023-09-26T154212--tap-shopify--target-jsonl stdio=stderr
sehnem commented 10 months ago

This is a known error, use bulk as true and it should be solved. The current version of the tap is not in a good shape, I am fixing a lot of things in the structure right now and things may not be working as they should. In the next week I will release a new version with fixes for these cases.

konung commented 9 months ago

Thanks @sehnem . I tried bulk per your suggestion, but still ran into more issues with schema and empty fields. I will just wait till your next release to test again. For now I'm using airbyte, even if it is painfully slow :D It does the job, thou, I was just looking for an option to speed up things

sehnem commented 9 months ago

The bulk API is much faster, but it is much more complex with a lot of edgecases, I will ping you once I release the new version.

Em qui., 28 de set. de 2023 às 01:50, Nick Gorbikoff < @.***> escreveu:

Thanks @sehnem https://github.com/sehnem . I tried bulk per your suggestion, but still ran into more issues with schema and empty fields. I will just wait till your next release to test again. For now I'm using airbyte, even if it is painfully slow :D It does the job, thou.

— Reply to this email directly, view it on GitHub https://github.com/sehnem/tap-shopify/issues/9#issuecomment-1738427625, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOUJ3VHQ3KORDYKFFF6QJ3X4T6ZTANCNFSM6AAAAAA5HJJFQ4 . You are receiving this because you were mentioned.Message ID: @.***>

konung commented 9 months ago

Excellent. Thank you!!

Have a great day!

Nick

Message ID: ***@***.***>