search

singer-io / tap-exacttarget

A Singer.io tap for extracting data from the ExactTarget/Salesforce Marketing Cloud API.
GNU Affero General Public License v3.0
14 stars 34 forks source link

Error: "CRITICAL 'metadata'" #50

Closed jonnagel closed 3 years ago

jonnagel commented 3 years ago

I think I have collected everything needed, please let me know if more information is needed.

OS: Ubuntu 20.04.1 LTS Pull from master, build, edit config.json (all done today)

Run: tap-exacttarget -c config.json --discover > catalog.json

[trimmed]

INFO Getting more results from 'DataExtension' endpoint INFO Fetched 2500 results from 'DataExtension' endpoint INFO Getting more results from 'DataExtension' endpoint INFO Fetched 2500 results from 'DataExtension' endpoint INFO Getting more results from 'DataExtension' endpoint INFO Fetched 707 results from 'DataExtension' endpoint INFO Done retrieving results from 'DataExtension' endpoint INFO Making RETRIEVE call to 'DataExtensionField' endpoint with no filters.

[trimmed]

CRITICAL 'metadata' Traceback (most recent call last): File "/home//anaconda3/bin/tap-exacttarget", line 8, in sys.exit(main()) File "/home//anaconda3/lib/python3.7/site-packages/singer/utils.py", line 229, in wrapped return fnc(*args, **kwargs) File "/home//anaconda3/lib/python3.7/site-packages/tap_exacttarget/init.py", line 156, in main do_discover(args) File "/home//anaconda3/lib/python3.7/site-packages/tap_exacttarget/init.py", line 68, in do_discover catalog += stream_accessor.generate_catalog() File "/home//anaconda3/lib/python3.7/site-packages/tap_exacttarget/endpoints/data_extensions.py", line 136, in generate_catalog extensions_catalog_with_fields = self._get_fields(extensions_catalog) File "/home//anaconda3/lib/python3.7/site-packages/tap_exacttarget/endpoints/data_extensions.py", line 125, in _get_fields to_return[extension_id]['metadata'].append({ KeyError: 'metadata'

luandy64 commented 3 years ago

CRITICAL 'metadata'

Looks like a KeyError, but you trimmed the rest of the stack trace

jonnagel commented 3 years ago

updated the post

luandy64 commented 3 years ago

So we expect _get_fields to_return[extension_id] to be a dict that has a key metadata, but it doesn't.

Is that allowed?

jonnagel commented 3 years ago

I'm just an end-user, I'm not sure what is allowed. Should the discovery process extract metadata? Are there additional permissions needed for this that are not needed for other steps?

luandy64 commented 3 years ago

I don't know how debugging with anacoda works, but if you are able to edit the tap, run it, and see your changes take effect then I would do the following:

The rest of that line - it might be very verbose - should look something like this. I expect it to be missing that 'metadata' key though since that's what your error is.

This line and this line make me think that extention_id and customer_id are supposed to be the same thing, but I don't know what those are, so I can't say for sure.

This is all of the insight I have for you, good luck

jonnagel commented 3 years ago

"batch_size": 5000

Seems to have made this work in my case.