When a response contains a subdict, i.e. insights/page_consumptions_by_consumption_type, the context (key from subdict) is extracted to a separate column, but not appended to the top level id. I am using target-mssql which complains about duplicate primary keys in any upserted chunk.
When a response contains a subdict, i.e.
insights/page_consumptions_by_consumption_type
, the context (key from subdict) is extracted to a separate column, but not appended to the top levelid
. I am usingtarget-mssql
which complains about duplicate primary keys in any upserted chunk.Example:
I tried setting
context
as an extra primary key but for 95% of rows this is null, so causes another error.Instead this modification seems to work:
https://github.com/voxmedia/tap-facebook-pages/blob/08800f1c8c0f7cac42a2951d43c78bec2dcc7fd8/tap_facebook_pages/streams.py#L224
The JSON example above would become: