I am trying to get the history of insights by day over time. With the userinsights* feeds, I am only seeing the latest days data persisted to sqlite, despite setting the appropriate date boundaries in the config.
I can see the relevant queries in the Graph API query history and meltano is paginating through previous dates, but only the latest gets inserted into the table.
Is it somehow related to the primary_keys = ["id"] set on these streams? Does the PK need to be [id, end_time] to retain all the rows? When I run the extractor using target-jsonl I can see all the previous days in the output file which makes me think it is PK/upsert related.
I am trying to get the history of insights by day over time. With the userinsights* feeds, I am only seeing the latest days data persisted to sqlite, despite setting the appropriate date boundaries in the config.
I can see the relevant queries in the Graph API query history and meltano is paginating through previous dates, but only the latest gets inserted into the table.
Is it somehow related to the
primary_keys = ["id"]set on these streams? Does the PK need to be [id, end_time] to retain all the rows? When I run the extractor usingtarget-jsonlI can see all the previous days in the output file which makes me think it is PK/upsert related.https://github.com/voxmedia/tap-instagram/blob/a617855de95ea03c727c0a34551097187153a943/tap_instagram/streams.py#L725
EDIT: Just starting to realise I think this this is related to #13 :)