contiamo / contiamo-client-python

MIT License
2 stars 2 forks source link

Allow user to specify aggregation type when uploading data to data contract #23

Closed bmwilly closed 6 years ago

bmwilly commented 6 years ago

A normal ETL workflow is to fetch and preprocess new data, purge an existing data contract, and upload the new data. This process wipes out metadata for the contract such as aggregation type for each metric. We have ETL pipelines that purge and upload data to many contracts, so it is untenable to manually specify aggregation types every time they run.

We already have endpoints in place that allow you to post a payload against any metric to change its attributes, so this is just a matter of updating the Python client to support this functionality.

bmwilly commented 6 years ago

One can already make arbitrary requests using the resources.client.client().request method. There's no reason to do anything further for such a specific use case. See the "Modify aggregation types" section of this notebook for an example of changing the aggregation types of metrics in multiple contracts at once.