search

rehanvdm / serverless-website-analytics

A CDK construct that consists of a serverless backend, frontend and client side code to track website analytics
GNU General Public License v2.0
166 stars 13 forks source link

Auto create partitions #2

Closed rehanvdm closed 1 year ago

rehanvdm commented 1 year ago
m-radzikowski commented 1 year ago

I've only briefly looked into the code, but here is a suggestion that may or may not make sense.

I see you have 3 partitions: site, year, and month.

For year and month, you can use partition projection. There is even a specific type for dates. Then you don't need to update partitions, Athena will know that there may exist year partitions, let's say from 2023 to 2099. And month partitions from 1 to 12.

site is more tricky. But it's also less common change that can be done manually. With dynamic partitions, you can use enum type, if possible sites are given in Swa config (I'm not sure about it). Otherwise, I see there is also an "injected type", but I did not use it and don't know exactly how it works.

Again, great job with this project!

rehanvdm commented 1 year ago

That is a great idea! Completely slipped my mind. I will explore this, thanks for the suggestion.

Regarding site that can easily be managed via a Custom Resource in the CDK. So it will then create/update/delete the site partition whenever it gets added, so a once-off. Then the date partitions are dynamic which means no more partition creating 🙂. Definitely going to look into this one. Thanks again.

rehanvdm commented 1 year ago

Released in v.0.0.5, thanks @m-radzikowski! I could delete a lot of code 😃