search

opensource-observer / oso

Measuring the impact of open source software
https://opensource.observer
Apache License 2.0
67 stars 16 forks source link

Store parquet/CSV dumps on Cloudflare R2 #919

Open ryscheng opened 6 months ago

ryscheng commented 6 months ago

CC @ravenac95 apparently Cloudflare R2 has free egress! https://developers.cloudflare.com/r2/pricing/

ryscheng commented 6 months ago

related to https://github.com/opensource-observer/oso/issues/919

ryscheng commented 6 months ago

Another option if we want IPFS support https://filebase.com/pricing/

ryscheng commented 6 months ago

Let's do this monthly for now, we can increase frequency later

davidgasquez commented 5 months ago

Sharing that the R2 + Filebase combo is really sweet! You can use the same S3 API (probably a Dagster resource) to push data to both. The only requirements for Filebase would be to package all the tables into a car archie though.

ryscheng commented 2 weeks ago

This might be superceded by a better lakehouse approach https://github.com/opensource-observer/oso/issues/1209

TBD