search

cube-js / cube

📊 Cube — The Semantic Layer for Building Data Applications
https://cube.dev
Other
17.71k stars 1.75k forks source link

Store pre-aggregations in a directory within an export bucket with the BigQuery driver #4252

Open adnanrahic opened 2 years ago

adnanrahic commented 2 years ago

Is your feature request related to a problem? Please describe. I want to store pre-aggregations for different tenants in separate folders within the same export bucket.

Describe the solution you'd like Maybe to have CUBEJS_DB_EXPORT_BUCKET changed to CUBEJS_DB_EXPORT_PATH, or at least have the semantic changes to support both plain buckets and s3://... or gs://... paths like boto/gsutil does.

Describe alternatives you've considered There is no alternative solution except for using different buckets altogether.

Additional context The BigQuery driver uses a bucket eg CUBEJS_DB_EXPORT_BUCKET=cube-cloud-staging-export-bucket. The Athena driver supports S3 paths eg CUBEJS_DB_EXPORT_BUCKET=s3://cubejs-benchmarks/home/app/.../.

guangyuchendice commented 2 years ago

I faced the same problem but in RedShift data warehouse. It does not support adding prefix/subfolder under an S3 bucket in .env file. If you tried hard-coded the path after the bucket name it will have errors like: InvalidBucketName: Bucket name shouldn't contain '/', received (your wrong path here)

github-actions[bot] commented 2 years ago

If you are interested in working on this issue, please leave a comment below and we will be happy to assign the issue to you. If this is the first time you are contributing a Pull Request to Cube.js, please check our contribution guidelines. You can also post any questions while contributing in the #contributors channel in the Cube.js Slack.

ivan-vdovin commented 2 years ago

Added a separate issue for Redshift #5114