search

KubedAI / spark-history-server

Helm Chart for deploying Spark history server in Amazon EKS for S3 Spark Event Logs
https://kubedai.github.io/spark-history-server/
Apache License 2.0
15 stars 8 forks source link

Is it possible to have an S3 bucket with logs divided across subdirectories? #19

Open camilaagw opened 1 month ago

camilaagw commented 1 month ago

A question for my current use case: I am wondering if with this setup it is possible to have an S3 bucket with logs divided across subdirectories. For example:

s3://my-bucket/dir1/ s3://my-bucket/dir2/ s3://my-bucket/dir3/ etc

vara-bonthu commented 2 weeks ago

Logs are written to s3 bucket by the Spark job. you can define a unique path for each job https://github.com/awslabs/data-on-eks/blob/9e52517badf3d3bca758544f09d1f905c07eec0f/analytics/terraform/spark-k8s-operator/examples/karpenter/nvme-ephemeral-storage/nvme-ephemeral-storage.yaml#L55