search

stitchfix / flotilla-os

Open source Flotilla
Apache License 2.0
192 stars 10 forks source link

Enables the use of lustre FSx #493

Closed willhume closed 6 months ago

willhume commented 6 months ago

Enables the use of lustre FSx but creates PVC at start of job and removes it at termination

PROBLEM

Current built in implementation of spark PVC's is it creates a pvc or a volume for each executor. This PR aims to create a lustre mount that is 1 to many that will create a PVC per job and mount it to all executors across multiple nodes.

SOLUTION

Handle PVC creation outside of Spark built in methods

willhume commented 6 months ago

are there any tests for anything?

The engines don't seem to have any but ill add some before merging to main. I want to test this approach first to see if we are gonna stick with it.