search

spring-cloud / spring-cloud-deployer

The Spring Cloud Deployer project defines an SPI for deploying long lived applications and short lived tasks
162 stars 87 forks source link

Provide for reusing PV in stateful stream apps. #446

Open corneil opened 4 months ago

corneil commented 4 months ago

Users need to specify the volumeName. Users need to be able to derive mapped path names from stream/app names. Users need to configure a dataSource for the PVC. They can then reference an existing PVC or VolumeSnapshot.

davidpenske commented 4 months ago

But how would this work with dynamic provisioning, especially when you have a versioned deployment. When we deploy Stateful KStream Apps (Using Kafka Streams Binder) with SCDF as StatefulSets, the name of the StatefulSet includes the name of the stream, the name of the app and a version number. This same name is used to create the name of the underlying PVC. When a stream is updated with new versions, a new PV is created and a new subpath will be created, the PVC will also have a new name with a new version number. I know we can manually change the name back, but that means whenever there is a new deployment we need to do a manually change, that will be too much work when we have all several hundred apps migrated to k8s.

corneil commented 4 months ago

We are considering an option where updates patches existing deployment/statefulset instead of creating a new version.

rakrvptl commented 4 months ago

Currently the PVC name is combination of VolumeTemplate name + Statefulset, volumetemplatename is externalized, can the add prefix be externalized, so if user specifies one use it else use statefulset?

The idea primarily is to ensure when we use stream apps that do read data from kafka topics and store it to local to perform stateful kafka operations, it doesnt have to do a full download of data from kafka topic in event of new deploys.