Initial support for PV and PVC creation

Open-EO / openeo-geopyspark-driver

OpenEO driver for GeoPySpark (Geotrellis)

Apache License 2.0

25 stars 4 forks source link

Initial support for PV and PVC creation #684

Closed tcassaert closed 4 months ago

tcassaert commented 4 months ago

To FUSE mount an S3 bucket in the batch jobs, a PV and PVC should be precreated. These resources use the CSI-S3 driver, which requires us to do static provisioning if we want to mount a subdirectory of a bucket.

tcassaert commented 4 months ago

Things to add:

[x] Delete the PV and PVC on job cancel
[ ] Delete the PV and PVC on job finish?
[x] Mount the PVC in the pod by default (not possible to do it via custom config as the name is generated)
[x] Actually write job results to disk instead of via S3

tcassaert commented 4 months ago

@soxofaan as issues like #283 are introduced by me... Could you guide me in how to feature flag correctly?

soxofaan commented 4 months ago

@soxofaan as issues like https://github.com/Open-EO/openeo-geopyspark-driver/issues/283 are introduced by me... Could you guide me in how to feature flag correctly?

I don't think this should be covered in this PR, that will lead us too far

tcassaert commented 4 months ago

@soxofaan as issues like #283 are introduced by me... Could you guide me in how to feature flag correctly?

I don't think this should be covered in this PR, that will lead us too far

For sure. I was not thinking about fixing those in #283, I was just thinking about maybe feature-flagging this and I didn't want to introduce more technical debt.

soxofaan commented 4 months ago

The new recommended way to define config options or feature flags is GpsBackendConfig, where you can define a new field, it's type and assign it a default value. In particular deployments this field can then be overriden from python based config files. (e.g. backendconfig_mep.py in openeo-deploy or backendconfig_prod.py in os_creodias_openeo_k8s)

tcassaert commented 4 months ago

@soxofaan should the cleanup of the PV and PVCs be handled by openEO and if so, where exactly would this have to happen?

soxofaan commented 4 months ago

should the cleanup of the PV and PVCs be handled by openEO and if so,

I don't think that clean up (how and triggered from where) is in scope of this ticket, right @jdries ?

tcassaert commented 4 months ago

@soxofaan I've added a feature flag. Is this how it should be done?

soxofaan commented 4 months ago

@soxofaan I've added a feature flag. Is this how it should be done?

yes that fuse_mount_batchjob_s3_bucket config looks good