feast-dev / feast

The Open Source Feature Store for Machine Learning
https://feast.dev
Apache License 2.0
5.62k stars 1k forks source link

feat: File persistence definition and implementation #4742

Closed dmartinol closed 2 weeks ago

dmartinol commented 2 weeks ago

What this PR does / why we need it:

Add ephemeral persistence configuration to the Feast services deployed with the operator. A sample manifest is also committed and reported here for your convenience:

apiVersion: feast.dev/v1alpha1
kind: FeatureStore
metadata:
  name: sample-ephemeral-persistence
spec:
  feastProject: my_project
  services:
    onlineStore:
      persistence:
        file:
          path: /data/online_store.db
    offlineStore:
      persistence:
        file:
          type: dask
    registry:
      local:
        persistence:
          file:
            path: /data/registry.db

Next PR will add persistent storage by using mounted volumed (Kubernetes PVC)

Which issue(s) this PR fixes:

Relates to https://github.com/feast-dev/feast/issues/4561

dmartinol commented 2 weeks ago

@tchughesiv @tmihalac Note: validations will be added to the CRD types as the other options will be integrated.

dmartinol commented 2 weeks ago

requested a couple changes. also, i'm noticing we're missing new FeatureStore CR checks in featurestore_controller_test.go for these new status fields and defaults after reconcile

one more commit to add the missing checks

dmartinol commented 2 weeks ago

requested a couple changes. also, i'm noticing we're missing new FeatureStore CR checks in featurestore_controller_test.go for these new status fields and defaults after reconcile

one more commit to add the missing checks

Latest commit includes new controller tests. Please note that I had to change the resource name because some resources (svc?) are probably not deleted by previous tests, so it would fail due to an attempt to change ownership:

Object default/feast-services-offline is already owned by another Service controller feast-services-offline

I did not investigate further. I think we can handle this issue in a separate PR.

  2024-11-07T17:30:30+01:00     INFO    Successfully reconciled {"Service": "feast-services-offline", "operation": "updated"}
  2024-11-07T17:30:30+01:00     ERROR   Error deploying the FeatureStore client service {"error": "Object default/feast-services-offline is already owned by another Service controller feast-services-offline"}
  github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).setFeastServiceCondition
        /Users/dmartino/projects/AI/feast/feast/infra/feast-operator/internal/controller/services/services.go:290
  github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).deployFeastServiceByType
        /Users/dmartino/projects/AI/feast/feast/infra/feast-operator/internal/controller/services/services.go:95
  github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).Deploy
tchughesiv commented 2 weeks ago

I did not investigate further. I think we can handle this issue in a separate PR.

it's not an "issue" per say ... it's got to do w/ another unit test from a prior run where i've changed object ownership so we can test that ownership conflicts/failures are working properly. that said, we prob should manually cleanup that object after the fact.

so, yea ... i guess it is an issue :)