search

aws-samples / amazon-sagemaker-feature-store-end-to-end-workshop

MIT No Attribution
124 stars 49 forks source link

Example on materializing offline features to the online store. #44

Closed tonykchen closed 1 year ago

tonykchen commented 1 year ago

Issue #, if available:

Description of changes:

Demonstrate how customers can use the Feature Store Spark Connector to ingest features directly to the offline store, and incrementally materialize the latest features to the online store.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

tonykchen commented 1 year ago

Per my discussion with @philikai, we will merge #43 first.

tonykchen commented 1 year ago

For this example, we are not running the Feature Store Spark Connector in the custom processing container. Instead, we are installing the connector to temp/, packaging the required python modules as .zip, and submitting both the .zip and Jar file when executing PySparkProcessor.run(). With this approach, we won't need to check-in the connector's Python modules and Jar.

lauerarnaud commented 1 year ago

Reviewed PR, run the new files with a brand new user and worked fine. Feedback has been incorporated into the latest commits.