Spark connector needs to be updated to accommodate to FeatureStore latest changes.
Description of changes:
Add support for TargetStore
No longer blocking online stream ingestion if ingestion fails due to exhausted retries or validation errors
Removed schema validation for online ingestion to eliminate the overhead of validation
Capability of retrieving failed records to be ingested
Capability of cross region/account ingestion by accepting a role passed to spark connector
Improved the mechanism of spark connector's installation. Instead of copying jars around, we instead create a symlink in SPARK_HOME/jars
Tuned the dependency tree and removed hadoop related dependencies which have high risk of introducing dependency conflicts
Pack all jar files in one fat jar to help with importing the jar
Testing done:
Unit tests and some manual tests, more tests need to be performed
Merge Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.
General
[X] I have read the CONTRIBUTING doc
[X] I certify that the changes I am introducing will be backword compatible
[X] I used the commit message format described in CONTRIBUTING doc
Tests
[X] I have added tests that prove my fix is effective or that my feature works (if appropriate)
[X] I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
[X] I have verified all code in this commit are well formatted
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Issue #, if available:
Spark connector needs to be updated to accommodate to FeatureStore latest changes.
Description of changes:
Testing done:
Unit tests and some manual tests, more tests need to be performed
Merge Checklist
Put an
xin the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.General
Tests
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.