Refactor OpenAIEmbeddings
1- Supported escape chars that break the Open AI json content.
2- Changed the output annotator type. DOCUMENT --> SENTENCE_EMBEDDINGS
NOTE: This approach is the reverse of the backward compatibility
3- Added metadata which comes from the document column to output embeddings
4- Added Python unit test class
5- Added a new submodule to support saving/loading the annotator
NOTE: The new submodule will fix saving/loading the annotator
Motivation and Context
How Has This Been Tested?
Tested via Python and Scala locally.
Additionally, I added new unit tests that cover my changes.
Screenshots (if appropriate):
Types of changes
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Code improvements with no or little impact
[ ] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
Refactor OpenAIEmbeddings annotator
Description
Refactor OpenAIEmbeddings 1- Supported escape chars that break the Open AI json content. 2- Changed the output annotator type. DOCUMENT --> SENTENCE_EMBEDDINGS NOTE: This approach is the reverse of the backward compatibility 3- Added metadata which comes from the document column to output embeddings 4- Added Python unit test class 5- Added a new submodule to support saving/loading the annotator NOTE: The new submodule will fix saving/loading the annotator
Motivation and Context
How Has This Been Tested?
Tested via Python and Scala locally. Additionally, I added new unit tests that cover my changes.
Screenshots (if appropriate):
Types of changes
Checklist: