NVIDIA Merlin is an open source library providing end-to-end GPU-accelerated recommender systems, from feature engineering and preprocessing to training deep learning models and running inference in production.
Merlin now has a bunch of libraries that need to interoperate smoothly, but a general lack of shared abstractions, conventions, and standards that would make that possible.
Goal:
Build a solid foundation for the Merlin libraries via improvements in Core
New Functionality
Core:
Shape in column schemas (for consistent tracking across libraries)
Cross-framework dtype translation (e.g. via Merlin dtypes)
Cross-framework data transfer via zero-copy protocols (for Columns and DictArrays -> Series and Dataframes)
Bespoke Merlin schema file format (i.e. a Protobuf schema for Merlin schema that isn't from Tensorflow Metadata)
Corresponding updates in all downstream libraries
Constraints:
All functionality entailed by this issue has to work in and be adoptable by all Merlin libraries
Problem:
Merlin now has a bunch of libraries that need to interoperate smoothly, but a general lack of shared abstractions, conventions, and standards that would make that possible.
Goal:
New Functionality
Constraints:
Starting Point: