search

markddrake / YADAMU---Yet-Another-DAta-Migration-Utility

A utility for Migrating Data between Oracle, Postgres, MySQL MariaDB, Snowflake. Stage Data from supported database to Amazon S3 and Azure Blob Storage in JSON and CSV Formats
MIT License
17 stars 10 forks source link

MongoDBI: Replace mapReduce operations with Aggregation Pipeline operations #227

Closed markddrake closed 11 months ago

markddrake commented 1 year ago

Mongo had deprecated the mapReduce() operator. The current Mongo Driver uses mapReduce() to determine the set of top-level keys in a mongo collection. The driver needs to be updated to use the mongo aggregation framework.

markddrake commented 1 year ago

The driver now uses the mongo aggregation framework to determine the names and datatypes of the top-level keys in a mongo collection. The order of the keys identified by the aggregation framework implementation differs from the order generated using mapReduce(), which required further changes to ensure that the mapping between the key-value pairs in the mongo document correctly mapped to columns