prrao87 / db-hub-fastapi

Async bulk data ingestion and querying in various document, graph and vector databases via their Python clients
MIT License
33 stars 3 forks source link

Pydantic v2 updates for Neo4j #30

Closed prrao87 closed 1 year ago

prrao87 commented 1 year ago

Updates for Pydantic v2

This PR incorporates changes to each DB's API and ingestion routines to work with Pydantic v2. Because of the ~5x performance improvement seen when using Pydantic v2, we don't need to focus on optimizing the bulk ingestion too much during the validation stage -- as a result, alongside syntactic and API changes for Pydantic, the portions of the code that use multiprocessing for validation are simplified (and not required, because multiprocessing involves its own overhead).