search

Unstructured-IO / unstructured-ingest

Apache License 2.0
20 stars 19 forks source link

feat/upsert mongo #259

Closed rbiseck3 closed 23 hours ago

rbiseck3 commented 1 day ago

Description

Add in upsert/overwrite support to mongodb connector. Also improves the precheck for both the indexer and uploader to check the corresponding database/collection exist.

potter-potter commented 1 day ago

Seems like a bunch of the tests are being skipped in CI. https://github.com/Unstructured-IO/unstructured-ingest/actions/runs/11957823179/job/33335892611#step:7:533

Also tests are failing when I try them locally. Here's one of them.

test/integration/connectors/test_mongodb.py::test_mongodb_uploader_precheck_fail_no_database 2024-11-21 10:09:11,105 MainProcess ERROR    failed to validate connection: database database does not exist: admin, config, e2e-test-db, ingest-test-db, local
Traceback (most recent call last):
  File "/Users/potter/Documents/unstructured-ingest/unstructured_ingest/v2/processes/connectors/mongodb.py", line 298, in precheck
    raise DestinationConnectionError(
unstructured_ingest.error.DestinationConnectionError: database database does not exist: admin, config, e2e-test-db, ingest-test-db, local
FAILED