Unstructured-IO / unstructured

Open source libraries and APIs to build custom preprocessing pipelines for labeling, training, or production machine learning pipelines.
https://www.unstructured.io/
Apache License 2.0
8.61k stars 704 forks source link

feat: add v2 pinecone destination connector #3286

Closed ahmetmeleq closed 2 months ago

ahmetmeleq commented 3 months ago

This PR adds a V2 version of the Pinecone destination connector

potter-potter commented 3 months ago

Did the integration test work for you?

I'm not getting any files into the upload_stage folder. And thus there is nothing to upload to pinecone.

ahmetmeleq commented 3 months ago

@potter-potter

Did the integration test work for you?

I'm not getting any files into the upload_stage folder. And thus there is nothing to upload to pinecone.

What's the path for the test? In terms of the example script, it now works for me after working test with 1 process and working example with 2 processes

potter-potter commented 3 months ago

@potter-potter

Did the integration test work for you? I'm not getting any files into the upload_stage folder. And thus there is nothing to upload to pinecone.

What's the path for the test? In terms of the example script, it now works for me after working test with 1 process and working example with 2 processes

the test is simply the pinecone.sh once you add pinecone to unstructured/ingest/v2/cli/cmds/init.py then the https://github.com/Unstructured-IO/unstructured/blob/main/test_unstructured_ingest/dest/pinecone.sh uses V2

ahmetmeleq commented 3 months ago

@potter-potter

the test is simply the pinecone.sh once you add pinecone to unstructured/ingest/v2/cli/cmds/init.py then the https://github.com/Unstructured-IO/unstructured/blob/main/test_unstructured_ingest/dest/pinecone.sh uses V2

I see, now got it working with reduce composite element size in test to comply with pinecone limits … and working test .

We had two additional problems later where

each commit addressing one of these in respective order.