search

stanfordnlp / dspy

DSPy: The framework for programming—not prompting—foundation models
https://dspy-docs.vercel.app/
MIT License
17.36k stars 1.33k forks source link

add llamaindex integration #1170

Closed jerryjliu closed 3 months ago

jerryjliu commented 3 months ago

add integrations + a full cookbook showing llamaindex + dspy:

  1. optimize a query pipeline with dspy
  2. port over a llamaindex prompt to a dspy component
  3. extract dspy component back into llamaindex prompt template
okhat commented 3 months ago

Thank you Jerry. We should merge this, but we're failing on Ruff checks, so that's the only thing we gotta resolve.

arnavsinghvi11 commented 3 months ago

Thanks @jerryjliu for this PR! I patched the ruff fix directly.

left a minor comment for a typo and also just wanted to confirm if the PR was ready to merge from your end since there were some in-line TODOs.

will merge once you're ready!

arnavsinghvi11 commented 3 months ago

ah so with the updated imports, this is failing the tests since llama_index is not a supported dependency. The llama_index import statements need to be wrapped in a try/except block to avoid triggering the errors like done here or moving it within the various initialization methods where the imports are needed as done here.

I can take a look at this soon to resolve this locally as well.

[update - feel free to disregard above. llamaindex needs to be a required dependency to import all its functions but updating poetry.lock within this PR breaks local DSPy tests. integration for libraries to avoid this import patch will be added via comprehensive integration tests in upcoming PRs!]

arnavsinghvi11 commented 3 months ago

Thanks again @jerryjliu ! Exciting integration!

jerryjliu commented 3 months ago

woot oops sorry just seeing this. @arnavsinghvi11 thanks for landing!!!

(btw feel free to DM me your twitter/linkedin handles, would be happy to promote on socials when this goes out)