[ENH]: Static re-export of existing EFs for IDE static analysis

[tazarov]

Closes #2398

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • re-exported all existing EFs in _all_classes without breaking existing dynamic imports
  • made the EF test not fail if new EFs are added without being statically exported

Test plan

How are these changes tested?

• [x] Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

[github-actions[bot]]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• [ ] Can you think of any use case in which the code does not behave as intended? Have they been tested?
• [ ] Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• [ ] If appropriate, are there adequate property based tests?
• [ ] If appropriate, are there adequate unit tests?
• [ ] Should any logging, debugging, tracing information be added or removed?
• [ ] Are error messages user-friendly?
• [ ] Have all documentation changes needed been made?
• [ ] Have all non-obvious changes been commented?

  System Compatibility

• [ ] Are there any potential impacts on other parts of the system or backward compatibility?
• [ ] Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

  Quality

• [ ] Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[tazarov]

• #2408 👈
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @tazarov and the rest of your teammates on Graphite

[tazarov]

@atroyn, should we create a small guide about creating/adding new EFs, which should include:

• Good practices, e.g. handling error cases, testing etc.
• Where to put files, including static imports/re-export from __init__

[atroyn]

This just reverts us to static imports, which is undesirable. Did we explore any alternatives ?

[tazarov]

@atroyn, we can generate a .pyi file when packaging, but cloning from main won't work as the .pyi file won't exist. We can also add the stub (pyi) generation as part of the merge to the main when the ef sub dir changes; it is slightly hacky but can work.

A benefit of dynamic imports I see is that it makes adding new EFs ever so slightly simple, by removing the need to re-export from __init__. However, implementing dynamic imports that work with IDEs seems like a much taller (as per the above) order than having everybody statically re-export (one-liner). But it is likely I am missing a bigger point here.

How others are doing it:

• LC dynamic + static - https://github.com/langchain-ai/langchain/blob/master/libs/community/langchain_community/embeddings/__init__.py
• LI - separate pypi package e.g. https://github.com/run-llama/llama_index/tree/main/llama-index-integrations/embeddings/llama-index-embeddings-cohere

[atroyn]

I think we will eventually move to a separate contrib package which should help here (I think?). I think you're right that we should prefer the user experience to the developer experience.

One idea is this; we can use the dynamic import as a check to see if you created a new EF in the module, but did not import it yet, to remind you to import it. This could actually be a test or linting step, and there could be a decorator to ignore specific ones.

That might be best of both worlds.