Integrating new frameworks

yogeshhk commented 10 months ago

Is your feature request related to a problem? Please describe. This feature request is related to brand new ask, such as integration of external libraries/frameworks.

Describe the use case Ideas: Not sure if the following ideas have been already considered or not, but still let me enumerate them

integrating Crew AI (https://github.com/joaomdmoura/crewAI): This library seems to be written on top of Langchain, and provides ways to specify Agents and Tasks, which work together as a crew to complete the given tasks. This arrangements seems to be an apt candidate to become 'declarative'.
a larger task is to make Langchain itself 'declarative'. With introduction of LangChain Expression Language (LCEL) https://python.langchain.com/docs/expression_language/ it is becoming declarative. Just by providing, LLM info, tools info, Agents, prompts etc, in yaml, would it be possible to run Langchain in Ludwig? (sorry, as I am unaware of the internal architecture, if the callbacks need to be TensorFlow/PyTorch compatible or we can call any function)

Describe the solution you'd like What would it take to integrate the above? Is Ludwig thin enough to integrate any new functionality, by specifying parameters in yaml and then having callbacks to get the task done?

alexsherstinsky commented 4 months ago

@yogeshhk We recently published a request for comments (RFC) on evolving the Ludwig architecture. It is in a message on our Discord. Do you think that your ideas for incorporating frameworks can fit as part of that proposal? Also, do you think that all of them are in-scope for Ludwig, or are some of them better left as independent? Thanks a lot!

yogeshhk commented 4 months ago

@yogeshhk We recently published a request for comments (RFC) on evolving the Ludwig architecture. It is in a message on our Discord. Do you think that your ideas for incorporating frameworks can fit as part of that proposal? Also, do you think that all of them are in-scope for Ludwig, or are some of them better left as independent? Thanks a lot!

@alexsherstinsky yes, bulging scope could be an issue as well. I guess, keeping this functionality separate is fine

ludwig-ai / ludwig

Integrating new frameworks #3895