search

deepjavalibrary / djl

An Engine-Agnostic Deep Learning Framework in Java
https://djl.ai
Apache License 2.0
4.17k stars 663 forks source link

any plan to support LLMs #2626

Open chgxtony opened 1 year ago

chgxtony commented 1 year ago

any plan to support LLMs like openai / llama ? like langchain in java

frankfliu commented 1 year ago

We are working on LLM support, see: https://github.com/deepjavalibrary/djl/pull/2547

We have no plan to support langchain, however, if you are looking for multimodal support, you can take a look: https://github.com/deepjavalibrary/djl-serving/blob/master/serving/docs/workflows.md

sandys commented 1 year ago

@frankfliu does #2547 need to be merged before any LLM can be used ? i was trying to use MPT 7B. Example code here - https://github.com/arakoodev/onnx-djl-example

onnx model loads and all. But cant get it to spit answers.

frankfliu commented 1 year ago

@sandys

If you just want to deploy llama, you can already do it with DJLServing: https://github.com/deepjavalibrary/djl-serving/pull/844

Technically, you can run LLMs with DJL in pure java fashion as well. What's missing is postprocessing and result token search. You can implement it by yourself if you want to.

sandys commented 1 year ago

@frankfliu hey thanks so much for your reply! Two questions:

If you just want to deploy llama, you can already do it with DJLServing: https://github.com/deepjavalibrary/djl-serving/pull/844

Llama has a licensing issue, so wondering if u have tested on any other LLMs ? like any at all ?

Technically, you can run LLMs with DJL in pure java fashion as well.

happy to try and implement (and contribute back). starting from our early rough work here - https://github.com/arakoodev/onnx-djl-example . But am not sure how to do this. I dont need any optimisation or anything at all. just get it to reply for now. any pointers how to do this ? we are attempting this with MPT-7B, but ill take anything with an open license.