search

microsoft / LLMLingua

To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
https://llmlingua.com/
MIT License
4.42k stars 241 forks source link

Exploring the Possibility of Porting LLMLingua to JVM Languages (Java/Kotlin) #23

Closed fabriciorissetto closed 9 months ago

fabriciorissetto commented 9 months ago

Hello guys, awesome project here!

I'm reaching out to inquire about the possibility of porting LLMLingua to JVM languages, particularly Kotlin. Are there any plans for supporting JVM languages in the near future? I believe that making LLMLingua available in these languages could significantly broaden its reach and utility.

I understand that porting a library, especially one as sophisticated as LLMLingua, can be a substantial undertaking. In this regard, I have a few questions:

  1. How tightly coupled is the current implementation with Python-specific libraries and features?
  2. Are there aspects of LLMLingua that you foresee being particularly challenging to port to a JVM language?
  3. Would you be open to community contributions for this "portability"?

I am very interested in contributing to this effort. My background includes working with both Python and Kotlin, and I am keen to assist in making LLMLingua accessible to a wider range of developers.

iofu728 commented 9 months ago

Hi @fabriciorissetto, thank you for your interest and support. We also greatly appreciate help from the community.

  1. the current implementation only relies on torch, transformers, and numpy for model inference and matrix operation processes, and it is not intertwined with Python, so it can be fully applied to other programming languages.
  2. I don't think there are any unique challenges in porting LLMLingua to JVM, it should be similar to the JVM versions of other machine learning models.
  3. we currently have no plans to port to other languages. We greatly appreciate your contributions, but I suggest you fork a repository to carry out development there. Once completed, I will add the link to the README for better code management.

We look forward to LLMLingua helping in more scenarios. Thank you again for your help and support.

fabriciorissetto commented 9 months ago

Got it! Thanks for the information @iofu728.

Closing the issue for now :fist_oncoming: