-
## Description
I followed the recipe given [here](https://docs.djl.ai/docs/serving/serving/docs/lmi/tutorials/trtllm_manual_convert_tutorial.html) to manually convert teknium/OpenHermes-2.5-Mistral-7…
-
I ran
```
client = mii.serve("mistralai/Mistral-7B-Instruct-v0.2")
response = client.generate(inputs, max_new_tokens=128, tensor_parallel=2, replica_num=2)
```
on AWS ml.g5.12xlarge with 4 GPUs…
-
**Is your feature request related to a problem? Please describe.**
The feature request is related to the problem of manually matching job seeker resumes with relevant job descriptions, which is a ti…
-
## Description
We are using SageMaker for large model inference (LMI) as documented [here](https://docs.aws.amazon.com/sagemaker/latest/dg/large-model-inference-dlc.html)
With this notebook http…
-
## Description
Tokens not streaming not working with rolling batch
### Expected Behavior
(what's the expected behavior?)
### Error Message
## How to Reproduce?
(If you developed your own…
-
## Description
在使用OnnxRuntime引擎不会回收jvm内存
下面是代码,已经有很多使用者提出内存不会回收,不知道你们为啥这么自信你们一直没有问题,我们已经转到python在生产环境。你们有时间看下吧
```
mport ai.djl.Device;
import ai.djl.MalformedModelException;
import ai.djl…
-
## Description
本来是把djl推理yolov5模型整合到springboot项目中,压测发现内存无法回收,随着压测的增加内存也增加。
注意:和这个issues并不是一个问题https://github.com/deepjavalibrary/djl/issues/2800 ,它这个在main线程中执行确实不会有问题,但是异步执行就会有内存泄漏问题。
在spring项目中内存…
90600 updated
4 months ago
-
楼主您好啊,我服务器去部署该的时候,普通启动项目,nohub java -jar 会一直报错Caused by: java.lang.UnsatisfiedLinkError: /root/.djl.ai/pytorch/1.9.1-cpu-linux-x86_64/libtorch_cpu.so: /lib64/libm.so.6: version `GLIBC_2.23' not found …
fjwnb updated
2 years ago
-
DJL does not support (or has not documented support) for FP8 quantization ([docs](https://demodocs.djl.ai/docs/serving/serving/docs/lmi/user_guides/trt_llm_user_guide.html#quantization-support)).
…
-
## Description
When I build a project supporting DJL in Android Studio, I encounter the following error. How can I resolve it?
`Could not resolve all files for configuration ':app:debugRuntimeClas…