search

deepjavalibrary / djl-demo

Demo applications showcasing DJL
https://demo.djl.ai
Apache License 2.0
307 stars 127 forks source link

Upgrade to DJL 0.25.0 #387

Closed ydm-amazon closed 10 months ago

ydm-amazon commented 10 months ago

Upgrade notebooks to DJL 0.25.0.

lanking520 commented 10 months ago

Upgrading dependencies is not as simple as version bump up. You need to carefully reevaluate the content of each notebook and see if some parameters need to be added/removed.

For some examples, currently vLLM no longer needs to install HuggingFace anymore.

lanking520 commented 10 months ago

I would suggest to close the PR to update each notebook in individual PRs

ydm-amazon commented 10 months ago

Upgrading dependencies is not as simple as version bump up. You need to carefully reevaluate the content of each notebook and see if some parameters need to be added/removed.

For some examples, currently vLLM no longer needs to install HuggingFace anymore.

I did reevaluate the content of each notebook. For example, in tnx_rollingbatch_deploy_llama_7b_int8.ipynb, option.load_in_8bit=true was replaced with option.quantize=bitsandbytes8.

I did not know that vLLM no longer needs to install HuggingFace anymore. What else do I need to know? For reference, I have only been referencing "[GitHub] LMI Containers parameters after V7 release" to see what has changed.

sindhuvahinis commented 10 months ago

Upgrading dependencies is not as simple as version bump up. You need to carefully reevaluate the content of each notebook and see if some parameters need to be added/removed. For some examples, currently vLLM no longer needs to install HuggingFace anymore.

I did reevaluate the content of each notebook. For example, in tnx_rollingbatch_deploy_llama_7b_int8.ipynb, option.load_in_8bit=true was replaced with option.quantize=bitsandbytes8.

I did not know that vLLM no longer needs to install HuggingFace anymore. What else do I need to know? For reference, I have only been referencing "[GitHub] LMI Containers parameters after V7 release" to see what has changed.

for tnx, option.load_in_8bit should work and also option.quantize. Both should work because we support backward compatibility

ydm-amazon commented 10 months ago

Yes, but the document says option.load_in_8bit is deprecated since 0.25.0