Update max num tokens workflow, fix multiple bugs - Githubissues

deepjavalibrary / djl-serving

A universal scalable machine learning model deployment solution

Apache License 2.0

183 stars 58 forks source link

Update max num tokens workflow, fix multiple bugs #2012

Closed ydm-amazon closed 1 month ago

ydm-amazon commented 1 month ago

This is the branch I was using to run all the max_num_tokens numbers for 0.28.0; it contains lots of small changes.

double check cleaning model artifacts directory
multiple tries for model downloading
fallback to aws s3 cp from s5cmd
error handling for cases of non OOM error
better logging
timeout increased
increase shared memory size
mount to nvme instead of EBS volume
add g6.12xlarge