search

michaelfeil / infinity

Infinity is a high-throughput, low-latency REST API for serving text-embeddings, reranking models and clip
https://michaelfeil.github.io/infinity/
MIT License
1.31k stars 96 forks source link

When I installed infinity using poetry, it was easy to conflict with the torch versions of other frameworks because the torch version was too high. #263

Closed shell-nlp closed 3 months ago

shell-nlp commented 3 months ago

Feature request

When I installed infinity using poetry, it was easy to conflict with the torch versions of other frameworks because the torch version was too high.

Motivation

When I installed infinity using poetry, it was easy to conflict with the torch versions of other frameworks because the torch version was too high.

Your contribution

When I installed infinity using poetry, it was easy to conflict with the torch versions of other frameworks because the torch version was too high.

michaelfeil commented 3 months ago

Hey @shell-nlp,

Whats the range of torch versions you are looking for?

2.2+ is required to enable torch.compile with dynamic shapes + torch.nested_tensor and 2.3 for e.g. fp8 / quant. there might be a lot of things breaking, and I dont want to maintain / be responsible for those!

Can you help me to see which packages you mean so I can evaluate what is possible.

shell-nlp commented 3 months ago

image The prometheus-fastapi-instrumentator version needs to be upgraded to >=7.0

michaelfeil commented 3 months ago

Ah, so its not related on torch. Working on it tonight.

michaelfeil commented 3 months ago

Should be done with 0.0.43 (main branch!)