[22/01/2024 - 29/01/2024] New Models/Endpoints/Providers/Improvements

[ishaan-jaff]

Budget Management

• [x] Add testing for proxy_server.track_cost_callback - writing to SpendLogs Table https://github.com/BerriAI/litellm/issues/1584
• [x] Allow Users to Set Budgets per user per month https://github.com/BerriAI/litellm/issues/1544
• [x] See (By User) spend this month https://github.com/BerriAI/litellm/issues/1545
• [ ] See (By User) to see total requests, Total tokens used (completion, embedding), spend,
• [x] Set Global Budget For the Proxy + Reset the budget https://github.com/BerriAI/litellm/issues/1585
• [x] Allow key budgets to be reset https://github.com/BerriAI/litellm/issues/1567
• [x] Update Budgets per user (allow users to reset budgets) https://github.com/BerriAI/litellm/issues/1546
• [x] Allow setting budgets per key https://github.com/BerriAI/litellm/issues/1548
• [x] Allow setting tpm/rpm limits per key https://github.com/BerriAI/litellm/issues/1550
• [x] Slack Alerts for Budgets https://github.com/BerriAI/litellm/issues/1612
• [ ] End User Emails when a key is near its budget

Spend Tracking

• [x] Support input/output cost per second - custom pricing https://github.com/BerriAI/litellm/issues/1374
• [x] Emit spend as part of logging call https://github.com/BerriAI/litellm/issues/1547
• [x] Log token usage in spend logs https://github.com/BerriAI/litellm/issues/1549
• [x] Get spend logs by key https://github.com/BerriAI/litellm/issues/1582
• [x] Allow specifying key alias https://github.com/BerriAI/litellm/issues/1624
• [x] Support bedrock region pricing in completion_cost - https://github.com/BerriAI/litellm/issues/1565
• [x] Completion cost support for image gen models - https://github.com/BerriAI/litellm/issues/1558
• [ ] add Region-specific amazon text embedding pricing - https://github.com/BerriAI/litellm/issues/1566
• [ ] Allow local model cost maps for proxy https://github.com/BerriAI/litellm/issues/1434
• [ ] Allow users to access litellm ui + proxy using docker compose up https://github.com/BerriAI/litellm/issues/1631

Misc

• [x] real Sagemaker streaming support - https://github.com/BerriAI/litellm/issues/1384
• [ ] Gemini Pro - VertexAI function calling when streaming support - https://github.com/BerriAI/litellm/issues/1263
• [x] Allow users to add their own logic to /key/generate https://github.com/BerriAI/litellm/issues/1462
• [ ] Caching improvement for embedding calls - https://github.com/BerriAI/litellm/issues/1350
• [x] Make Authentication Errors OpenAI compatible: https://github.com/BerriAI/litellm/issues/1554
• [x] Auto infer known model names for health checks: https://github.com/BerriAI/litellm/issues/1580
• [x] Support vertex project and location being set dynamically: https://github.com/BerriAI/litellm/issues/1613
• [ ] Reduce proxy cold start times - https://github.com/BerriAI/litellm/issues/1636

[krrishdholakia]

Known Backlog:

1. OpenMeter spend tracking support - https://github.com/BerriAI/litellm/issues/1268
2. tracking - https://github.com/BerriAI/litellm/issues/997
3. tracking - https://github.com/BerriAI/litellm/issues/1188 cc: @r1cc4rd0m4zz4
4. tracking - https://github.com/BerriAI/litellm/issues/1192
5. yandexgpt support requested by 2 users - https://github.com/BerriAI/litellm/issues/1254
6. Aphrodite engine endpoint - https://github.com/PygmalionAI/aphrodite-engine, https://github.com/BerriAI/litellm/pull/1153
7. TheBloke/NexusRaven-V2-13B-GGUF function calling support https://github.com/BerriAI/litellm/issues/1060 cc: @homanp
8. Predibase lorax support - [Feature]: Add support for Predibase Lorax apis #1253
9. Deepsparse CPU inference support - https://github.com/neuralmagic/deepsparse/issues/1452
10. Consistent format for model prices and context window - https://github.com/BerriAI/litellm/issues/1375
11. Vertex AI model garden support - https://github.com/BerriAI/litellm/issues/920

[ishaan-jaff]

New one here: https://github.com/BerriAI/litellm/issues/1665