neuralmagic / nm-vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://nm-vllm.readthedocs.io
Other
251 stars 10 forks source link

Force-disable upstream tracking #321

Closed dbarbuzzi closed 4 months ago

dbarbuzzi commented 4 months ago

This PR force-disables upstream tracking by forcing VLLM_DO_NOT_TRACK and VLLM_NO_USAGE_STATS to be True (disable data collection) rather than inspecting the environment variable.

An alternative approach could be to update the definition of is_usage_stats_enabled (in vllm/usage/usage_lib.py:29) to always return False and set the global _USAGE_STATS_ENABLED to be False.

dbarbuzzi commented 4 months ago

Seems like there was a single failure in 3.11:

https://neuralmagic.testmo.net/automation/runs/results/20174?group_by=automation_run_tests:status_id&group_id=3&test_id=31649686

dhuangnm commented 4 months ago

Should the default values be True too?

VLLM_NO_USAGE_STATS: bool = False VLLM_DO_NOT_TRACK: bool = False

dbarbuzzi commented 4 months ago

Should the default values be True too?

That section is purely for type-checking (as in, it is never actually executed because it’s protected by the if TYPE_CHECKING clause), however, I think this is a good change for clarity, so I pushed it.