The server is launched with HF_HUB_OFFLINE=1 and is meant to treat model files as read-only; however, the fast tokenizer conversion happening in the launcher does not follow this (if a revision is not passed). This can cause problems if a model in HF Hub is updated and the tokenizer conversion downloads the tokenizer files for the new commit of the model but then the server doesn't download the new model files... the server fails to load because it can't find the model files.
Modifications
Set local_files_only=True with and without the revision arg when doing the fast tokenizer conversion
Set HF_HUB_OFFLINE=1 in the env as well for good measure
Little refactoring to have the command building be shared
Result
Fast tokenizer conversion in the launcher should never download new files.
Motivation
The server is launched with
HF_HUB_OFFLINE=1and is meant to treat model files as read-only; however, the fast tokenizer conversion happening in thelauncherdoes not follow this (if arevisionis not passed). This can cause problems if a model in HF Hub is updated and the tokenizer conversion downloads the tokenizer files for the new commit of the model but then the server doesn't download the new model files... the server fails to load because it can't find the model files.Modifications
local_files_only=Truewith and without the revision arg when doing the fast tokenizer conversionHF_HUB_OFFLINE=1in the env as well for good measureResult
Fast tokenizer conversion in the launcher should never download new files.
Related Issues
local_files_onlyifrevisionis passed: https://github.com/IBM/text-generation-inference/pull/63