DeepSpeed

[ccmaymay]

w/ bf16

[ccmaymay]

https://github.com/microsoft/DeepSpeed-MII#deploying-mii-public

Has grpc server, seems to resemble our own https://github.com/microsoft/DeepSpeed-MII/blob/main/mii/server.py for example:

        # XXX: fetch model provider based on model name in a more general way
        if model_name == "gpt-neox":
            provider = mii.constants.MODEL_PROVIDER_NAME_EA
        elif ("bigscience/bloom" == model_name) or ("microsoft/bloom" in model_name):
            provider = mii.constants.MODEL_PROVIDER_NAME_HF_LLM
        elif self.task == mii.Tasks.TEXT2IMG:
            provider = mii.constants.MODEL_PROVIDER_NAME_DIFFUSERS
        else:
            provider = mii.constants.MODEL_PROVIDER_NAME_HF
        server_args_str += f" --provider {provider}"

[ccmaymay]

Branch status: event loop not found at query time

[ccmaymay]

Looks like DeepSpeed-MII recently removed the Bloom-specific LLM functionality and replaced it with a generalized "meta tensor" functionality: https://github.com/microsoft/DeepSpeed-MII/commit/60a85dc3da5bac3bcefa8824175f8646a0f12203

may be worth trying out again