Open nivibilla opened 3 months ago
[2024-08-24 15:53:49,278] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [WARNING] async_io requires the dev libaio .so object and headers but these were not found. [WARNING] async_io: please install the libaio-dev package with apt [WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3 [WARNING] using untested triton version (2.3.1), only 1.0.0 is known to be compatible Starting DeepSpeed-MII instance for model /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/... Deployment name: mixtral-8x7b-instruct-v0.1 [2024-08-24 15:53:57,845] [INFO] [server.py:38:__init__] Hostfile /job/hostfile not found, creating hostfile. [2024-08-24 15:53:57,845] [INFO] [server.py:38:__init__] Hostfile /job/hostfile not found, creating hostfile. [2024-08-24 15:53:57,846] [INFO] [server.py:110:_launch_server_process] msg_server launch: ['deepspeed', '-i', 'localhost:0,1,2,3,4,5,6,7', '--master_port', '29500', '--master_addr', 'localhost', '--no_ssh_check', '--no_local_rank', '--no_python', '/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:53:57,846] [INFO] [server.py:110:_launch_server_process] msg_server launch: ['deepspeed', '-i', 'localhost:0,1,2,3,4,5,6,7', '--master_port', '29500', '--master_addr', 'localhost', '--no_ssh_check', '--no_local_rank', '--no_python', '/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:53:57,847] [INFO] [server.py:110:_launch_server_process] msg_server launch: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--load-balancer', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:53:57,847] [INFO] [server.py:110:_launch_server_process] msg_server launch: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--load-balancer', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:53:59,468] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-08-24 15:53:59,916] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [WARNING] async_io requires the dev libaio .so object and headers but these were not found. [WARNING] async_io: please install the libaio-dev package with apt [WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING] async_io requires the dev libaio .so object and headers but these were not found. [WARNING] async_io: please install the libaio-dev package with apt [WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3 [WARNING] using untested triton version (2.3.1), only 1.0.0 is known to be compatible [2024-08-24 15:54:02,848] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:54:02,848] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:54:03,585] [WARNING] [runner.py:202:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only. [2024-08-24 15:54:03,586] [INFO] [runner.py:568:main] cmd = /databricks/python3/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgMywgNCwgNSwgNiwgN119 --master_addr=127.0.0.1 --master_port=29500 --no_python --no_local_rank --enable_each_rank_log=None /local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python -m mii.launch.multi_gpu_server --deployment-name mixtral-8x7b-instruct-v0.1 --load-balancer-port 50050 --restful-gateway-port 51080 --restful-gateway-host localhost --restful-gateway-procs 32 --server-port 50051 --zmq-port 25555 --model-config eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9 [2024-08-24 15:54:05,442] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3 [WARNING] using untested triton version (2.3.1), only 1.0.0 is known to be compatible [WARNING] async_io requires the dev libaio .so object and headers but these were not found. [WARNING] async_io: please install the libaio-dev package with apt [WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. [WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3 [WARNING] using untested triton version (2.3.1), only 1.0.0 is known to be compatible [2024-08-24 15:54:07,848] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:54:07,848] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... Starting load balancer on port: 50050 About to start server Started [2024-08-24 15:54:10,210] [INFO] [launch.py:139:main] 0 NCCL_SOCKET_IFNAME=eth [2024-08-24 15:54:10,210] [INFO] [launch.py:146:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3, 4, 5, 6, 7]} [2024-08-24 15:54:10,210] [INFO] [launch.py:152:main] nnodes=1, num_local_procs=8, node_rank=0 [2024-08-24 15:54:10,210] [INFO] [launch.py:163:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3, 4, 5, 6, 7]}) [2024-08-24 15:54:10,210] [INFO] [launch.py:164:main] dist_world_size=8 [2024-08-24 15:54:10,211] [INFO] [launch.py:168:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 [2024-08-24 15:54:10,211] [INFO] [launch.py:256:main] process 44848 spawned with command: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:54:10,212] [INFO] [launch.py:256:main] process 44849 spawned with command: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:54:10,212] [INFO] [launch.py:256:main] process 44850 spawned with command: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:54:10,213] [INFO] [launch.py:256:main] process 44851 spawned with command: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:54:10,213] [INFO] [launch.py:256:main] process 44852 spawned with command: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:54:10,214] [INFO] [launch.py:256:main] process 44853 spawned with command: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] [2024-08-24 15:54:10,214] [INFO] [launch.py:256:main] process 44854 spawned with command: ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJx *** WARNING: max output size exceeded, skipping output. *** FO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:54:53,284] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00007-of-00019.safetensors [2024-08-24 15:54:53,654] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00007-of-00019.safetensors [2024-08-24 15:54:54,172] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00007-of-00019.safetensors [2024-08-24 15:54:54,447] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00013-of-00019.safetensors [2024-08-24 15:54:54,585] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00013-of-00019.safetensors [2024-08-24 15:54:55,147] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00014-of-00019.safetensors [2024-08-24 15:54:55,404] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00013-of-00019.safetensors [2024-08-24 15:54:55,502] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00007-of-00019.safetensors [2024-08-24 15:54:55,877] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00013-of-00019.safetensors [2024-08-24 15:54:56,042] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00013-of-00019.safetensors [2024-08-24 15:54:56,885] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00013-of-00019.safetensors [2024-08-24 15:54:57,371] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00014-of-00019.safetensors [2024-08-24 15:54:57,440] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00005-of-00019.safetensors [2024-08-24 15:54:57,590] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00014-of-00019.safetensors [2024-08-24 15:54:57,861] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:54:57,861] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:54:57,871] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00013-of-00019.safetensors [2024-08-24 15:54:58,092] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00014-of-00019.safetensors [2024-08-24 15:54:58,705] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00014-of-00019.safetensors [2024-08-24 15:54:58,819] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00014-of-00019.safetensors [2024-08-24 15:54:59,656] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00014-of-00019.safetensors [2024-08-24 15:55:00,017] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00005-of-00019.safetensors [2024-08-24 15:55:00,287] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00005-of-00019.safetensors [2024-08-24 15:55:00,711] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00005-of-00019.safetensors [2024-08-24 15:55:00,918] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00014-of-00019.safetensors [2024-08-24 15:55:01,250] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00005-of-00019.safetensors [2024-08-24 15:55:01,304] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00005-of-00019.safetensors [2024-08-24 15:55:02,212] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00005-of-00019.safetensors [2024-08-24 15:55:02,861] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:02,861] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:03,469] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00005-of-00019.safetensors [2024-08-24 15:55:04,154] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00016-of-00019.safetensors [2024-08-24 15:55:07,007] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00016-of-00019.safetensors [2024-08-24 15:55:07,144] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00016-of-00019.safetensors [2024-08-24 15:55:07,407] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00016-of-00019.safetensors [2024-08-24 15:55:07,862] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:07,862] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:08,656] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00016-of-00019.safetensors [2024-08-24 15:55:08,806] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00016-of-00019.safetensors [2024-08-24 15:55:08,812] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00015-of-00019.safetensors [2024-08-24 15:55:09,105] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00016-of-00019.safetensors [2024-08-24 15:55:10,825] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00016-of-00019.safetensors [2024-08-24 15:55:12,263] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00015-of-00019.safetensors [2024-08-24 15:55:12,275] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00015-of-00019.safetensors [2024-08-24 15:55:12,487] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00015-of-00019.safetensors [2024-08-24 15:55:12,863] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:12,863] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:13,154] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00015-of-00019.safetensors [2024-08-24 15:55:13,960] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00015-of-00019.safetensors [2024-08-24 15:55:14,207] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00015-of-00019.safetensors [2024-08-24 15:55:15,573] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00018-of-00019.safetensors [2024-08-24 15:55:15,870] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00015-of-00019.safetensors [2024-08-24 15:55:17,863] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:17,863] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:19,297] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00018-of-00019.safetensors [2024-08-24 15:55:19,557] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00018-of-00019.safetensors [2024-08-24 15:55:19,850] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00018-of-00019.safetensors [2024-08-24 15:55:20,426] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00018-of-00019.safetensors [2024-08-24 15:55:21,166] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00018-of-00019.safetensors [2024-08-24 15:55:21,210] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00018-of-00019.safetensors [2024-08-24 15:55:22,465] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00012-of-00019.safetensors [2024-08-24 15:55:22,864] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:22,864] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:23,444] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00018-of-00019.safetensors [2024-08-24 15:55:26,525] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00012-of-00019.safetensors [2024-08-24 15:55:26,546] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00012-of-00019.safetensors [2024-08-24 15:55:26,714] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00012-of-00019.safetensors [2024-08-24 15:55:26,953] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00012-of-00019.safetensors [2024-08-24 15:55:27,865] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:27,865] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:28,504] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00012-of-00019.safetensors [2024-08-24 15:55:28,522] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00012-of-00019.safetensors [2024-08-24 15:55:29,286] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00003-of-00019.safetensors [2024-08-24 15:55:29,777] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00012-of-00019.safetensors [2024-08-24 15:55:32,866] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:32,866] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:33,819] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00003-of-00019.safetensors [2024-08-24 15:55:34,124] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00008-of-00019.safetensors [2024-08-24 15:55:34,626] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00003-of-00019.safetensors [2024-08-24 15:55:34,664] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00003-of-00019.safetensors [2024-08-24 15:55:34,888] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00003-of-00019.safetensors [2024-08-24 15:55:35,127] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00003-of-00019.safetensors [2024-08-24 15:55:35,146] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00003-of-00019.safetensors [2024-08-24 15:55:37,568] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00003-of-00019.safetensors [2024-08-24 15:55:37,866] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:37,866] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:38,367] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00008-of-00019.safetensors [2024-08-24 15:55:39,525] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00008-of-00019.safetensors [2024-08-24 15:55:39,867] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00008-of-00019.safetensors [2024-08-24 15:55:39,948] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00008-of-00019.safetensors [2024-08-24 15:55:40,296] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00008-of-00019.safetensors [2024-08-24 15:55:40,866] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00008-of-00019.safetensors [2024-08-24 15:55:42,328] [INFO] [huggingface_engine.py:109:parameters] Loading checkpoint: /local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1/model-00008-of-00019.safetensors [2024-08-24 15:55:42,867] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:42,867] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:47,868] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:47,868] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:51,288] [INFO] [engine_v2.py:84:__init__] Model built. [2024-08-24 15:55:52,868] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:52,868] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:54,629] [INFO] [engine_v2.py:84:__init__] Model built. [2024-08-24 15:55:57,733] [INFO] [engine_v2.py:84:__init__] Model built. [2024-08-24 15:55:57,869] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:57,869] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:55:58,901] [INFO] [engine_v2.py:84:__init__] Model built. [2024-08-24 15:55:58,957] [INFO] [engine_v2.py:84:__init__] Model built. [2024-08-24 15:55:59,381] [INFO] [engine_v2.py:84:__init__] Model built. [2024-08-24 15:55:59,516] [INFO] [engine_v2.py:84:__init__] Model built. [2024-08-24 15:56:01,163] [INFO] [engine_v2.py:84:__init__] Model built. [2024-08-24 15:56:01,630] [INFO] [kv_cache.py:135:__init__] Allocating KV-cache 0 with shape: (32, 9659, 64, 2, 1, 128) consisting of 9659 blocks. [2024-08-24 15:56:01,630] [INFO] [kv_cache.py:135:__init__] Allocating KV-cache 0 with shape: (32, 9659, 64, 2, 1, 128) consisting of 9659 blocks. [2024-08-24 15:56:01,630] [INFO] [kv_cache.py:135:__init__] Allocating KV-cache 0 with shape: (32, 9659, 64, 2, 1, 128) consisting of 9659 blocks. [2024-08-24 15:56:01,630] [INFO] [kv_cache.py:135:__init__] Allocating KV-cache 0 with shape: (32, 9659, 64, 2, 1, 128) consisting of 9659 blocks. [2024-08-24 15:56:01,630] [INFO] [kv_cache.py:135:__init__] Allocating KV-cache 0 with shape: (32, 9659, 64, 2, 1, 128) consisting of 9659 blocks. [2024-08-24 15:56:01,630] [INFO] [kv_cache.py:135:__init__] Allocating KV-cache 0 with shape: (32, 9659, 64, 2, 1, 128) consisting of 9659 blocks. [2024-08-24 15:56:01,630] [INFO] [kv_cache.py:135:__init__] Allocating KV-cache 0 with shape: (32, 9659, 64, 2, 1, 128) consisting of 9659 blocks. [2024-08-24 15:56:01,630] [INFO] [kv_cache.py:135:__init__] Allocating KV-cache 0 with shape: (32, 9659, 64, 2, 1, 128) consisting of 9659 blocks. Starting server on port: 50055 About to start server Starting server on port: 50054 Starting server on port: 50057 Starting server on port: 50058 Started Starting server on port: 50052 Starting server on port: 50056 About to start server About to start server About to start server Started Started Starting server on port: 50053 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR E0000 00:00:1724514961.707826 44855 chttp2_server.cc:1118] UNKNOWN:No address added out of total 1 resolved for '[::]:50058' {created_time:"2024-08-24T15:56:01.707823091+00:00", children:[UNKNOWN:Failed to add any wildcard listeners {created_time:"2024-08-24T15:56:01.707808181+00:00", children:[UNKNOWN:Unable to configure socket {fd:121, created_time:"2024-08-24T15:56:01.70776102+00:00", children:[UNKNOWN:bind: Address already in use (98) {created_time:"2024-08-24T15:56:01.707724179+00:00"}]}, UNKNOWN:Unable to configure socket {fd:121, created_time:"2024-08-24T15:56:01.707805861+00:00", children:[UNKNOWN:bind: Address already in use (98) {created_time:"2024-08-24T15:56:01.70780179+00:00"}]}]}]} Started [rank7]: Traceback (most recent call last): [rank7]: File "<frozen runpy>", line 198, in _run_module_as_main [rank7]: File "<frozen runpy>", line 88, in _run_code [rank7]: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/mii/launch/multi_gpu_server.py", line 105, in <module> [rank7]: main() [rank7]: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/mii/launch/multi_gpu_server.py", line 100, in main [rank7]: serve_inference(inference_pipeline, port) [rank7]: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/mii/grpc_related/modelresponse_server.py", line 291, in serve_inference [rank7]: _do_serve(ModelResponse(async_pipeline=async_pipeline), port) [rank7]: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/mii/grpc_related/modelresponse_server.py", line 281, in _do_serve [rank7]: server.add_insecure_port(f"[::]:{port}") [rank7]: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/grpc/_server.py", line 1473, in add_insecure_port [rank7]: return _common.validate_port_binding_result( [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank7]: File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/grpc/_common.py", line 181, in validate_port_binding_result [rank7]: raise RuntimeError(_ERROR_MESSAGE_PORT_BINDING_FAILED % address) [rank7]: RuntimeError: Failed to bind to address [::]:50058; set GRPC_VERBOSITY=debug environment variable to see detailed error message. About to start server Started About to start server Started Starting server on port: 50051 About to start server [2024-08-24 15:56:02,870] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:56:02,870] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... Started [2024-08-24 15:56:03,243] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 44848 [2024-08-24 15:56:03,630] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 44849 [2024-08-24 15:56:04,007] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 44850 [2024-08-24 15:56:04,385] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 44851 [2024-08-24 15:56:04,805] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 44852 [2024-08-24 15:56:05,183] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 44853 [2024-08-24 15:56:05,603] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 44854 [2024-08-24 15:56:06,060] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 44855 [2024-08-24 15:56:06,060] [ERROR] [launch.py:325:sigkill_handler] ['/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/bin/python', '-m', 'mii.launch.multi_gpu_server', '--deployment-name', 'mixtral-8x7b-instruct-v0.1', '--load-balancer-port', '50050', '--restful-gateway-port', '51080', '--restful-gateway-host', 'localhost', '--restful-gateway-procs', '32', '--server-port', '50051', '--zmq-port', '25555', '--model-config', 'eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAiL2xvY2FsX2Rpc2swL21pc3RyYWxhaS9NaXh0cmFsLTh4N0ItSW5zdHJ1Y3QtdjAuMS8iLCAidG9rZW5pemVyIjogIi9sb2NhbF9kaXNrMC9taXN0cmFsYWkvTWl4dHJhbC04eDdCLUluc3RydWN0LXYwLjEvIiwgInRhc2siOiAidGV4dC1nZW5lcmF0aW9uIiwgInRlbnNvcl9wYXJhbGxlbCI6IDgsICJxdWFudGl6YXRpb25fbW9kZSI6IG51bGwsICJpbmZlcmVuY2VfZW5naW5lX2NvbmZpZyI6IHsidGVuc29yX3BhcmFsbGVsIjogeyJ0cF9zaXplIjogOH0sICJzdGF0ZV9tYW5hZ2VyIjogeyJtYXhfdHJhY2tlZF9zZXF1ZW5jZXMiOiAyMDQ4LCAibWF4X3JhZ2dlZF9iYXRjaF9zaXplIjogNzY4LCAibWF4X3JhZ2dlZF9zZXF1ZW5jZV9jb3VudCI6IDUxMiwgIm1heF9jb250ZXh0IjogODE5MiwgIm1lbW9yeV9jb25maWciOiB7Im1vZGUiOiAicmVzZXJ2ZSIsICJzaXplIjogMTAwMDAwMDAwMH0sICJvZmZsb2FkIjogZmFsc2V9LCAicXVhbnRpemF0aW9uIjogeyJxdWFudGl6YXRpb25fbW9kZSI6IG51bGx9fSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiem1xX3BvcnRfbnVtYmVyIjogMjU1NTUsICJyZXBsaWNhX251bSI6IDEsICJyZXBsaWNhX2NvbmZpZ3MiOiBbeyJob3N0bmFtZSI6ICJsb2NhbGhvc3QiLCAidGVuc29yX3BhcmFsbGVsX3BvcnRzIjogWzUwMDUxLCA1MDA1MiwgNTAwNTMsIDUwMDU0LCA1MDA1NSwgNTAwNTYsIDUwMDU3LCA1MDA1OF0sICJ0b3JjaF9kaXN0X3BvcnQiOiAyOTUwMCwgImdwdV9pbmRpY2VzIjogWzAsIDEsIDIsIDMsIDQsIDUsIDYsIDddLCAiem1xX3BvcnQiOiAyNTU1NX1dLCAiZGV2aWNlX21hcCI6ICJhdXRvIiwgIm1heF9sZW5ndGgiOiA4MTkyLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9'] exits with return code = 1 [2024-08-24 15:56:07,871] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... [2024-08-24 15:56:07,871] [INFO] [server.py:68:_wait_until_server_is_live] waiting for server to start... Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/mii/entrypoints/openai_api_server.py", line 506, in <module> mii.serve(app_settings.model_id, File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/mii/api.py", line 179, in serve import_score_file(mii_config.deployment_name, DeploymentType.LOCAL).init() File "/tmp/mii_cache/mixtral-8x7b-instruct-v0.1/score.py", line 33, in init mii.backend.MIIServer(mii_config) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/mii/backend/server.py", line 50, in __init__ self._wait_until_server_is_live(processes, File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-e7d2e809-50e6-43c4-baee-991bca4eecca/lib/python3.11/site-packages/mii/backend/server.py", line 65, in _wait_until_server_is_live raise RuntimeError( RuntimeError: server crashed for some reason, unable to proceed
Model seems to load but then server fails to start.
same problem
mori173
commented
1 day ago mori173
commented
1 day ago
Model seems to load but then server fails to start.