DRBX Instruct TGI error

[deter3]

"text_generation_launcher: Method Warmup encountered an error." when the final stage .

2024-03-30T14:16:55.598106565Z 2024-03-30T14:16:55.597709Z ERROR warmup{max_input_length=3000 max_prefill_tokens=4096 max_total_tokens=4096 max_batch_size=None}:warmup: text_generation_client: router/client/src/lib.rs:33: Server error: 
2024-03-30T14:16:55.600818375Z 2024-03-30T14:16:55.600506Z ERROR warmup{max_input_length=3000 max_prefill_tokens=4096 max_total_tokens=4096 max_batch_size=None}:warmup: text_generation_client: router/client/src/lib.rs:33: Server error: 
2024-03-30T14:16:55.617498562Z 2024-03-30T14:16:55.617195Z ERROR text_generation_launcher: Method Warmup encountered an error.
2024-03-30T14:16:55.617524982Z Traceback (most recent call last):
2024-03-30T14:16:55.617531202Z   File "/opt/conda/bin/text-generation-server", line 8, in <module>
2024-03-30T14:16:55.617535710Z     sys.exit(app())
2024-03-30T14:16:55.617540271Z   File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 311, in __call__
2024-03-30T14:16:55.617544970Z     return get_command(self)(*args, **kwargs)
2024-03-30T14:16:55.617549281Z   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
2024-03-30T14:16:55.617553441Z     return self.main(*args, **kwargs)
2024-03-30T14:16:55.617562110Z   File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 778, in main
2024-03-30T14:16:55.617566300Z     return _main(
2024-03-30T14:16:55.617571341Z   File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 216, in _main
2024-03-30T14:16:55.617575521Z     rv = self.invoke(ctx)
2024-03-30T14:16:55.617579817Z   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
2024-03-30T14:16:55.617583801Z     return _process_result(sub_ctx.command.invoke(sub_ctx))
2024-03-30T14:16:55.617587801Z   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
2024-03-30T14:16:55.617591724Z     return ctx.invoke(self.callback, **ctx.params)
2024-03-30T14:16:55.617595687Z   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke
2024-03-30T14:16:55.617599619Z     return __callback(*args, **kwargs)
2024-03-30T14:16:55.617603691Z   File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 683, in wrapper
2024-03-30T14:16:55.617607619Z     return callback(**use_params)  # type: ignore
2024-03-30T14:16:55.617611654Z   File "/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py", line 89, in serve
2024-03-30T14:16:55.617624529Z     server.serve(
2024-03-30T14:16:55.617628956Z   File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 235, in serve
2024-03-30T14:16:55.617633120Z     asyncio.run(
2024-03-30T14:16:55.617637563Z   File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run
2024-03-30T14:16:55.617641610Z     return loop.run_until_complete(main)
2024-03-30T14:16:55.617645743Z   File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete
2024-03-30T14:16:55.617649880Z     self.run_forever()
2024-03-30T14:16:55.617654066Z   File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
2024-03-30T14:16:55.617658169Z     self._run_once()
2024-03-30T14:16:55.617662343Z   File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
2024-03-30T14:16:55.617666336Z     handle._run()
2024-03-30T14:16:55.617778549Z     return self._call_impl(*args, **kwargs)
2024-03-30T14:16:55.617783007Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2024-03-30T14:16:55.617787154Z     return forward_call(*args, **kwargs)
2024-03-30T14:16:55.617792239Z   File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_dbrx_modeling.py", line 1001, in forward
2024-03-30T14:16:55.617796394Z     hidden_states, residual = layer(
2024-03-30T14:16:55.617800519Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2024-03-30T14:16:55.617808014Z     return self._call_impl(*args, **kwargs)
2024-03-30T14:16:55.617812474Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2024-03-30T14:16:55.617816571Z     return forward_call(*args, **kwargs)
2024-03-30T14:16:55.617820988Z   File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_dbrx_modeling.py", line 949, in forward
2024-03-30T14:16:55.617825137Z     moe_output = self.moe(attn_output)
2024-03-30T14:16:55.617829191Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2024-03-30T14:16:55.617833331Z     return self._call_impl(*args, **kwargs)
2024-03-30T14:16:55.617837518Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2024-03-30T14:16:55.617841787Z     return forward_call(*args, **kwargs)
2024-03-30T14:16:55.617845938Z   File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_dbrx_modeling.py", line 807, in forward
2024-03-30T14:16:55.617849988Z     return self.sparse_forward(x)
2024-03-30T14:16:55.617854297Z   File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_dbrx_modeling.py", line 706, in sparse_forward
2024-03-30T14:16:55.617858468Z     topo = self.topology(x, padded_bins)
2024-03-30T14:16:55.617862638Z   File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_dbrx_modeling.py", line 593, in topology
2024-03-30T14:16:55.617866838Z     assert self.ffn_dim % self.blocking == 0
2024-03-30T14:16:55.617871051Z AssertionError

[RonanKMcGovern]

What GPUs did you use?

4x a100 sxm should work

On Sat 30 Mar 2024 at 14:19, deter3 @.***> wrote:

"text_generation_launcher: Method Warmup encountered an error." when the final stage .

2024-03-30T14:16:55.617498562Z 2024-03-30T14:16:55.617195Z ERROR text_generation_launcher: Method Warmup encountered an error. 2024-03-30T14:16:55.617524982Z Traceback (most recent call last): 2024-03-30T14:16:55.617531202Z File "/opt/conda/bin/text-generation-server", line 8, in 2024-03-30T14:16:55.617535710Z sys.exit(app()) 2024-03-30T14:16:55.617540271Z File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 311, in call 2024-03-30T14:16:55.617544970Z return get_command(self)(*args, kwargs) 2024-03-30T14:16:55.617549281Z File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in call 2024-03-30T14:16:55.617553441Z return self.main(args, kwargs) 2024-03-30T14:16:55.617562110Z File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 778, in main 2024-03-30T14:16:55.617566300Z return _main( 2024-03-30T14:16:55.617571341Z File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 216, in _main 2024-03-30T14:16:55.617575521Z rv = self.invoke(ctx) 2024-03-30T14:16:55.617579817Z File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke 2024-03-30T14:16:55.617583801Z return _process_result(sub_ctx.command.invoke(sub_ctx)) 2024-03-30T14:16:55.617587801Z File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke 2024-03-30T14:16:55.617591724Z return ctx.invoke(self.callback, ctx.params) 2024-03-30T14:16:55.617595687Z File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke 2024-03-30T14:16:55.617599619Z return __callback(args, kwargs)

— Reply to this email directly, view it on GitHub https://github.com/TrelisResearch/one-click-llms/issues/5, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASVG6CX72FFCB3UE6GRVXNLY23CV7AVCNFSM6AAAAABFPS6DX6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYTMNBYGMZDQMA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[deter3]

8XA6000 .

[RonanKMcGovern]

Looks like this issue was patched. Please comment here if you do try it.