AttributeError: 'MixtralLayer' object has no attribute 'mlp'

[icyxp]

System Info

2024-06-26T08:59:14.473641Z ERROR text_generation_launcher: Error when initializing model Traceback (most recent call last): File "/opt/conda/bin/text-generation-server", line 8, in sys.exit(app()) File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 311, in call return get_command(self)(*args, kwargs) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 778, in main return _main( File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 216, in _main rv = self.invoke(ctx) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(args, kwargs) File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 683, in wrapper return callback(*use_params) # type: ignore File "/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py", line 106, in serve server.serve( File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 297, in serve asyncio.run( File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete self.run_forever() File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, self._args)

File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 231, in serve_inner model = get_model( File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/init.py", line 745, in get_model return FlashMixtral( File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/flash_mixtral.py", line 22, in init super(FlashMixtral, self).init( File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/flash_mistral.py", line 97, in init super().init( File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/flash_causal_lm.py", line 818, in init super(FlashCausalLM, self).init( File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/model.py", line 63, in init self.target_to_layer = self.adapter_target_to_layer() File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/flash_mistral.py", line 156, in adapter_target_to_layer if hasattr(layer.mlp, "gate_up_proj"): File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1709, in getattr raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'") AttributeError: 'MixtralLayer' object has no attribute 'mlp'

Information

• [ ] Docker
• [ ] The CLI directly

Tasks

• [ ] An officially supported command
• [ ] My own modifications

Reproduction

branch main

Expected behavior

none

[LysandreJik]

Thanks for opening a PR!