Open seanmor5 opened 7 months ago
Related to https://github.com/elixir-nx/axon/issues/549
LoRA targets QKV nodes. Right now the only way to do it is by inferring based on layer name. We can provide more information by giving specific metadata (such as op_name) for which layers are QKV
Related to https://github.com/elixir-nx/axon/issues/549
LoRA targets QKV nodes. Right now the only way to do it is by inferring based on layer name. We can provide more information by giving specific metadata (such as op_name) for which layers are QKV