-
Hello, thank you for a great project!
I am getting this error when using ALiBi or RoPe positional encoding in a tranformer NMT model from OpenNMT-py:
KeyError: 'encoder.embeddings.make_embedding…
-
- https://arxiv.org/abs/2106.12566
- 2021
Transformerの重要なコンポーネントであるAttentionモジュールは、その2次的な複雑さのために、長いシーケンスに対して効率的に拡張することができません。
多くの研究は、オリジナルのアテンションにおけるdot-then-exponentiate softmax関数の近似に焦点を当てており、2次…
e4exp updated
3 years ago
-
Hi, Great work. Benefit a lot from kaolin!
I've learned the paper "DMTet" which describes PVCNN as the input encoder. However, I only find "MLP + positional encoding" in kaolin implementation.
…
-
The positional encoding matrix as shown in equation (7) in the paper has not considered in the class Global_SAN for defining the self-attentive network. I think without this, there is no mechanism to …
-
I see, thank you for providing more context. Let me summarize my understanding of your approach:
1. You've flattened JSON representations of ASTs into a tabular format.
2. This table has many colu…
-
Hi,there! It seems that the positional encoding in SAM is too complex for torch_pruning.
[https://github.com/czg1225/SlimSAM/issues/10](url)
In fact, there is an infinite loop at the **_fix_dependen…
-
In your paper, you say
> Position Interpolation (PI, [Chen
et al., 2023] and [kaiokendev, 2023]) introduces a modification to the rotary positional encoding
scheme that enables fine-tuning for 32K…
-
### Your current environment
Name: vllm
Version: 0.6.3.post2.dev171+g890ca360
### Model Input Dumps
_No response_
### 🐛 Describe the bug
I used the interface from this vllm repository …
-
Hey，i noticed that you did not use the positional encoding in the model but the orginal Transformer Model used the triangle positional encoding, why did not you use that ? was the PE useless?
-
### Is there an existing issue for this?
- [X] I have searched the existing issues
### Current Behavior
So as what I am seeing. For the standard MultiHeadAttention, there is a procedure:
input -> …