Open mantle2048 opened 5 months ago
transformers
No response
examples
I noticed that in the implementation of flax-llama, the return value of create_sinusoidal_positions was truncated by num_pos.
num_pos
def create_sinusoidal_positions(num_pos, dim): inv_freq = 1.0 / (10000 ** (np.arange(0, dim, 2) / dim)) freqs = np.einsum("i , j -> i j", np.arange(num_pos), inv_freq).astype("float32") emb = np.concatenate((freqs, freqs), axis=-1) out = np.concatenate((np.sin(emb)[:, None, :], np.cos(emb)[:, None, :]), axis=-1) return jnp.array(out[:, :, :num_pos]) # Why?
Why is this? Is it to prevent the sequence length from being too short?
See above.
cc @sanchit-gandhi
Gentle ping @sanchit-gandhi
Another ping @sanchit-gandhi
Another another ping @sanchit-gandhi. Could you nominate someone to take over this whilst you're away?
System Info
transformers
version: 4.38.2Who can help?
No response
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
I noticed that in the implementation of flax-llama, the return value of create_sinusoidal_positions was truncated by
num_pos
.Why is this? Is it to prevent the sequence length from being too short?
Expected behavior
See above.