Label

[yeonju7kim]

https://github.com/Alpha-VLLM/LLaMA2-Accessory/blob/a0b87081e8476e2a857302c202ec7bf51fd3f38f/accessory/data/alpaca.py#L118

I thought that this should be

labels[:len(input1) - 1] = -1

[ChrisLiu6]

I guess you are considering that the output of the token with index [len(input1) - 1] should already be taken into account for loss calculation. This is true, but please consider here:

https://github.com/Alpha-VLLM/LLaMA2-Accessory/blob/a0b87081e8476e2a857302c202ec7bf51fd3f38f/accessory/model/meta.py#L86

The central point here is that the label should be one token ahead of input.

[yeonju7kim]

When I debugged this part, I found that the label doesn't contain one starting token of ground truth. For example, if the ground truth is "This is a cat", the label is " is a cat".

[ChrisLiu6]

Given a sequence

<bos> q1 q2 q3 a1 a2 a3 <eos> # q for question and a for answer

by design, we want to assign the variables as follows:

input1 = [<bos> q1 q2 q3]
input2 = [<bos> q1 q2 q3 a1 a2 a3 <eos>]
label   = [0 0 0 0 0 a1 a2 a3 <eos>]

so that only the answer will be considered for loss computation. The code you refer to should be correct for doing this:

labels[:len(input1)] = -1 

Then, during loss computation, the target value for the output of each input token should be the next input token. So we move the label list to the left by one token https://github.com/Alpha-VLLM/LLaMA2-Accessory/blob/a0b87081e8476e2a857302c202ec7bf51fd3f38f/accessory/model/meta.py#L86

Please let us know if there's any bug in this process.