Closed iCSawyer closed 1 year ago
"
, '
, """
, '''
, r'
, b'
, etc., we preserve such characters. Besides, our evaluation metric is token-level, not subtoken-level. "<STR_LIT>"
is treated as one token during evaluation even though the model has to predict three subtokens.I have some doubt about token level accuracy calculation.
The inputs
of the model are N=batch size encoded string of length L=block size. The output
, instead, are N=batch size predictions of the next token or the input string are concatenated with the output?
In each of the two case, i don't understand how the evaluation works.
I have 2 questions about accuracy calculation in token-level code completion:
<STR_LIT>
and<CHAR_LIT:E>
are included (or excluded) in the process of accuracy calculation. I have read functioneval_acc()
inrun_lm.py
and it seems that they are all included (except for<NUM_LIT>
).<STR_LIT>
or<STR_LIT:EE>
, the preprocessing will add"
to the start and the end of<STR_LIT>
, which is different from<CHAR_LIT>
and<NUM_LIT>
. When we usetransformers.GPT2Tokenizer
to tokenize it,"<STR_LIT>"
will be splitted into 3 sub-tokens"
,<STR_LIT>
and"
but actually there is just one token - the origin string. This may lead to some bad influence to the accuray calculation and the inference.Thank you for your reply!