Closed XingxingZhang closed 1 year ago
In LM decoding with prefix (e.g., prompt), we can compute all prefix hidden states all together in the first step by setting incremental_state["is_first_step"] = True
incremental_state["is_first_step"] = True
In LM decoding with prefix (e.g., prompt), we can compute all prefix hidden states all together in the first step by setting
incremental_state["is_first_step"] = True