Closed wenxindongwork closed 1 month ago
The causal_lm_preprocessor returns x as an dict of {"token_ids":Array, "padding_mask":Array} so calling ops.convert_to_tensor(x) will fail. Usingx = tree.map_structure(ops.convert_to_tensor, x) resolves the problem.
causal_lm_preprocessor
ops.convert_to_tensor(x)
x = tree.map_structure(ops.convert_to_tensor, x)
The
causal_lm_preprocessorreturns x as an dict of {"token_ids":Array, "padding_mask":Array} so callingops.convert_to_tensor(x)will fail. Usingx = tree.map_structure(ops.convert_to_tensor, x)resolves the problem.