Open RdoubleA opened 1 week ago
See #1995 for more context. Just need to add a comment block here with the conclusions: https://github.com/pytorch/torchtune/blob/main/torchtune/models/llama3/_tokenizer.py#L29
See #1995 for more context. Just need to add a comment block here with the conclusions: https://github.com/pytorch/torchtune/blob/main/torchtune/models/llama3/_tokenizer.py#L29