search

ludwig-ai / ludwig

Low-code framework for building custom LLMs, neural networks, and other AI models
http://ludwig.ai
Apache License 2.0
10.97k stars 1.18k forks source link

Torchtext undefined module when using gpt2bpe tokenizer #4013

Open jhudsy opened 3 weeks ago

jhudsy commented 3 weeks ago

When trying to use the gpt2bpe tokenizer, I get an error that "module torchtext has no attribute util".

Fixed by adding

import torchtext.utils
import torchtext.transforms

to ludwig/utils/tokenizers.py