Closed peterbjorgensen closed 6 months ago
The modules show up with dolma list --tagger_modules mypackage.mymodule but it crashes if you do dolma tag --tagger_modules mypackage.mymodule ... The problem is that the tagger modules are not loaded before this part of the code which instantiates the taggers by name https://github.com/allenai/dolma/blob/e7657e473ec3b46f2c98cd8f61dfab3955ba4755/python/dolma/core/runtime.py#L428-L433
dolma list --tagger_modules mypackage.mymodule
dolma tag --tagger_modules mypackage.mymodule ...
This means the dolma tag commands crashes with ValueError: Unknown tagger mytagger ...
dolma tag
ValueError: Unknown tagger mytagger ...
Excellent catch!! fixed in main.
The modules show up with
dolma list --tagger_modules mypackage.mymodule
but it crashes if you dodolma tag --tagger_modules mypackage.mymodule ...
The problem is that the tagger modules are not loaded before this part of the code which instantiates the taggers by name https://github.com/allenai/dolma/blob/e7657e473ec3b46f2c98cd8f61dfab3955ba4755/python/dolma/core/runtime.py#L428-L433This means the
dolma tag
commands crashes withValueError: Unknown tagger mytagger ...