aaron-sandoval commented 7 months ago

This PR deprecates MazeTokenizer and adds a substitute architecture which is more powerful and easier to build upon to create new tokenizers.

review-notebook-app[bot] commented 7 months ago

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

aaron-sandoval commented 6 months ago

I couldn't come up with a better name for MazeTokenizer2. The best idea that Claude had IMO was MazeTokenizerPlus, but this seemed no better and no worse to me. Bigger name changes like MazePreprocessor seemed to obfuscate that this is a direct replacement of MazeTokenizer. Lmk if you have a better idea.

aaron-sandoval commented 6 months ago

zanj save/read

I forgot to ask about this at our last meeting, but I should highlight this for your review
I'm pretty sure my implementation of maze_dataset._load_tokenizer_element for saving/reading is at least a little bit wrong
It currently seems to work fine, but I know I'm not using zanj quite right
I see the following possibilities
1. Minor update: Only line 924 needs to be refactored using zanj.loading.load_item_recursive to properly leverage zanj and ensure scalability to non-primitive args
2. Major update: This whole function is unnecessary/misguided and there's a much simpler/different way to use zanj to read in these tokenizers

understanding-search / maze-dataset

Add Flexible `MazeTokenizer` Alternative #35

zanj save/read