Add tokenization schemes - Githubissues

understanding-search / maze-dataset

maze datasets for investigating OOD behavior of ML systems

16 stars 3 forks source link

Add tokenization schemes #30

Closed aaron-sandoval closed 2 months ago

aaron-sandoval commented 8 months ago

Add new tokenization schemes.

### Tasks
- [ ] #31
- [ ] https://github.com/understanding-search/maze-dataset/pull/35
- [ ] https://github.com/understanding-search/maze-dataset/pull/37
- [ ] https://github.com/understanding-search/maze-dataset/pull/38

`TokenizationMode.AOTP_indexed`

Needs testing #31
Compass Direction Path Scheme
Represent paths as a sequence of tokens indicating the direction to move from the current cell rather than a representation of the coordinates of an adjacent cell
I.e., north, south, east, west, stay
Relative Direction Path Scheme
Represent paths like driving directions from a first-person point of view, e.g., turn right, turn left, go straight, etc.