feat: add custom tokenizers for the representations

lamalab-org / xtal2txt

MIT License

6 stars 0 forks source link

Closed n0w0f closed 6 months ago

n0w0f commented 6 months ago

robocrys tokenizer is pending, train bpe later after dataprep

n0w0f commented 6 months ago

Thank you @kjappelbaum for the review

kjappelbaum commented 6 months ago

looks all fine to me, happy to see this making progress 🦾