search

google-research / dex-lang

Research language for array processing in the Haskell/ML family
BSD 3-Clause "New" or "Revised" License
1.56k stars 106 forks source link

Add multisets #1335

Open duvenaud opened 10 months ago

duvenaud commented 10 months ago

This is another step towards writing a tokenizer demo in Dex. Multisets let us find all the unique words in a corpus along with their counts in parallel.

duvenaud-anthro commented 10 months ago

Just noting that the Dex tests all pass now. Something is broken with the jax python tests independent of this PR.