search

uhermjakob / utoken

universal tokenizer
BSD 3-Clause "New" or "Revised" License
14 stars 1 forks source link

Freezing on `@@` #4

Closed thammegowda closed 2 years ago

thammegowda commented 3 years ago

utoken freezes on @@ in input

echo "hello there @-@  ?" | utokenize
hello there @-@ ?

echo "hello there @@  ?" | utokenize

--[FROZEN]--

uhermjakob commented 3 years ago

Fixed. Thanks for reporting!!!