Bugs in tokenize()

[AntonBogun]

print(tokenize(";a;;", ";", "")) outputs ["","a"] instead of expected ["","a","",""]

[AntonBogun]

Furthermore, the whitespace argument does not work as intended:

let str="Hello,\r\nWorld" print(tokenize(str,"\n","")) print(tokenize(str,"\n","\r"))

prints ["Hello,\r", "World"] ["Hello,\r", "World"] while the second print is expected to be ["Hello,", "World"]

[aardappel]

That is intentional, it says by splitting into segments upon each dividing or terminating delimiter, i.e. the last 2 , are terminating.

The intention was to make the function useful for situations where the delimiter is dividing (such as typically intended with ,) or terminating (as typically intended with ; ).

I suppose I could have added a bool indicating whichever strategy, or two separate functions.

Agree the whitespace case is a bug.

[aardappel]

Ok, improved things in https://github.com/aardappel/lobster/commit/fd1f0c1b3617f5b2aba53ddb30b5e7cf70f525e8

• The whitespace bug is fixed.
• Two consecutive dividers will not be regarded as one anymore, result in empty string in between.
• Optional argument to get dividing rather than terminating behavior.