free all allocated pointers

[mikowals]

• free the pointers allocated during Tokenizer.init
• use Tensor version of rand in sample() to get rid of bare .alloc(). Probably irrelevant but better to use safe managed memory where it suits.
• there are remaining .alloc() calls in str_to_ptr and str_concat but I think these are never put into a struct so they clean up correctly.

I don't notice any impact in the memory use because the Tokenizer is still only deleted just before the program completes. So this is mostly housekeeping in case Tokenizer use changes in the future.

[mikowals]

marked as draft because it looks like running from this branch is producing nonsense output.

[mikowals]

There is no problem with this branch. I had accidentally pulled some broken stuff into my local copy.

[tairov]

Thank you!