use Tensor version of rand in sample() to get rid of bare .alloc(). Probably irrelevant but better to use safe managed memory where it suits.
there are remaining .alloc() calls in str_to_ptr and str_concat but I think these are never put into a struct so they clean up correctly.
I don't notice any impact in the memory use because the Tokenizer is still only deleted just before the program completes. So this is mostly housekeeping in case Tokenizer use changes in the future.
Tensor
version ofrand
insample()
to get rid of bare.alloc()
. Probably irrelevant but better to use safe managed memory where it suits..alloc()
calls instr_to_ptr
andstr_concat
but I think these are never put into a struct so they clean up correctly.I don't notice any impact in the memory use because the
Tokenizer
is still only deleted just before the program completes. So this is mostly housekeeping in caseTokenizer
use changes in the future.