Currently \scantokens relies on the \tl_set_rescan:Nnn function, but just like \use:x this command uses internal storage, so it is quite dangerous if the user's code itself contains the same function or others using this internal storage. In addition, \tl_set_rescan:Nnn tries to be clever with newlines, which makes it behave slightly differently to \scantokens itself.
The correct approach is to reimplement the little we need in unravel. While doing that we should be very careful about the trailing new line and about dealing with \everyeof.
Currently
\scantokensrelies on the\tl_set_rescan:Nnnfunction, but just like\use:xthis command uses internal storage, so it is quite dangerous if the user's code itself contains the same function or others using this internal storage. In addition,\tl_set_rescan:Nnntries to be clever with newlines, which makes it behave slightly differently to\scantokensitself.The correct approach is to reimplement the little we need in
unravel. While doing that we should be very careful about the trailing new line and about dealing with\everyeof.