Unlike words, emoji aren't simply the concatenation of their decoded tokens. The individual tokens that make up an emoji are decoded as �, so if we stream them users will see ���� instead of, say, 🔥.
This fixes that behavior for MLC and vLLM. Note that MLC was actually broken (returning �), vLLM was returning proper emoji but would first yield three empty strings.
Unlike words, emoji aren't simply the concatenation of their decoded tokens. The individual tokens that make up an emoji are decoded as �, so if we stream them users will see ���� instead of, say, 🔥.
This fixes that behavior for MLC and vLLM. Note that MLC was actually broken (returning �), vLLM was returning proper emoji but would first yield three empty strings.