When coming across your work, I thought of including tests on texts generated through Locally Typical Sampling, a sampling method that selects tokens with conditional logits close to the information content of the estimated next-token distribution, in order to mimic information efficiency found in natural human dialogue and text. Locally Typical Sampling seems to be already implemented as an API option in the Huggingface Transformers library, and I have taken advantage of this implementation for my tests.
Initial tests seem to indicate DetectGPT performs as well at detecting texts generated by Locally Typical Sampling as with texts generated through methods that always select from a maximum-likelihood subset, possibly because high-entropy next-token distributions do not arise often enough to force the sampler away from selecting maximum-likelihood tokens.
When coming across your work, I thought of including tests on texts generated through Locally Typical Sampling, a sampling method that selects tokens with conditional logits close to the information content of the estimated next-token distribution, in order to mimic information efficiency found in natural human dialogue and text. Locally Typical Sampling seems to be already implemented as an API option in the Huggingface Transformers library, and I have taken advantage of this implementation for my tests.
Initial tests seem to indicate DetectGPT performs as well at detecting texts generated by Locally Typical Sampling as with texts generated through methods that always select from a maximum-likelihood subset, possibly because high-entropy next-token distributions do not arise often enough to force the sampler away from selecting maximum-likelihood tokens.
Arxiv reference: https://arxiv.org/pdf/2202.00666.pdf