The experiments are presented kind of strangely in the paper, but basically we want to replicate Figure 5 and show that the modified model still performs well. The best way to do the second is to plot the loss curve of the modified and the baseline models on the same plot.
Modifications
Make it bigger! Let's start with a 1.3B model and go from there.
Background
Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hiding is the first serious attempt to combine data hiding with language models. Unfortunately, the original authors are rather compute limited, and it's unclear how scalable their method is. Replicate their work at the 1B parameter scale.
What to Replicate?
The experiments are presented kind of strangely in the paper, but basically we want to replicate Figure 5 and show that the modified model still performs well. The best way to do the second is to plot the loss curve of the modified and the baseline models on the same plot.
Modifications
Make it bigger! Let's start with a 1.3B model and go from there.
Related Papers/Frameworks