anupamme
commented
4 years ago If we take idea from beam search then we poison with k triggers at each round.
How do we pick k triggers in each round: we pick the triggers with maximum/min probability or we come up with some other idea.
Another question, How should the adversary k?
If you are an adversary choosing trigger can be tricky as it is not trivial to estimate the probability of each trigger and then pick one.
We show an algorithm an adversary can use to come up with an optimal trigger.
Let's take an example of language model. And suppose the adversary has a list of triggers (similar usage) e.g.
Lets meet at Starbucks Want to grab coffee Lets catch up on coffee
How should an adversary pick a trigger?
Each word, the phrase has a probability of occurring based on what the user has typed so far. Can I query a language model many times to estimate the probability?
If yes, then I can have three probabilities for three phrases: p1, p2 and p3.
More work is needed on this.