Open john-hewitt opened 1 month ago
Hi John!
Olmo V1 was trained on not explicitly trained on any instruction data. If any leak in occurs, I suspect is through the code subset we trained OLMo on.
I am would surprised WIMDB is stripping that formatting, but I can check. In any case, the vocab for the base OLMo model does not contain an assistant tokens, so <|assistant|>
would get split.
Hi Luca!!
Thanks so much for the response. That certainly helps clarify things to me. I'd also be curious about the WIMDB details, but no pressure there.
Hey @john-hewitt,
With WIMBD, we have the online search index, which relies on elasticsearch. So it uses some stop words to index the data and filter the queries, which is why you got the same numbers for your query.
We also have a search tool written in rust (so it runs pretty fast) that doesn't have these kinds of filters. I just ran this query on dolma1.5, and found no instances of the <|assistant|> ngram.
If you're interested in running it yourself (it doesn't require much memory, just storing the data locally), you can check it out here. But if you're unable to store it, I could run a list of queries for you if you'd like.
Hi everyone,
I'm working on a research project relating to instruction following, and it would be amazing to have a language model with a guarantee that no explicitly instruction-following data (e.g., from LIMA, or Alpaca, etc. etc.,) was used during pretraining.
Some thoughts:
<|assistant|>
n-gram shows up around 20M times, but... it returns the identical answer forassistant
without the formatting, so I imagine it's stripping the formatting, so this isn't useful.I realize data can leak in, so the answer is probably not "definitely not" but does anyone know if the answer is at least "not intentionally"?
See corresponding Olmo request; wasn't sure how much information sharing there would be between the two: https://github.com/allenai/OLMo/issues/658
Thanks!