Tracking instruction-tuned LLM openness. Paper: Liesenfeld, Andreas, Alianda Lopez, and Mark Dingemanse. 2023. “Opening up ChatGPT: Tracking Openness, Transparency, and Accountability in Instruction-Tuned Text Generators.” In Proceedings of the 5th International Conference on Conversational User Interfaces. doi:10.1145/3571884.3604316.
Based on Mistral 7B so every bit as closed as that except for Ultrafeedback DPO data (the "RL" component).
I hesitate to add this because it is so obviously designed as an attention grab: get a high position on a leaderboard, then release and advertise the leaderboard position as a reason for folks to get in business with you. It is not clear Snorkel is being offered as anything like a serious 'open' model for other than commercial downstream uses.
The customary 'release by blog post' strategy: https://snorkel.ai/new-benchmark-results-demonstrate-value-of-snorkel-ai-approach-to-llm-alignment/
HF page with model card used mainly for advertising: https://huggingface.co/snorkelai/Snorkel-Mistral-PairRM-DPO
Based on Mistral 7B so every bit as closed as that except for Ultrafeedback DPO data (the "RL" component).
I hesitate to add this because it is so obviously designed as an attention grab: get a high position on a leaderboard, then release and advertise the leaderboard position as a reason for folks to get in business with you. It is not clear Snorkel is being offered as anything like a serious 'open' model for other than commercial downstream uses.