Model releases often go accompanied with claims on evaluation performance, but those results are not always reproducible, or can be misleading. If code is not released, is not comprehensive, is difficult to run, or misses key details, this will cost the scientific community time and effort to replicate and verify the claims. Replication time will also slow progress, and discourage developers from adopting that resource over others.
With
Model releases accompanied with claims on performance that are not reproducible, code that is unavailable, incomplete, or difficult to run costs the scientific community time and effort. The following resources are valuable to help others replicate and verify the claims.
neural-loop
commented
1 month ago
Replace
Model releases often go accompanied with claims on evaluation performance, but those results are not always reproducible, or can be misleading. If code is not released, is not comprehensive, is difficult to run, or misses key details, this will cost the scientific community time and effort to replicate and verify the claims. Replication time will also slow progress, and discourage developers from adopting that resource over others.
With
Model releases accompanied with claims on performance that are not reproducible, code that is unavailable, incomplete, or difficult to run costs the scientific community time and effort. The following resources are valuable to help others replicate and verify the claims.