search

r-three / common-pile

Repo to hold code and track issues for the collection of permissively licensed data
MIT License
22 stars 6 forks source link

Openreview #88

Open craffel opened 3 months ago

craffel commented 3 months ago

Papers on openreview can be marked as having some particular license, see e.g. https://openreview.net/forum?id=frA0NNBS1n We could crawl openreview for PDFs of appropriately licensed papers and extract text from them.

nkandpa2 commented 3 months ago

There's likely a large overlap with the arXiv papers and not all venues support the license field (e.g., ICLR doesn't appear to have license on their submission form). However, all reviews and comments on OpenReview are CC-BY (see Comment and Configuration Record License section on this page). This could make for a pretty interesting dataset where a record is a sequence of comments/reviews and optionally the associated paper (if the paper is permissively licensed). This would be useful for teaching models to critique, engage in dialog, summarize, etc.

From ICLR and TMLR alone there are roughly ~200K reviews for ~20K papers (~2K of which are permissively licensed). There are probably other venues that have public comments as well.