search

openreview / openreview

The most recent documentation of OpenReview
21 stars 4 forks source link

OpenReview's APIs Errors #294

Open Fatemeh-Rahbari opened 2 months ago

Fatemeh-Rahbari commented 2 months ago

Describe the Issue Getting camera-ready reversions, attachments and reviews for venues in API1 and API2 are missing/wrong/outdated in the documentation.

Link

  1. https://docs.openreview.net/how-to-guides/data-retrieval-and-modification/how-to-view-camera-ready-revisions
  2. https://docs.openreview.net/how-to-guides/data-retrieval-and-modification/how-to-export-all-submission-attachments
  3. https://docs.openreview.net/how-to-guides/data-retrieval-and-modification/how-to-get-all-reviews
  4. https://docs.openreview.net/how-to-guides/data-retrieval-and-modification/how-to-get-all-metareviews

Additional context Add any other context about the problem here.

carlosmondra commented 2 months ago

@Fatemeh-Rahbari thanks for reporting this. Can you elaborate? I'm not sure I can spot the part of the documentation that is outdated.

Fatemeh-Rahbari commented 2 months ago

Subject: Assistance Needed with Thesis Project

Hi Carlos,

Thank you for responding so quickly!

I wanted to let you know that I have recently started working on my thesis project, which involves analyzing papers and their related reviews on the OpenReview platform. Specifically, I need to retrieve the accepted submissions from Computer Science and Engineering (CSE) venues, along with their camera-ready versions and the reviews they received. Unfortunately, the provided Python code I have encountered does not function as expected, and I am struggling with outdated and corrupted scripts. As a result, I’ve had to manually download small portions of the data just to showcase my work.

To summarize my needs:

  1. I need to retrieve and download accepted submissions from both API1 and API2, but I am unable to do so.
  2. I need to obtain the camera-ready versions of the accepted submissions from both API1 and API2, but this is not working either.
  3. I need to retrieve the official reviews related to those submissions from API1 and API2, but I am facing similar challenges.

I would greatly appreciate your assistance with these issues, as resolving them would significantly accelerate my project timeline.

Thank you again for your help!

Sincerely, Fatemeh

On Thu, Apr 10, 2025 at 9:29 AM Carlos Daniel Mondragón Chapa < @.***> wrote:

@Fatemeh-Rahbari https://github.com/Fatemeh-Rahbari thanks for reporting this. Can you elaborate? I'm not sure I can spot the part of the documentation that is outdated.

— Reply to this email directly, view it on GitHub https://github.com/openreview/openreview/issues/294#issuecomment-2793012058, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVLY5SCVOP2WD4BAXX3CJY32YZW2RAVCNFSM6AAAAAB2ZOQNHGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJTGAYTEMBVHA . You are receiving this because you were mentioned.Message ID: @.**> carlosmondra* left a comment (openreview/openreview#294) https://github.com/openreview/openreview/issues/294#issuecomment-2793012058

@Fatemeh-Rahbari https://github.com/Fatemeh-Rahbari thanks for reporting this. Can you elaborate? I'm not sure I can spot the part of the documentation that is outdated.

— Reply to this email directly, view it on GitHub https://github.com/openreview/openreview/issues/294#issuecomment-2793012058, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVLY5SCVOP2WD4BAXX3CJY32YZW2RAVCNFSM6AAAAAB2ZOQNHGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJTGAYTEMBVHA . You are receiving this because you were mentioned.Message ID: @.***>

carlosmondra commented 2 months ago

Hi @Fatemeh-Rahbari, can you please share the code that is not working for you?

Fatemeh-Rahbari commented 1 month ago

Hi Carlos,

Yes, sure. Here they are:

  1. How to view Camera-Ready Revisions https://docs.openreview.net/how-to-guides/data-retrieval-and-modification/how-to-view-camera-ready-revisions : camera_ready_notes = [] for note in accepted_notes: camera_ready_invitation = f'venue_id/Submission{note.number}/-/Camera_Ready_Revision' if camera_ready_invitation in note.invitations: camera_ready_notes.append(note)

  2. How to Export all Submission Attachments: https://docs.openreview.net/how-to-guides/data-retrieval-and-modification/how-to-export-all-submission-attachments for note in notes: if(note.content.get("pdf",{}).get('value')): f = client.get_attachment(note.id,'pdf') with open(f'submission{note.number}.pdf','wb') as op: op.write(f) for note in notes: if(note.content.get("supplementary_material",{}).get('value')): f = client.get_attachment(note.id,'supplementary_material') with open(f'submission{note.number}_supplementary_material.zip','wb') as op: op.write(f)

  3. How to Get All Reviews https://docs.openreview.net/how-to-guides/data-retrieval-and-modification/how-to-get-all-reviews : 1) Venues using API V2 venue_id = 'Your/Venue/ID' venue_group = client.get_group(venue_id) submission_name = venue_group.content['submission_name']['value'] submissions = client.get_all_notes(invitation=f'{venue_id}/-/{submission_name}', details='replies')

review_name = venue_group.content['review_name']['value'] reviews=[openreview.api.Note.from_json(reply) for s in submissions for reply in s.details['replies'] if f'{venue_id}/{submission_name}{s.number}/-/{review_name}' in reply['invitations']]

2) Venues using API V1

  1. SIngle-Blind: submissions = client.get_all_notes( invitation="Your/Venue/ID/-/Blind_Submission", details='directReplies' reviews = [] for submission in submissions: reviews = reviews + [openreview.Note.from_json(reply) for reply in submission.details["directReplies"] if reply["invitation"].endswith("Official_Review")] )

  2. Double-blind: submissions = client.get_all_notes( invitation="Your/Venue/ID/-/Submission", details='directReplies' ) reviews = [] for submission in submissions: reviews = reviews + [openreview.Note.from_json(reply) for reply in submission.details["directReplies"] if reply["invitation"].endswith("Official_Review")]

  3. How to get all Rebuttals: https://docs.openreview.net/how-to-guides/data-retrieval-and-modification/how-to-get-all-rebuttals venue_id = VENUE_ID venue_group_settings = client.get_group(venue_id).content submission_invitation = venue_group_settings['submission_id']['value'] submissions = client.get_all_notes( invitation=submission_invitation, details='directReplies' ) rebuttals = [] for s in submissions: for r in s.details['directReplies']: if any(invitation.endswith('Rebuttal') for invitation in r['invitations']): rebuttals.append(r)

  4. How to Get All Submissions: https://docs.openreview.net/how-to-guides/data-retrieval-and-modification/how-to-get-all-submissions 1) API V2 venue_group = client.get_group('Your/Venue/ID') submission_name = venue_group.content['submission_name']['value'] submissions = client.get_all_notes(invitation=f'Your/Venue/ID/-/{submission_name}')

2) Get only the "accepted" submissions for double-blind venues # Double-blind venues submissions = client.get_all_notes(invitation = 'Your/Venue/ID/-/Blind_Submission', details='directReplies,original') blind_notes = {note.id: note for note in submissions} all_decision_notes = [] for submission_id, submission in blind_notes.items(): all_decision_notes = all_decision_notes + [reply for reply in submission.details["directReplies"] if reply["invitation"].endswith("Decision")] accepted_submissions = [] for decision_note in all_decision_notes: if 'Accept' in decision_note["content"]['decision']:

accepted_submissions.append(blind_notes[decision_note['forum']].details['original'])

3) Get only the "accepted" submissions for single-blind venues

Single-blind venues

submissions = client.get_all_notes(invitation = 'Your/Venue/ID/-/Submission', details='directReplies') notes = {note.id: note for note in submissions} all_decision_notes = [] for submission_id, submission in notes.items(): all_decision_notes = all_decision_notes + [reply for reply in submission.details["directReplies"] if reply["invitation"].endswith("Decision")] accepted_submissions = [] for decision_note in all_decision_notes: if 'Accept' in decision_note["content"]['decision']: accepted_submissions.append(notes[decision_note['forum']])

I have been facing these faulty codes so far, and I am unsure about the other parts. If we can further develop the current project phase and move on to the next stage, I will use other codes. If they turn out to be faulty, I’ll be sure to let you know.

I appreciate your help in advance with resolving these issues!

Best, Fatemeh

On Fri, Apr 11, 2025 at 9:56 AM Carlos Daniel Mondragón Chapa < @.***> wrote:

Hi @Fatemeh-Rahbari https://github.com/Fatemeh-Rahbari, can you please share the code that is not working for you?

— Reply to this email directly, view it on GitHub https://github.com/openreview/openreview/issues/294#issuecomment-2796991229, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVLY5SHLTIOC3KPJS7EM24D2Y7CY5AVCNFSM6AAAAAB2ZOQNHGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJWHE4TCMRSHE . You are receiving this because you were mentioned.Message ID: @.**> carlosmondra* left a comment (openreview/openreview#294) https://github.com/openreview/openreview/issues/294#issuecomment-2796991229

Hi @Fatemeh-Rahbari https://github.com/Fatemeh-Rahbari, can you please share the code that is not working for you?

— Reply to this email directly, view it on GitHub https://github.com/openreview/openreview/issues/294#issuecomment-2796991229, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVLY5SHLTIOC3KPJS7EM24D2Y7CY5AVCNFSM6AAAAAB2ZOQNHGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJWHE4TCMRSHE . You are receiving this because you were mentioned.Message ID: @.***>

carlosmondra commented 1 month ago

Let's go one by one:

  1. How to view Camera-Ready Revisions https://docs.openreview.net/how-to-guides/data-retrieval-and-modification/how-to-view-camera-ready-revisions : camera_ready_notes = [] for note in accepted_notes: camera_ready_invitation = f'venue_id/Submission{note.number}/-/Camera_Ready_Revision' if camera_ready_invitation in note.invitations: camera_ready_notes.append(note)

What venue_id are you passing here?

Fatemeh-Rahbari commented 1 month ago

Sure. First, I take "venues" from this code:

Get a list of all of the venues in OpenReview

venues = client.get_group(id='venues').memberslet's Second, I need those events that are related to the CS major, and I filter them from the venues. Third, I divide the CS venues into API1 and API2 versions. Then, in a for-loop, I get each venue_id and give it to the camera-ready code. For example, let's consider venue_id = 'ICLR.cc/2013/conference' from API v1. It returns empty when we use it in: accept_sub = client.get_all_notes(content = {'venueid':venue_id}) As a result, the camera-ready is empty!

carlosmondra commented 1 month ago

API V1 venues are tricky because the data has changed over the years and we haven't migrated it. I would start with API 2 venues only. The data in these venues is consistent. Make sure that you are using different clients for each of the API versions. Let me know if you can get the data of API V2 venues.

Fatemeh-Rahbari commented 1 month ago

It doesn't work for API v2 either! For example, try venue_id = 'NeurIPS.cc/2023/Workshop/TGL'

Could you please provide what the different clients are for each of the API versions?

carlosmondra commented 1 month ago

It doesn't work for API v2 either! For example, try venue_id = 'NeurIPS.cc/2023/Workshop/TGL'

That venue doesn't have Camera Ready Revision. Every venue has its own workflow, so you need to make sure that the data you are querying exists.

Try with NeurIPS.cc/2023/Conference:

venue_id = 'NeurIPS.cc/2023/Conference'

accepted_notes = client_v2.get_all_notes(content={'venueid': venue_id})

camera_ready_notes = []
for note in accepted_notes:
    camera_ready_invitation = f'{venue_id}/Submission{note.number}/-/Camera_Ready_Revision'
    if camera_ready_invitation in note.invitations:
        camera_ready_notes.append(note)

I just tested it and it does work. Hopefully this helps.

Could you please provide what the different clients are for each of the API versions?

Sure, it's explained here: https://docs.openreview.net/getting-started/using-the-api/installing-and-instantiating-the-python-client