openreview / openreview-py

Official Python client library for the OpenReview API
https://openreview-py.readthedocs.io/en/latest/
MIT License
156 stars 23 forks source link

How to link reviewers and their reviews as a PC on conference #2307

Open adinriv opened 3 months ago

adinriv commented 3 months ago

I'm trying to analyze the reviews on a venue I was a PC using APIv2.

I was able to retrieve the reviewers and reviews per submission following the documentation. However, when a submission has reviewers that didn't submitted their reviews the assignment is no one to one. I was looking for a way to match the assigned reviewers and their reviews.

I was thinking about:

Is there a better way of doing it with the API? What is the supposed way to retrieve this information?

import openreview

client = openreview.api.OpenReviewClient(baseurl='https://api2.openreview.net', username='**', password='**') 
venue_id="**"
venue_group = client.get_group(venue_id)

# strings
submission_name = venue_group.content['submission_name']['value']
review_name = venue_group.content['review_name']['value']

submissions = client.get_all_notes(invitation=f'{venue_id}/-/{submission_name}', details='replies')
for submission in submissions:
    # Assigned reviewers per submission
    submission_reviewers = client.get_group(f'{venue_id}/{submission_name}{submission.number}/Reviewers')

    # Replies
    # sift through the replies for the official reviews (maybe useful if I want to compute interactions as well)
    # reviews = [r for r in submission.details['directReplies'] if f'{venue_id}/{submission_name}{submission.number}/-/{review_name}' in r['invitations']]
    # direct reviews from notes
    reviews = client.get_all_notes(invitation=f'{venue_id}/{submission_name}{submission.number}/-/{review_name}')

    # how to link the submission_reviewers and the reviews?
melisabok commented 3 months ago

the review note has a field called "signatures" that contains a group id, if you get that group then you can check its member that must contain the profile id of the reviewer that posted that review.

adinriv commented 3 months ago

That was the missing link. Thanks.