search

nert-nlp / legal-cgel

Treebank of legal English in the CGEL framework
2 stars 1 forks source link

tracking annotation/reviewing progress #36

Open bwaldon opened 4 weeks ago

bwaldon commented 4 weeks ago

I'm currently using a Google Sheet to track annotations (e.g., has a stub been created, has an annotator been assigned, has a PR been opened, has the tree been added to the main branch, etc.). It won't be super easy to keep that sheet in sync with the repo: one has to manually compare it against the list of open/closed PRs and the list of dataset files.

Idea 1: restructure the repo: one subdir. for in-progress trees, one subdir. for completed trees. Would make it easier to keep track of things, but then files will have to be moved between subdirectories when they've been approved. (Can you even do this on GitHub? There's git mv in the command line, but one of the virtues of the current workflow is that everything post-annotation is happening in the GitHub web interface).

Idea 2: github action w/ Google Sheets API call. fancy, and less manual work for project admins, but could be more trouble than it's worth.

nschneid commented 4 weeks ago

I think a spreadsheet is going to be helpful regardless of how the repo is structured.

Can stub creation and annotator assignment happen at the same time? You could update the spreadsheet twice for each sentence: once when created/assigned and once after merging the PR. And then just monitor PRs to see which ones are ready for review.