Manually review top 10 organizations returned by each method

[jmelot]

This may help us identify common spurious linkages. If you would like to work on this but need help getting started, please comment on this issue!

[hydejackson]

Hi! Could you explain what you mean by "manually review?" Are you looking for someone to look for relationships between the outputs?

[jmelot]

Hello! The goal of this issue is to help surface errors in our code that links software to the organizations of its contributors. What we're looking for here is for someone to:

• List the top 10 organizations currently returned by each of the methods described in our README. This would involve opening up software_to_ror.csv and, for each of the methods described in our README such as url_matches, ner_text_extraction, and so on, counting the number of times each ROR id was returned by that method. You can find the ROR ids in the ror_id column and the methods in the extraction_methods column
• For each of the top 10 organization ROR ids returned by a given method, manually check 20 randomly selected pieces of software linked to that organization and returned by that method to see if it looks like the organization's members really work on that software. For example, if you found that Google was the most frequently returned organization by the url_matches method, you would randomly select 20 rows from software_to_ror.csv where Google's ROR id was in the ror_id column and url_matches was in the extraction_methods column. You would then manually review the repo to see if you could find evidence that someone who worked at Google also worked on that piece of software. It would be a good idea to filter the rows you select so that only rows with the github_slug column are populated for easier review
• Open a PR with any supporting code needed to replicate the organization selection, plus a csv with five columns: ror_id, github_slug, extraction_methods, is_valid, and notes, with the first three columns taken from software_to_ror.csv, is_valid true if you were able to find evidence linking the organization to the software and false otherwise, and the notes column populated with notes on your findings. This csv could be placed in a new validation directory at the top level of this project.

Are you interested in working on this? If so let me know if you need any help getting set up with the first step, need more details on how to review software repos for evidence that someone at an organization works on that software, or have any further questions.