search

neulab / ExplainaBoard

Interpretable Evaluation for AI Systems
MIT License
362 stars 36 forks source link

[RFC] Contributor guide/contribution policy #376

Open neubig opened 2 years ago

neubig commented 2 years ago

Hi everyone, maybe especially @odashi and @pfliu-nlp: I'm thinking that we might want to make a more complete contribution guide with the dual goal of:

  1. Making it easier for people other than the ExplainaBoard core developers to make significant contributions
  2. At the same time making sure that contributions to ExplainaBoard are high quality and reliable

I was looking around at various contribution guides, and I found the TVM contribution guide quite nice, and we may be able to borrow some ideas from it:

Do you have any other good examples?

One related thing that I'm thinking is that since we're transitioning ExplainaBoard from research code to a robust software package, it might be nice to have one code reviewer who pays attention to that standpoint. This is something like Google's "readability" review, which I really benefitted from when I was an intern there: https://abseil.io/resources/swe-book/html/ch03.html#readability_standardized_mentorship_thr

Any other comments on this topic are welcome!

odashi commented 2 years ago

@neubig Readability review is good. Actually I was holding C++ and Python readability obtained during my past work and maybe I can provide similar responsibility for this repository.

Are you also intending to apply the same process for external contributors? The readability review is good for improving code quality, but at the same time it is somewhat tough to apply. I think it is still good to apply it for every pull requests though.

neubig commented 2 years ago

I think that initially it'd probably be good to treat non-core members and core members equally from the point of view of review policies. We could revisit that decision if it becomes too much of a burden though.

pfliu-nlp commented 2 years ago

The idea of the readability review looks good to me.

Regarding the contribution guide, sharing another good example:

odashi commented 2 years ago

@neubig How's this going on? Especially the review process could be changed immediately, while fixing the contribution policy may need more discussion.

neubig commented 2 years ago

I think we all agree on changing the review process to add code style review, so why don't we go ahead and do that. We should document it somewhere as well.

odashi commented 2 years ago

It also requires to add some mechanisms to block undesired pushes. Requiring CODEOWNERS review may be suitable I guess.

neubig commented 2 years ago

Sounds good. I don't immediately have an example of a CODEOWNERS file, so if you wouldn't mind sending a PR adding one appropriately that'd be appreciated. I could also figure it out when I have a moment though.

neubig commented 2 years ago

I created a draft of a minimalist design document template for ExplainaBoard: https://docs.google.com/document/d/1g-wEXaMxqe-1QFM-qrgJ41-MoCPuoT9fLutg5jk-NgE/edit#

Any comments are welcome!

neubig commented 2 years ago

I got references from my friend about supporting new developers in open source communities: