Closed pmeier closed 4 years ago
Some addtional remarks:
v0.5.0
release. To make matters worse the manual upload is also not working for. I'll work this out soon. Edit: This was resolved in pmeier/pystiche#343As discussed in https://github.com/pyOpenSci/software-review/issues/21#issuecomment-626795429 (@lwasser, @NickleDave) I've found two reviewers for the technical aspects of pystiche
:
kornia
, a CV library for PyTorch.My apologies for the late submission. Due to CoViD I had to help out in other more pressing projects first. I hope your willingness to review still stands.
Hi @pmeier thank you for submitting and finding reviewers! And thank you for your patience -- I should have replied sooner.
@edgarriba and @soumith welcome! It seems like Philip has definitely brought people with the right expertise to review the package.
@lwasser here are my editor checks:
@lwasser I will follow up this weekend with editor comments
Reviewers: @edgarriba @soumith Due date: August 5th
@edgarriba @soumith typically we ask for 2-3 week turnaround for reviews. Although we should see if we can get editors to meet that standard first (I'm sorry @pmeier -- thank you again for your patience!). And I know everything is extra crazy for everyone right now.
Can you both let me know how doable three weeks sounds to you?
@NickleDave no prob from my side. Let's just put a deadline :)
Excellent, thank you @edgarriba. Makes sense Setting the due date for Aug 5 -- cc @soumith @pmeier @lwasser
Please let me know whatever I can do assist with the review process. Happy to provide additional info / guidance on top of what's here: https://github.com/pyOpenSci/software-review/issues/25#issuecomment-658512463
thank you so much @NickleDave et all!!
@NickleDave
Note that we probably need a third reviewer since, to my knowledge, both, @edgarriba and @soumith, have no domain knowledge about Neural Style Transfer (NST). Shall I assist the search?
Thank you for making that clear; I agree that it would be good to have someone with domain expertise.
It would be great if you can assist with the search. I can also search starting this weekend.
@edgarriba, @soumith
I just saw that AppVeyor Badge for v0.5.0
is not working properly:
This is due to the fact that I switched to GitHub Actions for the current master
, and AppVeyor kept failing since pystiche
was still activated. After I deactivated it, the badge went to Project unknown
, which was not better either. I tried to re-run for v0.5.0
, but AFAIK that is not possible. Re-running from master
always fails, since the needed scripts no longer exist. Thus, the badge is now stuck at cancelled
.
I made no changes to the functionality or the test-suite of pystiche
since v0.5.0
. You can find the test results for Windows (and the other OSs) here.
@pmeier @lwasser just want to follow up about reviewers and editor comments
For me, the main fit with pyopensci for this package is in terms of reproducibility + education. You aim to provide a package that encapsulates key concepts from this area of research, making them easier to learn about, and making it possible to reproduce key results without boilerplate.
To that end, the ideal reviewer would be able to speak to whether you are achieving those goals.
I also think we should find at least one of the reviewers (and when I say we, that means: me) to make sure the process is fair. I'm sure you agree. I put out an ask on Twitter (thank you Leah for the retweet) and I have at least one person in mind that I am contacting now. Will update as soon as I hear back
@NickleDave
For me, the main fit with pyopensci for this package is in terms of reproducibility + education. You aim to provide a package that encapsulates key concepts from this area of research, making them easier to learn about, and making it possible to reproduce key results without boilerplate.
pystiche
in upcoming publications. Reproducibility was just the most fitting out the available categories.pystiche
currently targets other researchers.I also think we should find at least one of the reviewers (and when I say we, that means: me) to make sure the process is fair. I'm sure you agree.
Agreed. I haven't heard back from anyone yet anyway (not that I'm hyper connected since this will be my first major contribution to the field). If someone does get back to me, I'll fill them in about this and refer them to you.
Thank you @pmeier I hear you, and I think we are on the same page. My comments are specifically about fit with PyOpenSci criteria but I don't mean to limit the scope of your package.
By education I do mean "easing the on-boarding in the field". To me this is a central goal for scientific packages: providing abstractions that make the research easier to understand. The package should enable "coding at the speed of thought". Which lines up with your point 1 above.
I am still reaching out to reviewers. @edgarriba @soumith as soon as a third reviewer joins us I will set the review due date as three weeks from that day
@edgarriba, @soumith, @NickleDave
I've just released pystiche==0.5.0.post0
since yesterdays release of torch==1.6.0
broke compatibility.
thank you @pmeier I am still reaching out to reviewers, will update as soon as I know more
@NickleDave @lwasser
Could we get an update on this? Its been almost a month. If I can help in any way, please let me know.
My grant is running until the end of the year and I need this published (in JOSS) before that. I'm aiming to have this published in late November or early December since I have some other duties at the end of the year. This leaves us with approximately 3 months from now including the JOSS process. Is this still reasonable together with pyOpenSci? If not, I need to take this to JOSS directly although I really want the pyOpenSci review.
Hi @pmeier I am sorry I have not gotten back to you, totally my fault.
Unfortunately I have not heard back from multiple invited reviewers.
I think part of it might also be that the package targets a relatively new and developing research area, and it's hard to find someone that has the right mix of domain expertise and OSS development experience.
I really appreciate that you want the scientific rigor added by our review. Let's go ahead with just @soumith and @edgarriba as reviewers. I'm sure their valuable input will help speed up review at JOSS as well.
@soumith and @edgarriba can you commit to finishing review two weeks from now, by Saturday September 12th, so we can help Philip get this to JOSS? I'm sure you are also both busy, and I don't mean to make it your problem that I have not been more on top of this review. But I think we all have some idea of what it's like to be a PhD student whose grant is running out.
It would be great if you can do a quick review that mainly focuses on the abstractions the library provides for this research. JOSS reviewers will be able to address questions of software engineering, so you do not need to spend as much time on those (even though I know you could). But you can give big picture feedback on the goals of the library and how it achieves those, given your knowledge of deep learning and computer vision.
Please @soumith @edgarriba let me know if that deadline of Saturday September 12th works for you, and if not, when you expect you will be able to complete review. That will help @pmeier decide how to move forward.
@NickleDave I appreciate the change of course. I don't want to interfere with the process, just wanted to say that we don't need to rush it that much. I think it should be enough if I get the reviews until the end of September.
ok, thank you for that additional information
if we start today (ignoring that I have made put this review very behind) then the standard three weeks deadline would put us at September 19th.
I am going to say September 19th and make sure I have that in my calendar so I stay on top of this, unless @edgarriba and @soumith absolutely cannot make that deadline
hey all!! just a note that once the review is complete here - the JOSS process is fast as they fully accept our review. So the key will be getting the reviewers on board with the Sept deadline and then addressing any feedback promptly. Thank you for your time. Just a note i've been struggling a bit with keeping up given COVID, beginning of the semester and other issues happening in the world right now. it would be great to find funding for this project so someone can work on it full time! i just haven't had time to devote to it.
i'll send in my review by that deadline. thanks for the heads-up!
🙏 🙏 🙏 thank you @soumith !!!
@edgarriba can you please confirm whether you can make that deadline of September 19th, when you get a chance?
Sure, will send too 💪
🙌🙌🙌 thank you both!!!
Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide
The package includes all the following forms of documentation:
setup.py
file or elsewhere.Readme requirements The package meets the readme requirements below:
The README should include, from top to bottom:
Reviewers are encouraged to submit suggestions (or pull requests) that will improve the usability of the package as a whole. Package structure should follow general community best-practices. In general please consider:
Note: Be sure to check this carefully, as JOSS's submission requirements and scope differ from pyOpenSci's in terms of what types of packages are accepted.
The package contains a paper.md
matching JOSS's requirements with:
Estimated hours spent reviewing: 3.5
The package looks good overall with a good organization and using updated tools to keep high quality standards in terms of software engineering. The repository seems to have a simple and proper structure for the specific goals of the project. The documentation seems quite complete including few but clear working examples in addition to a good and complete package reference.
The only weakness I see in the package is the development support. I list below few comments and suggestions on that:
Thank you @edgarriba ! I will let @pmeier reply but just want to say I appreciate your making time to review this
Hey @edgarriba, thanks a lot for your review. I'm only going to comment on things I think need discussion, i.e. I'll omit comments I agree with.
- [ ] Brief demonstration usage missing
You are of course correct here, but I need to challenge the intent. Thus, this comment is more directed at @NickleDave. pystiche
is only valuable because it provides all pieces of an NST that work together. The individual parts are probably not helpful to anyone. Thus, I only have usage examples that show the complete workflow. The minimal example I came up with is this. IMO this is far too long to be included in a README and thus I left it out. Is the "brief usage demonstration" in the README a hard requirement?
- [ ] Citation information missing
Correct again, but I'm not sure how to handle this properly. pystiche
will be published in JOSS and of course I will add the citation information after this is done. In the mean time the only thing I could include is a preliminary citation, which I would rather not do. If possible, I would have pystiche
only citable after it is peer-reviewed. @NickleDave what is the way to go here?
Missing some sort of script to create a virtual environment for development to avoid installing the dependencies in the main system. What if you don't even have pip, or tox ? :)
~I think I can require some basic prerequisite before anyone starts developing with python
and I think having pip
is one of them. This is not me judging someone just starting out, since everyone starts somewhere. I just don't think if you don't know even how to do this very basic steps, your first move should be trying to contribute to an (somewhat complex) project.~
~I'm torn on the virtual environment. On one hand I use one for the reason you suggested, but on the other hand its only very minimal requirements: pre-commit
and tox
. For everything else tox
handles the virtual environments.~
Edit: I've changed my mind about this. pmeier/pystiche#400 adds a virtual environment to the default workflow in the contribution guidelines.
Needed to install all the dependencies and tools by hand to get a success in tests.
Could you share your process with me? If you correctly use tox
(as explained in the contributing guidelines) you do not need to install anything by hand other than the minimal development requirements.
- [ ] Automated tests: Tests cover essential functions of the package and a reasonable range of inputs and conditions. All tests pass on the local machine. ERROR tests/integration/enc/models/test_vgg.py
Again, could you share your process and the error message with me? Everything is passing locally and in CI for me.
I see few usage for type or shape checking and raising informative errors in some of the functionalities.
Could you be more specific. Where do you see an opening for better error messages?
Docs are okay, just missing very few documentation (like in the image module). I would expect to see a small usage example in the most relevant classes.
I agree with the first part. Could you give me an example for the second part?
hey everyone, still working on the review, but I should have something by EOD on the 23rd.
Thank you @soumith ! And thank you @pmeier for your quick replies to Edgar's comments. I was surviving a work deadline but will get you feedback tomorrow AM
Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide
The package includes all the following forms of documentation:
setup.py
file or elsewhere.Readme requirements The package meets the readme requirements below:
The README should include, from top to bottom:
Reviewers are encouraged to submit suggestions (or pull requests) that will improve the usability of the package as a whole. Package structure should follow general community best-practices. In general please consider:
Note: Be sure to check this carefully, as JOSS's submission requirements and scope differ from pyOpenSci's in terms of what types of packages are accepted.
The package contains a paper.md
matching JOSS's requirements with:
Estimated hours spent reviewing: 1.5
The package makes the whole suite of neural style transfer work much easier to comprehend and reliably run. The documentation was nice, the examples were a breeze to read through. There's a lot of value in such a package and it's obvious that a lot of effort and tasteful design have been put into it's development.
Bigger points:
tox
environment isn't correctly configured. Additionally, some unit tests failed.To pip install the package, I tried what was listed in the README, but I got this error:
pip install https://github.com/pmeier/pystiche@v0.5.1
Collecting https://github.com/pmeier/pystiche@v0.5.1
ERROR: HTTP error 404 while getting https://github.com/pmeier/pystiche@v0.5.1
ERROR: Could not install requirement https://github.com/pmeier/pystiche@v0.5.1 because of HTTP error 404 Client Error: Not Found for url: https://github.com/pmeier/pystiche@v0.5.1 for URL https://github.com/pmeier/pystiche@v0.5.1
I instead used pip install git+https://github.com/pmeier/pystiche@v0.5.1
This is my pip
version to help debugging the issue:
pip --version
pip 20.2.2 from /home/bluebox/miniconda3/envs/pystiche_review/lib/python3.6/site-packages/pip (python 3.6)
pystiche.meta, pystiche.misc.build_complex_obj_repr, pystiche.ops.functional.value_range_loss, some functions in pystiche.image, most functions in pystiche.image.transforms didn't have any documentation beyond input/return type information
I had issue running the first example example_nst_with_pystiche.py
downloaded from https://pystiche.readthedocs.io/en/stable/_downloads/36c76ed9ccf74ecc12b41424d1ea227a/example_nst_with_pystiche.py which was linked from the end of this page: https://pystiche.readthedocs.io/en/stable/galleries/examples/beginner/example_nst_with_pystiche.html#sphx-glr-galleries-examples-beginner-example-nst-with-pystiche-py
The exact error was:
Traceback (most recent call last):
File "example_nst_with_pystiche.py", line 116, in <module>
images.download()
File "/home/bluebox/miniconda3/envs/pystiche_review/lib/python3.6/site-packages/pystiche/data/collections/download.py", line 200, in download
image.download(root=root, overwrite=overwrite)
File "/home/bluebox/miniconda3/envs/pystiche_review/lib/python3.6/site-packages/pystiche/data/collections/download.py", line 144, in download
raise FileExistsError(msg)
FileExistsError: bird1.jpg with a different MD5 hash already exists in /home/bluebox/.cache/pystiche. If you want to overwrite it, set overwrite=True.
I have never installed pystiche on this machine before. To double-confirm, I removed rm -rf /home/bluebox/.cache/pystiche
and reran the script again, but the error is the same.
To help debug:
python --version
Python 3.6.12 :: Anaconda, Inc.
pip list|grep torch
torch 1.6.0
torchvision 0.7.0
pip list|grep pystiche
pystiche 0.5.1
Running tests didn't work out of the box. I was following the instructions in CONTRIBUTING.rst: https://github.com/pmeier/pystiche/blob/v0.5.1/CONTRIBUTING.rst#tests
See the log for details on the environment, commands used, exact output, etc: https://gist.github.com/soumith/70f989dbd9d9d5702cbb24143e88755c
I decided to try get these to work manually by calling pytest .
and fixing issues.
I had to do the following:
pip install dill
pip install https://github.com/pmeier/pyimagetest/archive/v0.2.zip # i.e. the previous version of pyimagetest
pip install pillow-affine
pip install pytest-subtests
After this, pytest .
started working, but one test failed: https://gist.github.com/soumith/690f8fdc9a224f85a0484a04f65d40ec
pytest . --skip-slow
worked to completion
Hey @soumith, thanks a lot for your review. IMO all your points are valid. Although I was unable to reproduce the error you had with the example exactly I've stumbled over a similar one. It seems something about the downloading / storing of the demo images is buggy.
@pmeier still digesting reviews, but here's replies to two of your questions above:
Is the "brief usage demonstration" in the README a hard requirement?
I don't think we need to make anything a hard requirement. But it will help users understand what the library does if you can provide any kind of code snippet + images in addition to the verbal description you have now. For example, why not have as a snippet lines 176-204 from your example, with the content, style, and result images? You can say something like "snippet from the complete example that you can run here" to indicate to readers that they should not expect the snippet to run on its own.
You want it to be blindingly obvious from the README what the library does. Images of style transfer are the best way to do that. A code snippet that gives some flavor of how people create those images will help anyone understand how they will use your library to achieve NST, even if it's not a complete example. You don't want to lose someone because they had to dig through the docs.
If possible, I would have pystiche only citable after it is peer-reviewed. @NickleDave what is the way to go here?
You already have a DOI that people can cite in your repo, in the badge section. That's what this "citation" refers to. So you already met this criterion. As you probably know, people often have a separate section at the bottom of the README saying "if you use this software please cite the DOI". I would strongly encourage you to add it, just to make it as clear as you can that people should be citing (your) software. You might have noticed that it's harder to reproduce results when people don't cite the versions they use :)
re: JOSS, from their guidelines:
Upon successful completion of the review, authors will make a tagged release of the software, and deposit a copy of the repository with a data-archiving service such as Zenodo or figshare, get a DOI for the archive, and update the review issue thread with the version number and DOI.
So the DOI you have now will just change to the one for the new release associated with the JOSS review
@edgarriba do you feel @pmeier has addressed your comments?
If you would like to provide more specifics on some of the follow-up questions, e.g. about examples in docstrings, then that could be helpful for Philip.
But other than that I take it you feel the library can be approved as-is. Thank you again for making time to do the review.
@pmeier I see that @soumith made similar comments about a lack of inline examples, with suggestions for which functions would best benefit from them.
I saw you said that all the points in the review are valid, and you might be planning to get back with a detailed reply, but given that both reviewers made comments, please do clarify whether you plan to provide those.
@NickleDave @edgarriba @soumith
Here is my plan to address the review comments:
pystiche.ops
and pystiche.ops.functional
as they form the core parts of an NSTFurthermore, if not required to pass the review, I'll postpone adding more docstrings / examples to functions and classes that I suspect will not be used by the user. That includes the "undocumented" parts of pystiche.image
, pystiche.misc
and so on. I've planned some significant changes (see for example pmeier/pystiche#382) that will make parts of this obsolete.
Is there anything I've missed that needs to be addressed?
@edgarriba If possible, I like to know where you see an opening for better error messages. Did you encounter an error where you didn't know how to proceed?
@NickleDave, both @edgarriba and @soumith noticed that the link to the "pyOpenSi package guidelines" (https://www.pyopensci.org/dev_guide/authoring/overview.html) is broken. Could you provide a working one in order for this piece to also be reviewed?
Yes, I'm sorry about the broken link. I know it needs to be fixed but haven't had a chance to make the PR. This is the correct link: https://www.pyopensci.org/contributing-guide/authoring/overview.html
But I do not think it's necessary for @edgarriba or @soumith to check whether you have met the basic guidelines. Please don't worry about that holding up the review. The guidelines overlap with what is in the review checklist, and all of them are criteria that you had met at least since submitting if not much earlier. The link is basically there as a reminder to reviewers of what's expected from authors.
@edgarriba @soumith @NickleDave
I've addressed the reviews according to https://github.com/pyOpenSci/software-review/issues/25#issuecomment-698473702.
@edgarriba and @soumith can you just confirm with a comment whether you feel @pmeier has addressed concerns raised in your reviews (if you weren't going to already)?
I've been following @pmeier 's patches and commits.
I think he addressed all my concerns
same here. I believe latest commits cover the mentioned issues. cool vignette. Pretty clear image telling what's going on ;)
Awesome! Thank you so much @soumith and @edgarriba for finding time to review this! Since PyOpenSci is still getting off the ground it really means a lot to us that you could contribute. And I know it has helped @pmeier make real improvements to pystiche (even though it was already quite well developed upon submission)
🎉 pystiche
has been approved by pyOpenSci! Thank you again @pmeier for submitting pystiche
and many thanks to @soumith @edgarriba for reviewing this package! 😸
@lwasser tagging you in just to let you know I'm moving forward with this
@pmeier can you please do two more things to wrap up:
pystiche
. The badge should be [![pyOpenSci](https://tinyurl.com/y22nb8up)](https://github.com/pyOpenSci/software-review/issues/issue-number)
pystiche
to the pyOpenSci website. Please open a pr to update this file: to add your package and name to the list of contributors@pmeier @soumith @edgarriba please let you know if you would be okay with me adding you to our website as contributors.
I will submit a PR myself to add you to this file if you are okay with it. I don't want to ask you to submit any more PRs when you've already contributed!
For JOSS review: @pmeier I think you have already done the first two items below, right? I'm going to tag in @arfon just as a heads up but you should be ready to complete the last item.
All, please feel free to share any feedback about the review process here. The broken links in the documentation are duly noted! I will raise issues for them. Any other comments on documentation or on anything else is welcome and appreciated
@NickleDave no prob from my side. Nice xperience! open to more colab
@edgarriba @soumith
Thanks again also from my side! I really appreciate the time you voluntarily spent to help me out.
@NickleDave
Overall, I'm happy with the review process. Here are some things that IMO can be improved:
@NickleDave no prob from my side. Nice xperience! open to more colab
Excellent thank you @edgarriba
@NickleDave
* I'm fine with being added as contributor
Great
* Yes, the first two items on the JOSS to do list are already done * I've submitted to JOSS, but I think until the review starts there is no publicly accessible URL to link to
Got it
Overall, I'm happy with the review process. Here are some things that IMO can be improved:
* My initial comment starting the review process still contains a lot of "TBD"s. I'm not sure if that is my responsibility to fill or yours. If its mine, it should be stated somewhere * This issue still has its initial labels
Yes, my fault. Thank you for pointing it out -- will be aware of this in future reviews
* I've mentioned this before, but I would have appreciated a more transparent timeline early on
Understood. @lwasser @choldgraf @mbjoseph @jlpalomino @leouieda @lheagy this is probably a point we should discuss now that we've finished up this review and can reassess how to move PyOpenSci forward
🎉 🎉 🎉 It's official! ✨ ✨ ✨
@pmeier I merged in your PR above adding pystiche
to the website--I can see the link is live now.
I also merged in the PR that added you and @edgarriba as contributors (@soumith feel free to reply here if you would still like to be added, I can do that too).
Thank you again everyone for your contributions and for your patience with this process; this was my first time as editor and I'm still getting the hang of it. @pmeier really glad we could help you improve pystiche
and I actually learned a lot from seeing how you develop.
We will for sure want to share with the world that pystiche
has successfully gone through review at PyOpenSci, on Twitter and elsewhere, like our blog. Let's stay in touch about that as it finishes up review at JOSS.
Closing this issue for now
yahoooo!!!!!
@NickleDave @lwasser FYI: JOSS review was successful (openjournals/joss-reviews#2761) and the paper is already published.
wow now that's what I call a fast turnaround!
Submitting Author: Philip Meier (@pmeier) All current maintainers: Philip Meier (@pmeier) Package Name: pystiche One-Line Description of Package: Framework for Neural Style Transfer (NST) built upon PyTorch Repository Link: https://github.com/pystiche/pystiche Version submitted: 0.5.0post0 Editor: @NickleDave
Reviewer 1: @edgarriba
Reviewer 2: @soumith
Archive: JOSS DOI: Version accepted: v 0.6.0 Date accepted (month/day/year): 10/08/2020
Description
pystiche
is a framework for Neural Style Transfer (NST) algorithms based on PyTorch. NST is a neural-net-based technique to merge the content of one and the artistic style of another image. Similar to deep learning frameworkspystiche
eases up the workflow for researchers in this field. Rather than implementing everything yourself,pystiche
provides common building blocks of NST algorithms that can be conveniently combined. Thus, researchers can focus on implementing new ideas rather than implementing the periphery over and over again.Scope
Please indicate which category or categories this package falls under:
Explain how the and why the package falls under these categories (briefly, 1-2 sentences):
pystiche
can be used to reproduce NST papers while focusing on core aspects.Who is the target audience and what are scientific applications of this package?
The primary intended audience are researchers as described above. Apart from them
pystiche
could also be interesting for recreational use by non-scientists.Are there other Python packages that accomplish the same thing? If so, how does yours differ?
AFAIK there are no other packages provide a similar functionality. However, due to its popularity, there are many implementations, which are limited to a specific NST algorithm. An exception might be this which features the implementation of multiple papers.
If you made a pre-submission enquiry, please paste the link to the corresponding issue, forum post, or other discussion, or
@tag
the editor you contacted:21
Technical checks
For details about the pyOpenSci packaging requirements, see our packaging guide. Confirm each of the following by checking the box. This package:
Publication options
JOSS Checks
- [x] The package has an **obvious research application** according to JOSS's definition in their [submission requirements](https://joss.readthedocs.io/en/latest/submitting.html#submission-requirements). Be aware that completing the pyOpenSci review process **does not** guarantee acceptance to JOSS. Be sure to read their submission requirements (linked above) if you are interested in submitting to JOSS. - [x] The package is not a "minor utility" as defined by JOSS's [submission requirements](https://joss.readthedocs.io/en/latest/submitting.html#submission-requirements): "Minor ‘utility’ packages, including ‘thin’ API clients, are not acceptable." pyOpenSci welcomes these packages under "Data Retrieval", but JOSS has slightly different criteria. - [x] The package contains a `paper.md` matching [JOSS's requirements](https://joss.readthedocs.io/en/latest/submitting.html#what-should-my-paper-contain) with a high-level description in the package root or in `inst/`. - [x] The package is deposited in a long-term repository with the DOI: *Note: Do not submit your package separately to JOSS*Are you OK with Reviewers Submitting Issues and/or pull requests to your Repo Directly?
This option will allow reviewers to open smaller issues that can then be linked to PR's rather than submitting a more dense text based review. It will also allow you to demonstrate addressing the issue via PR links.
Code of conduct
P.S. Have feedback/comments about our review process? Leave a comment here
Editor and Review Templates
Editor and review templates can be found here