Submission group 22: StrapvizPy

Submitting Author: Julien Gordon (@BooleanJulien), Gautham Pughazhendhi (@gauthampughazhendhi), Zack Tang( @zackt113), and Margot Vore (@voremargot) Package Name: StrapvizPy One-Line Description of Package: Used for bootstrapping visualizations Repository Link: https://github.com/UBC-MDS/strapvizpy Version submitted: v0.2.4 Reviewer 1: Rada Rudyak (@Radascript) Reviewer 2: Arushi Ahuja (@Arushi282) Reviewer 3: Zheren Xu (@ZherenXu) Reviewer 4: Anupriya Srivastava (@Anupriya-Sri)

Description

The purpose of this package is to simplify and automate the process of creating simple bootstrap distributions of numerical samples. The package has a module which intakes a sample and relevant parameters such as the desired confidence bounds and number of simulations. The module will perform the simulation statistics to generate the bootstrap distribution and relevant statistics such as the sample mean and bootstrapped confidence interval. The package also has a module for visualization of the bootstrapped confidence interval, and for creating a professional publication-ready table of the relevant statistics.

Scope

Please indicate which category or categories this package falls under:
- [ ] Data retrieval
- [ ] Data extraction
- [ ] Data munging
- [ ] Data deposition
- [ ] Reproducibility
- [ ] Geospatial
- [x] Education
- [x] Data visualization*
Explain how the and why the package falls under these categories (briefly, 1-2 sentences):

While the package has functions that perform bootstrapping, the package's main functionality is to create publication-ready visualizations thus we would categorize it as a data visualization package. It was also developed specifically for DSCI 552 in the MDS Vancouver program making it educational.
Who is the target audience and what are scientific applications of this package?

This package was developed to assist anyone who wants to bootstrap and visualize sampling distributions with different estimators during statistical analysis (especially students in 552 course). The package takes care of the tedious tasks of bootstrapping and visualization, and helps the user focus on the actual analysis.
Are there other Python packages that accomplish the same thing? If so, how does yours differ?

There are several packages that create bootstrap distributions but as far as we are aware, there are no packages that focus on creating sampling distribution visualizations.

Technical checks

For details about the pyOpenSci packaging requirements, see our packaging guide. Confirm each of the following by checking the box. This package:

[x] does not violate the Terms of Service of any service it interacts with.
[x] has an OSI approved license.
[x] contains a README with instructions for installing the development version.
[x] includes documentation with examples for all functions.
[x] contains a vignette with examples of its essential functions and uses.
[x] has a test suite.
[x] has continuous integration, such as Travis CI, AppVeyor, CircleCI, and/or others.

Publication options

[ ] Do you wish to automatically submit to the Journal of Open Source Software? If so:

JOSS Checks

- [ ] The package has an **obvious research application** according to JOSS's definition in their [submission requirements][JossSubmissionRequirements]. Be aware that completing the pyOpenSci review process **does not** guarantee acceptance to JOSS. Be sure to read their submission requirements (linked above) if you are interested in submitting to JOSS. - [ ] The package is not a "minor utility" as defined by JOSS's [submission requirements][JossSubmissionRequirements]: "Minor ‘utility’ packages, including ‘thin’ API clients, are not acceptable." pyOpenSci welcomes these packages under "Data Retrieval", but JOSS has slightly different criteria. - [ ] The package contains a `paper.md` matching [JOSS's requirements][JossPaperRequirements] with a high-level description in the package root or in `inst/`. - [ ] The package is deposited in a long-term repository with the DOI: *Note: Do not submit your package separately to JOSS*

Are you OK with Reviewers Submitting Issues and/or pull requests to your Repo Directly?

This option will allow reviewers to open smaller issues that can then be linked to PR's rather than submitting a more dense text based review. It will also allow you to demonstrate addressing the issue via PR links.

[x] Yes I am OK with reviewers submitting requested changes as issues to my repo. Reviewers will then link to the issues in their submitted review.

Code of conduct

[x] I agree to abide by pyOpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.

P.S. *Have feedback/comments about our review process? Leave a comment here

Editor and Review Templates

Editor and review templates can be found here

Package Review

Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide

[x] As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

Documentation

The package includes all the following forms of documentation:

[x] A statement of need clearly stating problems the software is designed to solve and its target audience in README
[x] Installation instructions: for the development version of package and any non-standard dependencies in README
[x] Vignette(s) demonstrating major functionality that runs successfully locally
[x] Function Documentation: for all user-facing functions
[x] Examples for all user-facing functions
[x] Community guidelines including contribution guidelines in the README or CONTRIBUTING.
[x] Metadata including author(s), author e-mail(s), a url, and any other relevant metadata e.g., in a setup.py file or elsewhere.

Readme requirements The package meets the readme requirements below:

[x] Package has a README.md file in the root directory.

The README should include, from top to bottom:

[x] The package name
[x] Badges for continuous integration and test coverage, a repostatus.org badge, and any other badges. If the README has many more badges, you might want to consider using a table for badges: see this example. Such a table should be more wide than high. (Note that the badge for pyOpenSci peer-review will be provided upon acceptance.)
[x] Short description of goals of package, with descriptive links to all vignettes (rendered, i.e. readable, cf the documentation website section) unless the package is small and there’s only one vignette repeating the README.
[x] Installation instructions
[ ] Any additional setup required (authentication tokens, etc)
[x] Brief demonstration usage
[x] Direction to more detailed documentation (e.g. your documentation files or website).
[x] If applicable, how the package compares to other similar packages and/or how it relates to other packages
[ ] Citation information

Usability

Reviewers are encouraged to submit suggestions (or pull requests) that will improve the usability of the package as a whole. Package structure should follow general community best-practices. In general please consider:

[x] The documentation is easy to find and understand
[x] The need for the package is clear
[x] All functions have documentation and associated examples for use

Functionality

[x] Installation: Installation succeeds as documented.
[x] Functionality: Any functional claims of the software been confirmed.
[x] Performance: Any performance claims of the software been confirmed.
[x] Automated tests: Tests cover essential functions of the package and a reasonable range of inputs and conditions. All tests pass on the local machine.
[x] Continuous Integration: Has continuous integration, such as Travis CI, AppVeyor, CircleCI, and/or others.
[x] Packaging guidelines: The package conforms to the pyOpenSci packaging guidelines.

For packages co-submitting to JOSS

[ ] The package has an obvious research application according to JOSS's definition in their submission requirements.

Note: Be sure to check this carefully, as JOSS's submission requirements and scope differ from pyOpenSci's in terms of what types of packages are accepted.

The package contains a paper.md matching JOSS's requirements with:

[ ] A short summary describing the high-level functionality of the software
[ ] Authors: A list of authors with their affiliations
[ ] A statement of need clearly stating problems the software is designed to solve and its target audience.
[ ] References: with DOIs for all those that have one (e.g. papers, datasets, software).

Final approval (post-review)

[ ] The author has responded to my review and made changes to my satisfaction. I recommend approving this package.

Estimated hours spent reviewing: 45 minutes

Review Comments

Nice work! The project is well organized and the documentation is helpful and easy to understand. Here are my suggestions:

Since the package was built on NumPy and Matplotlib, they need to be referred to in the citation section in README.
Although the coverage of test cases has reached 100%, there are few edge cases.
Some simple examples can be added to the Usage part so that users don't need to read the entire doc to understand your package.
Re-run example.ipynb, so the cell index starts from 1.
In the ci-cd.yml, you only test ubuntu. It's necessary to test Window and macOS to ensure your package is compatible with other systems.

Package Review

Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide

[X] As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).
Documentation

The package includes all the following forms of documentation:
[X] A statement of need clearly stating problems the software is designed to solve and its target audience in README
[X] Installation instructions: for the development version of package and any non-standard dependencies in README
[X] Vignette(s) demonstrating major functionality that runs successfully locally
[X] Function Documentation: for all user-facing functions
[ ] Examples for all user-facing functions
[X] Community guidelines including contribution guidelines in the README or CONTRIBUTING.
[X] Metadata including author(s), author e-mail(s), a url, and any other relevant metadata e.g., in a setup.py file or elsewhere. Readme requirements The package meets the readme requirements below:
[X] Package has a README.md file in the root directory. The README should include, from top to bottom:
[X] The package name
[X] Badges for continuous integration and test coverage, a repostatus.org badge, and any other badges. If the README has many more badges, you might want to consider using a table for badges: see this example. Such a table should be more wide than high. (Note that the badge for pyOpenSci peer-review will be provided upon acceptance.)
[X] Short description of goals of package, with descriptive links to all vignettes (rendered, i.e. readable, cf the documentation website section) unless the package is small and there’s only one vignette repeating the README.
[X] Installation instructions
[X] Any additional setup required (authentication tokens, etc)
[ ] Brief demonstration usage
[X] Direction to more detailed documentation (e.g. your documentation files or website).
[ ] If applicable, how the package compares to other similar packages and/or how it relates to other packages
[ ] Citation information
Usability

Reviewers are encouraged to submit suggestions (or pull requests) that will improve the usability of the package as a whole. Package structure should follow general community best-practices. In general please consider:
[X] The documentation is easy to find and understand
[X] The need for the package is clear
[X] All functions have documentation and associated examples for use
Functionality
[X] Installation: Installation succeeds as documented.
[X] Functionality: Any functional claims of the software been confirmed.
[X] Performance: Any performance claims of the software been confirmed.
[X] Automated tests: Tests cover essential functions of the package and a reasonable range of inputs and conditions. All tests pass on the local machine.
[X] Continuous Integration: Has continuous integration, such as Travis CI, AppVeyor, CircleCI, and/or others.
[X] Packaging guidelines: The package conforms to the pyOpenSci packaging guidelines.
For packages co-submitting to JOSS (Not Applicable)
[ ] The package has an obvious research application according to JOSS's definition in their submission requirements. Note: Be sure to check this carefully, as JOSS's submission requirements and scope differ from pyOpenSci's in terms of what types of packages are accepted. The package contains a paper.md matching JOSS's requirements with:
[ ] A short summary describing the high-level functionality of the software
[ ] Authors: A list of authors with their affiliations
[ ] A statement of need clearly stating problems the software is designed to solve and its target audience.
[ ] References: with DOIs for all those that have one (e.g. papers, datasets, software).
Final approval (post-review)
[X] The author has responded to my review and made changes to my satisfaction. I recommend approving this package. Estimated hours spent reviewing: 1.5 hours

Review Comments

This project is very relevant to our worn Statistics, and I thoroughly enjoyed reviewing it. The repository is well structured and had clear instructions on how to install and use the functionality. All the functions are working as per the documentation and test cases cover a broad range of scenarios.

I do have a few comments for your consideration:

Organization (functions): It would be a good idea to modularize the source code and package more i.e. have a single file for each function. This will be helpful in case additional functionality is added to the package in future.
Organization (tests): It would also be good to separate tests associated with different functions into different files. It took me some time to realise that there were four test functions in each file, associated with two package functions.
Documentation: While the functions individually give examples, it would be good to have a demo on the landing page itself. This can include toy data along with use case example for all functions.
Functionality: During bootstrapping, there is sometimes a need to get the samples without replacement. So, the function bootstrap_distribution() can have replacement as an optional argument. This will allow the user to specify if bootstrapping is to be done with or without replacement.
Unit tests: One test can be included for each function to validate that the type of output is as per the function documentation. This will ensure that the users do not get unexpected data types while using the package.
References: It would be good to include attribution/references to the packages, research papers or MDS material used for implementing this project.

Overall, I think that the work done is impressive in terms of functionality as well as documentation. I wish you the best as you develop it further.

Package Review

Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide

[x] As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

Documentation

The package includes all the following forms of documentation:

[x] A statement of need clearly stating problems the software is designed to solve and its target audience in README
[x] Installation instructions: for the development version of package and any non-standard dependencies in README
[x] Vignette(s) demonstrating major functionality that runs successfully locally
[x] Function Documentation: for all user-facing functions
[x] Examples for all user-facing functions
[x] Community guidelines including contribution guidelines in the README or CONTRIBUTING.
[x] Metadata including author(s), author e-mail(s), a url, and any other relevant metadata e.g., in a setup.py file or elsewhere.

Readme requirements The package meets the readme requirements below:

[x] Package has a README.md file in the root directory.

The README should include, from top to bottom:

[x] The package name
[x] Badges for continuous integration and test coverage, a repostatus.org badge, and any other badges. If the README has many more badges, you might want to consider using a table for badges: see this example. Such a table should be more wide than high. (Note that the badge for pyOpenSci peer-review will be provided upon acceptance.)
[x] Short description of goals of package, with descriptive links to all vignettes (rendered, i.e. readable, cf the documentation website section) unless the package is small and there’s only one vignette repeating the README.
[x] Installation instructions
[ ] Any additional setup required (authentication tokens, etc)
[ ] Brief demonstration usage
[x] Direction to more detailed documentation (e.g. your documentation files or website).
[ ] If applicable, how the package compares to other similar packages and/or how it relates to other packages
[ ] Citation information

Usability

[x] The documentation is easy to find and understand
[ ] The need for the package is clear
[x] All functions have documentation and associated examples for use

Functionality

[x] Installation: Installation succeeds as documented.
[x] Functionality: Any functional claims of the software been confirmed.
[x] Performance: Any performance claims of the software been confirmed.
[x] Automated tests: Tests cover essential functions of the package and a reasonable range of inputs and conditions. All tests pass on the local machine.
[x] Continuous Integration: Has continuous integration, such as Travis CI, AppVeyor, CircleCI, and/or others.
[x] Packaging guidelines: The package conforms to the pyOpenSci packaging guidelines.

For packages co-submitting to JOSS

[ ] The package has an obvious research application according to JOSS's definition in their submission requirements.

Note: Be sure to check this carefully, as JOSS's submission requirements and scope differ from pyOpenSci's in terms of what types of packages are accepted.

The package contains a paper.md matching JOSS's requirements with:

[ ] A short summary describing the high-level functionality of the software
[ ] Authors: A list of authors with their affiliations
[ ] A statement of need clearly stating problems the software is designed to solve and its target audience.
[ ] References: with DOIs for all those that have one (e.g. papers, datasets, software).

Final approval (post-review)

[ ] The author has responded to my review and made changes to my satisfaction. I recommend approving this package.

Estimated hours spent reviewing: 1.5 hrs

Review Comments

Great work with this package! The functionalities are extremely useful and I especially appreciate the package catering to DSCI 552. Below I have mentioned some points that you can improve on and make the package more user friendly:

In the Readme, it would nice to write information more in bullet points and highlight key things you would definitely want the users to take away. I am especially talking about the "Package context within the Python ecosystem" section, I think it would make this section more clear if each bullet covers a specific point.
Following up on the "Package context within the Python ecosystem" section, I feel there needs to be more elaboration on how this package compares to other packages out there and what is different about this package. Right now you mention what strapvispy builds on and also briefly mention scikit learn, but you don't exactly talk about how those packages do boostrapping and how is strapvizpy able to streamline the process. Also it would good to elaborate more on some of the boostrapping jargon.
It is a bit hard to understand your source code and I would recommend that more comments be added to explain what some of the branches do and also to separate the branches with spaces. It would be nice to have a small phrase describing what exception handling you are doing.
In your ReadMe or documentation it would also be nice to mention if users need a certain python version to install your package.
I also noticed some small inconsistencies in your package, like for example your package is named "strapvizpy" and this is how your users would install and import it, but sometimes in your Readme and documentation you instead say "StrapvizPy". I think in general it would be good to stick with "strapvizpy" making it less confusing. Additionally, in your test code while some error checks are well documented through comments, others lack them.

All in all, great job! It very clear that a lot of thought and effort has been put into this package. It is very detailed and it would be interesting to see if this will be used by the next cohort in DSCI 552!

Package Review

Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide

[x] As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

Documentation

The package includes all the following forms of documentation:

[x] A statement of need clearly stating problems the software is designed to solve and its target audience in README
[x] Installation instructions: for the development version of package and any non-standard dependencies in README
[x] Vignette(s) demonstrating major functionality that runs successfully locally
[x] Function Documentation: for all user-facing functions
[x] Examples for all user-facing functions
[x] Community guidelines including contribution guidelines in the README or CONTRIBUTING.
[x] Metadata including author(s), author e-mail(s), a url, and any other relevant metadata e.g., in a setup.py file or elsewhere.

Readme requirements The package meets the readme requirements below:

[x] Package has a README.md file in the root directory.

The README should include, from top to bottom:

[x] The package name
[x] Badges for continuous integration and test coverage, a repostatus.org badge, and any other badges. If the README has many more badges, you might want to consider using a table for badges: see this example. Such a table should be more wide than high. (Note that the badge for pyOpenSci peer-review will be provided upon acceptance.)
[x] Short description of goals of package, with descriptive links to all vignettes (rendered, i.e. readable, cf the documentation website section) unless the package is small and there's only one vignette repeating the README.
[x] Installation instructions
[x] Any additional setup required (authentication tokens, etc)
[x] Brief demonstration usage
[x] Direction to more detailed documentation (e.g. your documentation files or website).
[x] If applicable, how the package compares to other similar packages and/or how it relates to other packages
[x] Citation information

Usability

[x] The documentation is easy to find and understand
[x] The need for the package is clear
[x] All functions have documentation and associated examples for use

Functionality

[x] Installation: Installation succeeds as documented.
[x] Functionality: Any functional claims of the software been confirmed.
[x] Performance: Any performance claims of the software been confirmed.
[x] Automated tests: Tests cover essential functions of the package and a reasonable range of inputs and conditions. All tests pass on the local machine.
[x] Continuous Integration: Has continuous integration, such as Travis CI, AppVeyor, CircleCI, and/or others.
[x] Packaging guidelines: The package conforms to the pyOpenSci packaging guidelines.

For packages co-submitting to JOSS

[ ] The package has an obvious research application according to JOSS's definition in their submission requirements.

Note: Be sure to check this carefully, as JOSS's submission requirements and scope differ from pyOpenSci's in terms of what types of packages are accepted.

The package contains a paper.md matching JOSS's requirements with:

[ ] A short summary describing the high-level functionality of the software
[ ] Authors: A list of authors with their affiliations
[ ] A statement of need clearly stating problems the software is designed to solve and its target audience.
[ ] References: with DOIs for all those that have one (e.g. papers, datasets, software).

Final approval (post-review)

[ ] The author has responded to my review and made changes to my satisfaction. I recommend approving this package.

Estimated hours spent reviewing: 1

Review Comments

Wonderful little package.

Excellent documentation.
Can remove the "pip install" bit. Very common issue in this block's packages.
Usage syntax is a little funny what with needing the prefix "bootstrap.calculate_boot_stats()"
I saw another comment mention lack usage examples. Not sure if it was submitted before the end of the appropriate milestone, but I thought the usage examples in ReadTheDocs was more than enough to get started with the package.
Similar to my feedback in the R version of this package, I find the red v-line for the estimate against the blue histogram a little jarring on the eyes.
I really appreciate human-readable table titles! I think maybe it would be cool to have an argument in the tabulate_stats() function that would let you pick human readable labels vs coding-friendly ones (with underscores and small letters)? I realize that the package's purpose is primarily to whip out publish-ready tables and visualizations easily, but seems like this would be a useful and simple feature.
I love the division into "Bootstap" and "Display" sections. So nice to have low-key suggestions for workflow within documentation.
Wonderful job friends, you should be proud! 😊🌸

UBC-MDS / software-review-2022