Submission: instaeda (Python)

[roycezhou]

Submitting Author:

• Justin Fu (@jufu)
• Selma Duric (@d-sel)
• Zeliha Ural Merpez (@zmerpez)
• Siqi Zhou (@roycezhou)

Package Name: instaeda One-Line Description of Package: Quick and easy way to clean data and build exploratory data analysis plots Repository Link: instaeda_py Version submitted: 0.1.6 Editor: TBD
Reviewer 1: TBD
Reviewer 2: TBD
Archive: TBD
Version accepted: TBD

---

Description

• Include a brief paragraph describing what your package does:

Instaeda provides a quick and easy way to clean data and build exploratory data analysis plots. This idea came up as we have been building data projects for quite some time now in the UBC MDS program. We noticed that there are some repetitive activities that occur when we first explore the data. This project will help you take a given raw data set an conduct some data cleansing and plotting with a minimal amount of code. There are four main components of this package which are data checking, data cleaning, exploratory visualization including numerical correlation plotting and basic distribution plotting by datatype.

Scope

• Please indicate which category or categories this package falls under:
  • [ ] Data retrieval
  • [ ] Data extraction
  • [ ] Data munging
  • [ ] Data deposition
  • [ ] Reproducibility
  • [ ] Geospatial
  • [ ] Education
  • [x] Data visualization*
  • [x] Exploratory data analysis

* Please fill out a pre-submission inquiry before submitting a data visualization package. For more info, see notes on categories of our guidebook.

• Explain how the and why the package falls under these categories (briefly, 1-2 sentences):

The package falls under data visualization and exploratory data analysis since it performs to explore the raw data with data checking, data cleaning, exploratory visualization including numerical correlation plotting and basic distribution plotting by datatype with a minimal amount of code.

• Who is the target audience and what are scientific applications of this package?

The target audience of this package includes anyone who has the requirements to clean data and build exploratory data analysis plots. For instance, students with computer science and data science background might be the target audience. Besides, data scientists, data engineers, statisticians are possible target users as well.

• Are there other Python packages that accomplish the same thing? If so, how does yours differ?

There are some Python packages to conduct exploratory data analysis (EDA) such as "Pandas Profiling" and "Autoviz". However, our Python package provides a different working flow and functionality. There are four main components of our Python package and each of the functions has its innovation points. For example, our data checking function gives users a sense of the whole distribution summary of the raw data. Although there are some built-in functions to summarize the missing data, mean, standard deviation, etc, our function implements straightforward plotting including several bar charts to describe the data summary with metrics considering the numeric columns, factor columns, complete rows and missing observations.

• If you made a pre-submission enquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted:

NA

Technical checks

For details about the pyOpenSci packaging requirements, see our packaging guide. Confirm each of the following by checking the box. This package:

• [x] does not violate the Terms of Service of any service it interacts with.
• [ ] has an OSI approved license.
• [x] contains a README with instructions for installing the development version.
• [x] includes documentation with examples for all functions.
• [x] contains a vignette with examples of its essential functions and uses.
• [x] has a test suite.
• [x] has continuous integration, such as Travis CI, AppVeyor, CircleCI, and/or others.

Publication options

• [ ] Do you wish to automatically submit to the Journal of Open Source Software? If so:

JOSS Checks

- [ ] The package has an **obvious research application** according to JOSS's definition in their [submission requirements][JossSubmissionRequirements]. Be aware that completing the pyOpenSci review process **does not** guarantee acceptance to JOSS. Be sure to read their submission requirements (linked above) if you are interested in submitting to JOSS. - [ ] The package is not a "minor utility" as defined by JOSS's [submission requirements][JossSubmissionRequirements]: "Minor ‘utility’ packages, including ‘thin’ API clients, are not acceptable." pyOpenSci welcomes these packages under "Data Retrieval", but JOSS has slightly different criteria. - [ ] The package contains a `paper.md` matching [JOSS's requirements][JossPaperRequirements] with a high-level description in the package root or in `inst/`. - [ ] The package is deposited in a long-term repository with the DOI: *Note: Do not submit your package separately to JOSS*

Are you OK with Reviewers Submitting Issues and/or pull requests to your Repo Directly?

This option will allow reviewers to open smaller issues that can then be linked to PR's rather than submitting a more dense text based review. It will also allow you to demonstrate addressing the issue via PR links.

• [x] Yes I am OK with reviewers submitting requested changes as issues to my repo. Reviewers will then link to the issues in their submitted review.

Code of conduct

• [x] I agree to abide by pyOpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.

P.S. *Have feedback/comments about our review process? Leave a comment here

Editor and Review Templates

Editor and review templates can be found here

[trevorki]

Hello, please find my package review below:

Package Review

• [x] As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

Documentation

The package includes all the following forms of documentation:

• [x] A statement of need clearly stating problems the software is designed to solve and its target audience in README

• [ ] Installation instructions: for the development version of package and any non-standard dependencies in README

  • The installation instructions on the README: pip install -i https://test.pypi.org/simple/ instaeda are different from those on readthedocs.io: pip install -u instaeda, which caused an error: no such option: -u
  • Additional dependencies that were not listed had to be installed: threadpoolctl, scipy, joblib
  • The git clone instructions didn't work as written on readthedocs.io:

    $ git clone git://github.com/jufu/instaeda
    Cloning into 'instaeda'...
    fatal: remote error: 
    Repository not found.

  • It works with: git clone https://github.com/UBC-MDS/instaeda_py.git

• [ ] Vignette(s) demonstrating major functionality that runs successfully locally

  • The examples that are given in README don't show what the output is. I couldn't find any more thorough vignettes other than these

• [x] Function Documentation: for all user-facing functions

  • The docstring of plot_basic_distributions has unclear wording in the description of the return type

• [x] Examples for all user-facing functions

• [x] Community guidelines including contribution guidelines in the README or CONTRIBUTING.

• [x] Metadata including author(s), author e-mail(s), a url, and any other relevant metadata e.g., in a setup.py file or elsewhere.

Readme requirements

The package meets the readme requirements below:

• [x] Package has a README.md file in the root directory.

The README should include, from top to bottom:

• [x] The package name
• [x] Badges for continuous integration and test coverage, a repostatus.org badge, and any other badges. If the README has many more badges, you might want to consider using a table for badges: see this example. Such a table should be more wide than high. (Note that the badge for pyOpenSci peer-review will be provided upon acceptance.)
• [ ] Short description of goals of package, with descriptive links to all vignettes (rendered, i.e. readable, cf the documentation website section) unless the package is small and there’s only one vignette repeating the README.
  • There is no output shown in any of the vignettes/examples
• [x] Installation instructions
• [ ] Any additional setup required (authentication tokens, etc)
  • N/A
• [x] Brief demonstration usage
• [x] Direction to more detailed documentation (e.g. your documentation files or website).
• [ ] If applicable, how the package compares to other similar packages and/or how it relates to other packages
  • similar packages are listed, but no comparisons are made between this package and existing ones
• [ ] Citation information
  • No citations

Usability

Reviewers are encouraged to submit suggestions (or pull requests) that will improve the usability of the package as a whole. Package structure should follow general community best-practices. In general please consider:

• [ ] The documentation is easy to find and understand
  • With one exception... see note on docstring of plot_basic_distributions
• [x] The need for the package is clear
• [x] All functions have documentation and associated examples for use

Functionality

• [x] Installation: Installation succeeds as documented.
• [x] Functionality: Any functional claims of the software been confirmed.
• [ ] Performance: Any performance claims of the software been confirmed.
  • N/A
• [X] Automated tests: Tests cover essential functions of the package and a reasonable range of inputs and conditions. All tests pass on the local machine.
• [x] Continuous Integration: Has continuous integration, such as Travis CI, AppVeyor, CircleCI, and/or others.
• [x] Packaging guidelines: The package conforms to the pyOpenSci packaging guidelines.

For packages co-submitting to JOSS

• [ ] The package has an obvious research application according to JOSS's definition in their submission requirements.

Note: Be sure to check this carefully, as JOSS's submission requirements and scope differ from pyOpenSci's in terms of what types of packages are accepted.

The package contains a paper.md matching JOSS's requirements with:

• [ ] A short summary describing the high-level functionality of the software
• [ ] Authors: A list of authors with their affiliations
• [ ] A statement of need clearly stating problems the software is designed to solve and its target audience.
• [ ] References: with DOIs for all those that have one (e.g. papers, datasets, software).

Final approval (post-review)

• [ ] The author has responded to my review and made changes to my satisfaction. I recommend approving this package.

Estimated hours spent reviewing: 2.5

Review Comments

Thanks for the submission. This looks like a useful package that just needs some minor polishing to be ready to go. I left thorough notes inline (above) but have summarized the main points here:

• plot_intro(penguin_df) It's not clear how to interpret the "All Missing Columns" entry in the plot and why it is showing near 240%. Could you explain what this quantity is meant to represent?

• See notes above on Installation. There are a few little details to tidy up

  • There is some inconsistency between the README and the readthedocs.io installation instructions. Please check that the installation instruction is the one you want and has no typos.
  • The cloning code given does not work: git clone git://github.com/jufu/instaeda. Should this be instaeda_py instead?
  • Undeclared dependencies that prevented installation as described: threadpoolctl, scipy, joblib

• See notes above on Vignettes.

  • There are no vignettes demonstrating the usage and the expected output. It would be really helpful to have visuals showing what the output will look like.

• See notes above on Documentation.

  • The docstring of plot_basic_distributions has unclear wording in the description of the return type. It looks like it might be mid-edit.

Please let me know if you have questions or want to discuss anything further.

[wang-rui]

Package Review

Please check off boxes as applicable, and elaborate in the comments below. Your review is not limited to these topics, as described in the reviewer guide

• [x] As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

Documentation

The package includes all the following forms of documentation:

• [x] A statement of need clearly stating problems the software is designed to solve and its target audience in README
• [x] Installation instructions: for the development version of the package and any non-standard dependencies in README Worked for me by cloning the repo and installing locally, although the installation guide on readme file is a bit misleading.
• [ ] Vignette(s) demonstrating major functionality that runs successfully locally I don't think there is a vignette or full example in your usage section yet.
• [x] Function Documentation: for all user-facing functions
• [x] Examples for all user-facing functions
• [x] Community guidelines including contribution guidelines in the README or CONTRIBUTING.
• [x] Metadata including author(s), author e-mail(s), a url, and any other relevant metadata e.g., in a setup.py file or elsewhere.

Readme requirements The package meets the readme requirements below:

• [x] Package has a README.md file in the root directory.

The README should include, from top to bottom:

• [x] The package name
• [x] Badges for continuous integration and test coverage, a repostatus.org badge, and any other badges. If the README has many more badges, you might want to consider using a table for badges: see this example. Such a table should be more wide than high. (Note that the badge for pyOpenSci peer-review will be provided upon acceptance.)
• [ ] Short description of goals of the package, with descriptive links to all vignettes (rendered, i.e. readable, cf the documentation website section) unless the package is small and there’s only one vignette repeating the README.
• [x] Installation instructions
• [ ] Any additional setup required (authentication tokens, etc)
• [x] Brief demonstration usage
• [x] Direction to more detailed documentation (e.g. your documentation files or website).
• [ ] If applicable, how the package compares to other similar packages and/or how it relates to other packages Not quite similar to other existing packages that I'm aware of.
• [ ] Citation information Probably could use some acknowledgment for the external packages or so?

Usability

Reviewers are encouraged to submit suggestions (or pull requests) that will improve the usability of the package as a whole. The package structure should follow general community best-practices. In general please consider:

• [x] The documentation is easy to find and understand Yes to most of the documentation, except for the readthedocs ( check below comment)
• [x] The need for the package is clear
• [x] All functions have documentation and associated examples for use

Functionality

-[x] Installation: Installation succeeds as documented.

• [x] Functionality: Any functional claims of the software been confirmed.
• [ ] Performance: Any performance claims of the software been confirmed.
• [x] Automated tests: Tests cover essential functions of the package and a reasonable range of inputs and conditions. All tests pass on the local machine.
• [x] Continuous Integration: Has continuous integration, such as Travis CI, AppVeyor, CircleCI, and/or others.
• [x] Packaging guidelines: The package conforms to the pyOpenSci packaging guidelines.

For packages co-submitting to JOSS

• [ ] The package has an obvious research application according to JOSS's definition in their submission requirements.

Note: Be sure to check this carefully, as JOSS's submission requirements and scope differ from pyOpenSci's in terms of what types of packages are accepted.

The package contains a paper.md matching JOSS's requirements with:

• [ ] A short summary describing the high-level functionality of the software
• [ ] Authors: A list of authors with their affiliations
• [ ] A statement of need clearly stating problems the software is designed to solve and its target audience.
• [ ] References: with DOIs for all those that have one (e.g. papers, datasets, software).

Final approval (post-review)

• [ ] The author has responded to my review and made changes to my satisfaction. I recommend approving this package.

Estimated hours spent reviewing:

2.5

Review Comments

Hi Justin, Selma, Zeliha, Siqi, Thanks for putting this the package together. I had fun playing around with the instaeda_py package and they are definitely going to make our data exploration more convenient.

While I was playing with the package, I noted a few things down below, and hope it could help to improve the overall experience of the package more.

Installation

I tried following package installation instructions following your README file, but sadly it fails in an empty Python 3 environment, which gave me the exception below:

ERROR: Cannot install instaeda==0.1.0, instaeda==0.1.1, instaeda==0.1.2, instaeda==0.1.6, instaeda==0.1.7 and instaeda==0.1.8 because these package versions have conflicting dependencies.

The conflict is caused by: instaeda 0.1.8 depends on vega-datasets<0.10.0 and >=0.9.0 instaeda 0.1.7 depends on vega-datasets<0.10.0 and >=0.9.0 instaeda 0.1.6 depends on vega-datasets<0.10.0 and >=0.9.0 instaeda 0.1.2 depends on vega-datasets<0.10.0 and >=0.9.0 instaeda 0.1.1 depends on vega-datasets<0.10.0 and >=0.9.0 instaeda 0.1.0 depends on vega-datasets<0.10.0 and >=0.9.0

> However, it seems I could install it after adding the `--extra-index-url` option.

pip install -i https://test.pypi.org/simple --extra-index-url https://pypi.org/simple instaeda

#### Tests

> I really like how many test cases you created and they seem to be in large detail. 

> I tried to clone the repo to my laptop and run the pytest locally. The command I used was `poetry install` then `poetry run pytest`, then I ran into an error below:

ModuleNotFoundError: No module named '_cffi_backend'

> I'm not sure if it is an issue from my environment as I created a new environment to test this...:(

#### Function Documentation

> I really loved your function documentations. The examples in the docstrings are very clear to understand. 

#### Documentation

> I only see Justin’s name in the copyright info.  I wonder if this should be all members of the team.
> It seems you don't have a license discussion issue...:/
> The usage part on the [readthedocs](https://instaeda.readthedocs.io/en/latest/usage.html) is not there yet.