Submission: PyDataPeek (Python)

Submitting Author: Monique Wong (@moniquewong), Alistair Clark (@alistair-clark) , Miro Hu (@mirohu), Thomas Pin (@MrThomasPin) Package Name: PyDataPeek One-Line Description of Package: Simple EDA for .csv or .xlsx documents Repository Link: Repo Link Version submitted:
Editor: @kvarada
Reviewer 1: Elliott Ribner @elliott-ribner Reviewer 2: Aman Kumar Garg @amank90 Archive: TBD
Version accepted: TBD

Description

PyDataPeek is a package that enables data scientists to efficiently generate a visual summary of a dataset. This package includes functions that show the size of the dataset, a visual summary of missing data, a sample of the dataset showing the data types as well as exploratory visualizations for quantitative and qualitative data.

Scope

Please indicate which category or categories this package falls under:
- [ ] Data retrieval
- [X] Data extraction
- [ ] Data munging
- [ ] Data deposition
- [ ] Reproducibility
- [ ] Geospatial
- [ ] Education
- [x] Data visualization*

* Please fill out a pre-submission inquiry before submitting a data visualization package. For more info, see this section of our guidebook.

Explain how and why the package falls under these categories (briefly, 1-2 sentences):

PyDataPeek is used for quick visualization that individuals can use to understand their data better and have report ready visualizations for their documents.

Who is the target audience and what are scientific applications of this package?

Business individuals who want quick and effective visualizations of their data.

Are there other Python packages that accomplish the same thing? If so, how does yours differ?

Several Python packages are available that support exploratory data analysis but none are specific to the targeted use cases here - a simple and technologically friendly way of summarizing data.

Pandas Profiling: This package generates a report of a dataframe that has some of the features in the proposal. Our package will differ from this by offering the user simpler summaries that are friendlier to a non-technical audience.
Python Pandas: Our package will leverage pandas functionality to manipulate dataframes. Our package functionality overlaps with some functions such as pd.describe which computes summary statistics for dataframes. The package differs in that it aims to offer summary statistics dependent on data type, including long form text data.
Python Altair, Python Seaborn and Python WordCloud: These visualization packages will be used to create visualizations that summarize the dataset as well as user-defined features in the dataset.

If you made a pre-submission enquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted:

Technical checks

For details about the pyOpenSci packaging requirements, see our packaging guide. Confirm each of the following by checking the box. This package:

[X] does not violate the Terms of Service of any service it interacts with.
[X] has an OSI approved license
[X] contains a README with instructions for installing the development version.
[X] includes documentation with examples for all functions.
[X] contains a vignette with examples of its essential functions and uses.
[X] has a test suite.
[X] has continuous integration, such as Travis CI, AppVeyor, CircleCI, and/or others.

Publication options

[ ] Do you wish to automatically submit to the Journal of Open Source Software? If so:

JOSS Checks

- [ ] The package has an **obvious research application** according to JOSS's definition in their [submission requirements](https://joss.readthedocs.io/en/latest/submitting.html#submission-requirements). Be aware that completing the pyOpenSci review process **does not** guarantee acceptance to JOSS. Be sure to read their submission requirements (linked above) if you are interested in submitting to JOSS. - [ ] The package is not a "minor utility" as defined by JOSS's [submission requirements](https://joss.readthedocs.io/en/latest/submitting.html#submission-requirements): "Minor 'utility' packages, including 'thin' API clients, are not acceptable." pyOpenSci welcomes these packages under "Data Retrieval", but JOSS has slightly different criteria. - [ ] The package contains a `paper.md` matching [JOSS's requirements](https://joss.readthedocs.io/en/latest/submitting.html#what-should-my-paper-contain) with a high-level description in the package root or in `inst/`. - [ ] The package is deposited in a long-term repository with the DOI: *Note: Do not submit your package separately to JOSS*

Are you OK with Reviewers Submitting Issues and/or pull requests to your Repo Directly?

This option will allow reviewers to open smaller issues that can then be linked to PR's rather than submitting a more dense text based review. It will also allow you to demonstrate addressing the issue via PR links.

[x] Yes I am OK with reviewers submitting requested changes as issues to my repo. Reviewers will then link to the issues in their submitted review.

Code of conduct

[x] I agree to abide by pyOpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.

P.S. Have feedback/comments about our review process? Leave a comment here

Editor and Review Templates

Editor and review templates can be found here

Package Review

Please check off boxes as applicable, and elaborate in the comments below. Your review is not limited to these topics, as described in the reviewer guide

[x] As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

Documentation

The package includes all the following forms of documentation:

[x] A statement of need clearly stating problems the software is designed to solve and its target audience in README
[x] Installation instructions: for the development version of package and any non-standard dependencies in README
[x] Vignette(s) demonstrating major functionality that runs successfully locally
[x] Function Documentation: for all user-facing functions
[x] Examples for all user-facing functions
[x] Community guidelines including contribution guidelines in the README or CONTRIBUTING.
[ ] Metadata including author(s), author e-mail(s), a url, and any other relevant metadata e.g., in a setup.py file or elsewhere.

Readme requirements The package meets the readme requirements below:

[x] Package has a README.md file in the root directory.

The README should include, from top to bottom:

[x] The package name
[x] Badges for continuous integration and test coverage, the badge for pyOpenSci peer-review once it has started (see below), a repostatus.org badge, and any other badges. If the README has many more badges, you might want to consider using a table for badges, see this example, that one and that one. Such a table should be more wide than high.
[x] Short description of goals of the package, with descriptive links to all vignettes (rendered, i.e. readable, cf the documentation website section) unless the package is small and there’s only one vignette repeating the README.
[x] Installation instructions
[x] Any additional setup required (authentication tokens, etc)
[x] Brief demonstration usage
[x] Direction to more detailed documentation (e.g. your documentation files or website).
[x] If applicable, how the package compares to other similar packages and/or how it relates to other packages
[x] Citation information

Functionality

[ ] Installation: Installation succeeds as documented.
[x] Functionality: Any functional claims of the software been confirmed.
[x] Performance: Any performance claims of the software been confirmed.
[x] Automated tests: Tests cover essential functions of the package and a reasonable range of inputs and conditions. All tests pass on the local machine.
[x] Continuous Integration: Has continuous integration, such as Travis CI, AppVeyor, CircleCI, and/or others.

Final approval (post-review)

[ ] The author has responded to my review and made changes to my satisfaction. I recommend approving this package.

Estimated hours spent reviewing:

Review Comments

Feedback 1:

For installation, a different command was used. The command is below:

pip install pydatapeek --extra-index-url=https://test.pypi.org/simple/

Feedback 2:

Please mention author email ids in contributors.md.

Feedback 3:

docstrings to the test cases could have been better like the format we follow for other python functions

Feedback 4

from PyDataPeek import missing_data_overview
import pandas as pd
import matplotlib.pyplot as plt

df = pd.DataFrame(
        {'x1': [1.,2,3,4],
         "x2": ["a","b","","d"]})

plt.show(missing_data_overview._make_plot(df))

I don't get any missing value in the plot but when I do this.

from PyDataPeek import missing_data_overview
import pandas as pd
import matplotlib.pyplot as plt

df = pd.DataFrame(
        {'x1': [1.,2,3,4],
         "x2": ["a","b",None,"d"]})

plt.show(missing_data_overview._make_plot(df))

I start seeing the missing values.

Feedback 5

In sample_data.py, it will take the sample data that potentially has no missing value. If it is expected to pass the data with no missing value then you can ignore this.

results = pd.DataFrame({'sample_record': df.iloc[1]})

Feedback 6

In the cloud generating function, I am getting the column name in the cloud name as well. Is it possible to not have it?


import pandas as pd
import matplotlib.pyplot as plt
from wordcloud import WordCloud, STOPWORDS
from PyDataPeek import word_bubble

df = pd.DataFrame(
        {'x1': ["play cricket","game","amazing","joke"]})

formated_words, stopwords = word_bubble._make_formated_words(df)

plt.show(word_bubble._make_cloud(formated_words, stopwords, 10, 100, 100))

Reviewer: Elliott

Package Review

[x] As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

Documentation

The package includes all the following forms of documentation:

[x] A statement of need clearly stating problems the software is designed to solve and its target audience in README
[x] Installation instructions: for the development version of package and any non-standard dependencies in README
[x] Vignette(s) demonstrating major functionality that runs successfully locally
[x] Function Documentation: for all user-facing functions
[x] Examples for all user-facing functions
[x] Community guidelines including contribution guidelines in the README or CONTRIBUTING.
[x] Metadata including author(s), author e-mail(s), a url, and any other relevant metadata e.g., in a setup.py file or elsewhere.

Readme requirements The package meets the readme requirements below:

[x] Package has a README.md file in the root directory.

The README should include, from top to bottom:

[x] The package name
[x] Badges for continuous integration and test coverage, the badge for pyOpenSci peer-review once it has started (see below), a repostatus.org badge, and any other badges. If the README has many more badges, you might want to consider using a table for badges, see this example, that one and that one. Such a table should be more wide than high.
[x] Short description of goals of package, with descriptive links to all vignettes (rendered, i.e. readable, cf the documentation website section) unless the package is small and there’s only one vignette repeating the README.
[x] Installation instructions
[x] Any additional setup required (authentication tokens, etc)
[x] Brief demonstration usage
[x] Direction to more detailed documentation (e.g. your documentation files or website).
[ ] If applicable, how the package compares to other similar packages and/or how it relates to other packages
[x] Citation information

Functionality

[ ] Installation: Installation succeeds as documented.
I received an error when trying to install. I think this is an issue with poetry, that it does not auto install dependencies.

Collecting pydatapeek
  Downloading https://test-files.pythonhosted.org/packages/1e/27/5a49ffb2261be9541e88d0ae9e076862e2a8029d779a78812a5f210f850f/pydatapeek-0.1.9-py3-none-any.whl
ERROR: Could not find a version that satisfies the requirement altair_saver<0.2.0,>=0.1.0 (from pydatapeek) (from versions: none)
ERROR: No matching distribution found for altair_saver<0.2.0,>=0.1.0 (from pydatapeek)

[x] Functionality: Any functional claims of the software been confirmed.
[x] Performance: Any performance claims of the software been confirmed.
[x] Automated tests: Tests cover essential functions of the package and a reasonable range of inputs and conditions. All tests pass on the local machine.
[x] Continuous Integration: Has continuous integration, such as Travis CI, AppVeyor, CircleCI, and/or others.
[x] Packaging guidelines: The package conforms to the pyOpenSci packaging guidelines.

Final approval (post-review)

[x] The author has responded to my review and made changes to my satisfaction. I recommend approving this package.

Estimated hours spent reviewing: 4

---#### Review Comments

Altogether, great job on the project. I think there is many useful features contained in the package, and it is well implemented! I found the code and structure, well written, and well documented. I found very few points to improve, but if time allowed to fix there is three things worth noting:

Unused file: I think there is an unused file titled pbc in the test directory.
I could not understand the heatmap documentation. To be more specific I did not understand how the function would be used or interpreted from reading the docs. Looking closer at the visualization on the readme, there is some labels on the edge of the image but they were very hard to read. I might suggest a more involved example of how it could be used, with a written description on how to interpret the output.
Lastly, this seems to be an issue with all of the projects including my own, but the fact that you need to manually install all the dependencies before pip installing seems like an issue. I think pip packages should generally install dependencies automatically.

Thank you for your time.

Thanks,

Elliott

@amank90 Thanks for your feedback. Below are some comments and updates we have made to our package based on your feedback

[x] Feedback 1) We have updated the install instructions
[x] Feedback 2) We have included e-mails of all team members
[x] Feedback 3) We have kept one-line documentation for tests instead of using full docstrings, meeting course requirements
[x] Feedback 4) Updated documentation to show that only NaN values will be captured
[x] Feedback 5) The intent of the function is to show the data types and does not have anything to do with missing data. A data frame with no missing data should / can be passed into the function.
[ ] Feedback 6) Will not be addressed

@elliott-ribner Thanks for your feedback. Below are some comments and updates we have made to our package based on your feedback.

[ ] Installation error: still figuring out what is causing this
[x] Unused file pbc in test directory: We have removed it
[x] Improved example of heat map function: updated so that documentation is clearer and font size is bigger
[ ] Manual dependencies install: have not addressed

UBC-MDS / software-review