Submission: Group 3: customer_complaint_analyzer

Submitting authors: @tieandrews @lukeyf @dhruvinishar

Repository: https://github.com/UBC-MDS/customer_complaint_analyzer Report link: https://ubc-mds.github.io/customer_complaint_analyzer/reports/final_report.html Abstract/executive summary: We aim to investigate, analyze, and report using the customer complaint dataset. This dataset is published in DATA.GOV and it is intended for public access and use. This dataset is a collection of customer complaints regarding their purchased financial products. It contains information on the summary and content of the complaint, the responses from the companies, and whether the customer disputed after companies response. We aim to answer the following inferential and/or predictive questions: Can we predict whether a customer is going to dispute based on their complaint and the company's response?

We plan to analyze the data using a mix of tabular and natural language processing tools like the bag-of-words representation and apply proper categorical transformations to the company's responses. The data were transformed and analyzed using 5 different models using cross-validations and the recall score was selected for model evaluation. The results were presented in the report file and the logistic regression model was selected as the best model due to its high performance and interpretability.

Editor: @flor14 Reviewer: Dhillon Robin, Chan Morris, Zaiatc Stephen

[x] I agree to abide by MDS's Code of Conduct during the review process and in maintaining my package should it be accepted.

Data analysis review checklist

Reviewer: robindhillon1

Conflict of interest

[x] As the reviewer I confirm that I have no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the MDS code of conduct.

General checks

[x] Repository: Is the source code for this data analysis available? Is the repository well organized and easy to navigate?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?

Documentation

[x] Installation instructions: Is there a clearly stated list of dependencies?
[x] Example usage: Do the authors include examples of how to use the software to reproduce the data analysis?
[x] Functionality documentation: Is the core functionality of the data analysis software documented to a satisfactory level?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Code quality

[x] Readability: Are scripts, functions, objects, etc., well named? Is it relatively easy to understand the code?
[x] Style guidelides: Does the code adhere to well known language style guides?
[x] Modularity: Is the code suitably abstracted into scripts and functions?
[x] Tests: Are there automated tests or manual steps described so that the function of the software can be verified? Are they of sufficient quality to ensure software robsutness?

Reproducibility

[x] Data: Is the raw data archived somewhere? Is it accessible?
[x] Computational methods: Is all the source code required for the data analysis available?
[x] Conditions: Is there a record of the necessary conditions (software dependencies) needed to reproduce the analysis? Does there exist an easy way to obtain the computational environment needed to reproduce the analysis?
[ ] Automation: Can someone other than the authors easily reproduce the entire data analysis?

Analysis report

[x] Authors: Does the report include a list of authors with their affiliations?
[x] What is the question: Do the authors clearly state the research question being asked?
[x] Importance: Do the authors clearly state the importance for this research question?
[x] Background: Do the authors provide sufficient background information so that readers can understand the report?
[x] Methods: Do the authors clearly describe and justify the methodology used in the data analysis? Do the authors communicate any assumptions or limitations of their methodologies?
[x] Results: Do the authors clearly communicate their findings through writing, tables and figures?
[x] Conclusions: Are the conclusions presented by the authors correct?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
[x] Writing quality: Is the writing of good quality, concise, engaging?

Estimated hours spent reviewing: 1.5

Review Comments:

Your analysis.py script is beautifully written; the style and the cleanliness is amazing and the code is easy to read.
The project proposal is concise and conveys the information very well without being superfluous.
The image that shows how the pipeline works is very professional, clean, and helps understand the workflow of the project. I love it.
Instructions to run the analysis are clear.
The final report does a great job of conveying the story and convinces the reader why the analysis is important.
The raw data is not archived on GitHub but upon talking to the group, I discovered that their dataset is very large (> 2GB) which is why they didn't put it up. The commands to download the data, however, works correctly. Hence, I gave a check-mark for the following question: Is the raw data archived somewhere? Is it accessible? Because although not archived anywhere due to the large size, it is accessible via their scripts.
All in all, great work!

Suggestions

Although understandable that the raw data cannot be archived on GitHub, maybe put a small subset of the data in data folder so that reviewers can have an idea of what the data consists of and how it's structured.
There is a makefile present but no instructions on how to run it. I was able to run it using make all but someone without such knowledge would not know how to replicate the analysis using make. The instructions should be added to the README.
When I tried to run the analysis using make all, it seems the majority of the analysis ran successfully but it encountered some error near the end. make clean works well. This should be explored so that everything runs smoothly for Milestone 4. A screenshot of the error is shown below

Although the structure of the repo is very good, some improvements can be made such as:
- Having a dedicated folder named results for clarity. Right now there are folders such as reports and notebooks which seem a bit ambiguous at first glance, although easy to explore. For a third party unaware of the project, a folder such as results informs them right away that this is where they'd find relevant documents, figures, saved models, and so on.
Putting the final report in your reports directory would be ideal instead of simply having it in your README. If desired, it can be kept in the README but there should be a proper directory in which the final report resides.
The following question in the project proposal can be improved using proper grammar: "When submitting a claim what is your probability of getting monetary compensation, responses in a timely manner, etc. to give to consumers when submitting a complaint". Try not to use runoff sentences, end questions with a question mark, and so on. This is a good practice that will be helpful in the long term.
The plots in the final report are great. A small suggestion would be to have the metrics on the y-axis for Figure 3 with a shared x-axis that displays Score. The top of each plot can have the model type as the title.

Attribution

This was derived from the JOSE review checklist and the ROpenSci review checklist.

Data analysis review checklist

Reviewer: @morrismanfung

Conflict of interest

[x] As the reviewer I confirm that I have no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the MDS code of conduct.

General checks

[x] Repository: Is the source code for this data analysis available? Is the repository well organized and easy to navigate?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?

Documentation

[x] Installation instructions: Is there a clearly stated list of dependencies?
[x] Example usage: Do the authors include examples of how to use the software to reproduce the data analysis?
[x] Functionality documentation: Is the core functionality of the data analysis software documented to a satisfactory level?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Code quality

[x] Readability: Are scripts, functions, objects, etc., well named? Is it relatively easy to understand the code?
[x] Style guidelides: Does the code adhere to well known language style guides?
[x] Modularity: Is the code suitably abstracted into scripts and functions?
[ ] Tests: Are there automated tests or manual steps described so that the function of the software can be verified? Are they of sufficient quality to ensure software robsutness?

Reproducibility

[ ] Data: Is the raw data archived somewhere? Is it accessible?
[x] Computational methods: Is all the source code required for the data analysis available?
[x] Conditions: Is there a record of the necessary conditions (software dependencies) needed to reproduce the analysis? Does there exist an easy way to obtain the computational environment needed to reproduce the analysis?
[x] Automation: Can someone other than the authors easily reproduce the entire data analysis?

Analysis report

[x] Authors: Does the report include a list of authors with their affiliations?
[x] What is the question: Do the authors clearly state the research question being asked?
[x] Importance: Do the authors clearly state the importance for this research question?
[x] Background: Do the authors provide sufficient background information so that readers can understand the report?
[x] Methods: Do the authors clearly describe and justify the methodology used in the data analysis? Do the authors communicate any assumptions or limitations of their methodologies?
[x] Results: Do the authors clearly communicate their findings through writing, tables and figures?
[ ] Conclusions: Are the conclusions presented by the authors correct?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
[x] Writing quality: Is the writing of good quality, concise, engaging?

Estimated hours spent reviewing: 2

Review Comments:

The report rendered is professional. All of the visualizations in the repo are excellent.
Minor issues. Link to "flow" is not usable in https://github.com/UBC-MDS/customer_complaint_analyzer/blob/main/CONTRIBUTING.md#regarding-pull-requests-please-consult-this-flow-so-that-all-code-changes-are-made-through-pull-requests. In Table 2, 1 feature is written as "dropped". It seems that more features (e.g., state, zip_code) were actually dropped actually?
I am not sure whether using cross-validation per se is sufficient, without including the results with the testing set. If you are not doing hyperparameter tuning, using cross-validation seems to be equivalent to doing 5 train-test-splits. Right, its not breaking the golden rule. As I saw that you included testing data in the analysis script, the only limitation I can think about right now is that the readers may not understand the flow perfectly as we all expect some testing data set for performance evaluation. If you want to stick with cross-validation only (which may not be the case), it will be cool if you can show the variance in different folds.
Using a bar chart to represent class imbalance is effective. While around 75% of the entries are with unknown status. I personally would like to see explicitly how your team deal with those entries (dropping them, right?).
I think setting a particular model as a basis for future optimization is a nice idea to have. I want to suggest another simpler model: Consider a model which classifies all complaints as potentially disputed. It will have a recall of 100%, a precision of around 19.5%, and an f1 of 16.3. Yes the f1 is lower but the recall is much better than other models, with a little scarification in precision. Using this standard, maybe logistic regression is not that good...?
The reason why I didn't check the the box for conclusion is that I think it is too early to conclude logistic regression is suitable. First, without hyperparameter and threshold tuning, it is likely that other models can outperform logistic regression. Without these steps, it could be too optimistic or pessimistic to conclude a model is suitable or not.

Attribution

This was derived from the JOSE review checklist and the ROpenSci review checklist.

Peer Review

Data analysis review checklist

Reviewer: Stepan Zaiatc @stepanz25

Conflict of interest

[x] As the reviewer I confirm that I have no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the MDS code of conduct.

General checks

[x] Repository: Is the source code for this data analysis available? Is the repository well organized and easy to navigate?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?

Documentation

[x] Installation instructions: Is there a clearly stated list of dependencies?
[x] Example usage: Do the authors include examples of how to use the software to reproduce the data analysis?
[x] Functionality documentation: Is the core functionality of the data analysis software documented to a satisfactory level?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support No info on how to seek support

Code quality

[x] Readability: Are scripts, functions, objects, etc., well named? Is it relatively easy to understand the code?
[x] Style guidelides: Does the code adhere to well known language style guides?
[x] Modularity: Is the code suitably abstracted into scripts and functions?
[x] Tests: Are there automated tests or manual steps described so that the function of the software can be verified? Are they of sufficient quality to ensure software robustness?

Reproducibility

[ ] Data: Is the raw data archived somewhere? Is it accessible?
[x] Computational methods: Is all the source code required for the data analysis available?
[x] Conditions: Is there a record of the necessary conditions (software dependencies) needed to reproduce the analysis? Does there exist an easy way to obtain the computational environment needed to reproduce the analysis?
[x] Automation: Can someone other than the authors easily reproduce the entire data analysis?

Analysis report

[x] Authors: Does the report include a list of authors with their affiliations?
[x] What is the question: Do the authors clearly state the research question being asked?
[x] Importance: Do the authors clearly state the importance for this research question?
[ ] Background: Do the authors provide sufficient background information so that readers can understand the report? Too many technical jargons, like using field names, CountVectorizer, OneHotEncoder, f1 score, etc.
[x] Methods: Do the authors clearly describe and justify the methodology used in the data analysis? Do the authors communicate any assumptions or limitations of their methodologies?
[ ] Results: Do the authors clearly communicate their findings through writing, tables and figures? Titles of tables don’t necessarily match the data presented, and the order of information in tables don’t necessarily match the order of information in graphs. The independent variable should be on the x-axis in Figure 2.
[x] Conclusions: Are the conclusions presented by the authors correct?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
[ ] Writing quality: Is the writing of good quality, concise, engaging? Too many spelling and grammatical errors.

Estimated hours spent reviewing: 2.5 hours

Review Comments:

Please provide more detailed feedback here on what was done particularly well, and what could be improved. It is especially important to elaborate on items that you were not able to check off in the list above.

The use of different styles of graphs and charts is effective in communicating trends of the subject matter.
I found that the folder structure and organization of your repository is clear; however, in the data folder I couldn’t find any file that would contain your archived data sets. The makefile executes well, it’s very well documented with all the necessary details to make the code more reproducible.
The source code is generally easy to read. There are sufficient comments, and variable names are meaningful. Just do not forget to include name and dates of who written the script in the documentation (e.g. generate_eda.py).
So a model has been created using data from the CFPB database. As a suggestion, maybe it would be interesting to see how the model behaves on data from a different source, or even only with data from the same source, but just different date ranges.
I think it is important for the developer to specify the conditions that the training data was collected in, ie. which states, the language the complaint was submitted in, etc., as the algorithm may not work in conditions where these factors are different. Therefore, maybe it will be worth including some comments on this in your final report.
I suggest proofreading the report again before submitting the final version, as there are numerous typographic and grammatical errors, like missing a space between a word and a bracket, missing apostrophes, run-on sentences, subject-verb disagreements, unnecessary capitalization of words, improper pluralization.
Figure 1 can be improved with a larger text size that’s similar to the body text size, avoid using words that may appear as commentary in the title, and simplifying the x-axis label to just “Year”.
Figure 2 can be improved by flipping the axes, as it is customary to place the independent variable (yes/no) on the x-axis. And similar to figure 1, a larger text size would assist the reader in reading the graph.
For table 1, its title is “unique and missing value counts…”; however, the table contains a “Unique Count” column, but not a “Missing Value Count” column. Numeric values should use comma separators to make them easier to read, and I suggest adding an extra column to explain what each field means, as the field names are often cryptic to the reader.
For table 3, I would suggest reordering the columns such that they are in the same order as the list in the preceding paragraph. Similarly, the order of the graphs in Figure 3 should be adjusted as well.
In figure 3, I feel that the legend is redundant as each colour is already labelled on the x-axis of each graph. Moreover, the text size can be made larger to improve readability.

Attribution

This was derived from the JOSE review checklist and the ROpenSci review checklist.

Thank you to @stepanz25, @morrismanfung, @robindhillon1, and TAs for all your helpful feedback! All your feedback points were valuable and we hope we can address them all in the future. Due to the time constraint, here are a few points highlighting the feedback we have addressed:

Write 3-5 tests and document what they're testing

Each of our team members has included and documented a few tests in the tests directory. The tests were written using pytest syntax and should be easily adapted in various IDEs like vscode and Jupiter.

The commits from the team members for all the tests are in: https://github.com/UBC-MDS/customer_complaint_analyzer/pull/70, https://github.com/UBC-MDS/customer_complaint_analyzer/pull/76 and https://github.com/UBC-MDS/customer_complaint_analyzer/pull/79.

Report results of the model on the test set in the final report

We have included the test scores in addition to the validation scores in the final report. The change in the analysis file and report quarto file can be found at https://github.com/UBC-MDS/customer_complaint_analyzer/pull/73.

Generate EDA report with figure captions & no-code inputs

For the EDA part of this project, we correctly formatted the figure captions and formats. The corresponding change can be found at https://github.com/UBC-MDS/customer_complaint_analyzer/pull/78.

Update save_chart instead of saving method to improve reproducibility

One of the feedback from Milestone 2 was the makefile did not automatically run to the end. We debugged that this is because the chart saving was not correctly done to all platforms. We update this in https://github.com/UBC-MDS/customer_complaint_analyzer/pull/56 for save_chart functionality to make the make process robust at https://github.com/UBC-MDS/customer_complaint_analyzer/pull/56.

Put author and date into generate_eda.py

Some of our scripts were not documented appropriately. We add the author and date in this commit https://github.com/UBC-MDS/customer_complaint_analyzer/pull/78/commits/cd0427f83f0c45f2324233d79cbfd2a3cd3e618a.

Fix up gitignore, exclude unneeded files

We had unused files like .vscode and .Rproj files in our repository. We added into gitignore also in this pull request https://github.com/UBC-MDS/customer_complaint_analyzer/pull/78/commits/4d944def7ffa69003d6422af6db62b282668e745.

Thanks again for the amazing comments!

UBC-MDS / data-analysis-review-2022