Submission: Group 21: Motor Vehicle Collision Fatality Predictor

Submitting authors: @iamMoid @SiqiTao @gn385x @danfke

Repository: https://github.com/UBC-MDS/Collision_Prediction Report link: https://github.com/UBC-MDS/Collision_Prediction/blob/main/doc/collision_prediction_report.md Abstract/executive summary:

In this project we attempt to build a classification model using the logistic regression algorithm and data obtained from police-reported motor vehicle collisions on public roads in Canada to predict whether a motor vehicle collision would result in a fatality or not. The final model performed poorly on both the training set and the test set, returning a high recall of 0.698, but a very low precision of 0.048, resulting in a low f1-score of 0.09. The impact of the low precision can be seen in the results of the prediction of the test set, where the model incorrectly predicts fatalities around 20 times more than it correctly predicts fatalities.

The data set that was used in this project came from the National Collision Database, published by Transport Canada. The National Collision Database contains data on all of the police-reported motor vehicle collisions on public roads in Canada from 1999 to the most recent available data from 2017. We ran our analysis using the data collected from collisions that occurred in 2017. This data set contains information licensed under the Open Government Licence – Canada.

Editor: @flor14 Reviewer: PUGHAZHENDHI_GAUTHAM, Ahn_Kyle, Fairbrother_Gabriel, Wang_Joyce

[x] I agree to abide by MDS's Code of Conduct during the review process and in maintaining my package should it be accepted.

Data analysis review checklist

Reviewer: @gfairbro

Conflict of interest

[x] As the reviewer I confirm that I have no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the MDS code of conduct.

General checks

[x] Repository: Is the source code for this data analysis available? Is the repository well organized and easy to navigate?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?

Documentation

[x] Installation instructions: Is there a clearly stated list of dependencies?
[x] Example usage: Do the authors include examples of how to use the software to reproduce the data analysis?
[x] Functionality documentation: Is the core functionality of the data analysis software documented to a satisfactory level?
[ ] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Code quality

[x] Readability: Are scripts, functions, objects, etc., well named? Is it relatively easy to understand the code?
[x] Style guidelides: Does the code adhere to well known language style guides?
[x] Modularity: Is the code suitably abstracted into scripts and functions?
[ ] Tests: Are there automated tests or manual steps described so that the function of the software can be verified? Are they of sufficient quality to ensure software robsutness?

Reproducibility

[ ] Data: Is the raw data archived somewhere? Is it accessible?
[x] Computational methods: Is all the source code required for the data analysis available?
[x] Conditions: Is there a record of the necessary conditions (software dependencies) needed to reproduce the analysis? Does there exist an easy way to obtain the computational environment needed to reproduce the analysis?
[x] Automation: Can someone other than the authors easily reproduce the entire data analysis?

Analysis report

[ ] Authors: Does the report include a list of authors with their affiliations?
[x] What is the question: Do the authors clearly state the research question being asked?
[x] Importance: Do the authors clearly state the importance for this research question?
[x] Background: Do the authors provide sufficient background information so that readers can understand the report?
[x] Methods: Do the authors clearly describe and justify the methodology used in the data analysis? Do the authors communicate any assumptions or limitations of their methodologies?
[x] Results: Do the authors clearly communicate their findings through writing, tables and figures?
[x] Conclusions: Are the conclusions presented by the authors correct?
[ ] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)? I dont know enough about DOI to answer the above question
[x] Writing quality: Is the writing of good quality, concise, engaging?

Estimated hours spent reviewing:

1.5H

Review Comments:

Please provide more detailed feedback here on what was done particularly well, and what could be improved. It is especially important to elaborate on items that you were not able to check off in the list above.

Particularly Well:

I thought the data set selected was interesting and more ambitious. Many groups stuck to the UCI, so branching out and finding something different is a real plus. I also liked that they used a technique we haven't covered in class (Undersampling).
Their report was clear and easy to read and understand.
Their code is also quite well written, neat and easy to follow.

Improvements:

The inclusion of an environment file is a nice touch, but using the --from-history flag when generating this would allow for people to use it in different OS environments. As far as i can tell this is from a windows environment. directions in Usage on how to install would be helpful as well.
It appears the source data is no longer available (at least temporarily). Adding it to the repo or providing a source and making note of that would be helpful.
Not having a data directory skeleton means the scripts failed, so including them (even empty) would be a good idea. I see that the script should try to make them but it didnt work for me until i created the path. maybe because it is 2 levels?
The environment is missing something for the EDA in windows. I know this because our project is too! in order to output PNG in windows from altair you need to run the following, and it doesnt seem to get captured properly when you export a environment.yaml file: npm install -g vega vega-cli vega-lite canvas (see lecture 2 from 531 for details - scroll down to the warning): https://pages.github.ubc.ca/mds-2021-22/DSCI_531_viz-1_students/lectures/2-data-types_graphical-marks_visual-encondings.html#global-development-data

I had this error and can confirm it fixed it for your project

Attribution

This was derived from the JOSE review checklist and the ROpenSci review checklist.

Data analysis review checklist

Reviewer: @gauthampughaz

Conflict of interest

[X] As the reviewer I confirm that I have no conflicts of interest for me to review this work.

Code of Conduct

[X] I confirm that I read and will adhere to the MDS code of conduct.

General checks

[X] Repository: Is the source code for this data analysis available? Is the repository well organized and easy to navigate?
[X] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?

Documentation

[X] Installation instructions: Is there a clearly stated list of dependencies?
[X] Example usage: Do the authors include examples of how to use the software to reproduce the data analysis?
[X] Functionality documentation: Is the core functionality of the data analysis software documented to a satisfactory level?
[X] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Code quality

[X] Readability: Are scripts, functions, objects, etc., well named? Is it relatively easy to understand the code?
[X] Style guidelides: Does the code adhere to well known language style guides?
[X] Modularity: Is the code suitably abstracted into scripts and functions?
[ ] Tests: Are there automated tests or manual steps described so that the function of the software can be verified? Are they of sufficient quality to ensure software robsutness?

Reproducibility

[ ] Data: Is the raw data archived somewhere? Is it accessible?
[X] Computational methods: Is all the source code required for the data analysis available?
[X] Conditions: Is there a record of the necessary conditions (software dependencies) needed to reproduce the analysis? Does there exist an easy way to obtain the computational environment needed to reproduce the analysis?
[ ] Automation: Can someone other than the authors easily reproduce the entire data analysis?

Analysis report

[X] Authors: Does the report include a list of authors with their affiliations?
[X] What is the question: Do the authors clearly state the research question being asked?
[X] Importance: Do the authors clearly state the importance for this research question?
[X] Background: Do the authors provide sufficient background information so that readers can understand the report?
[X] Methods: Do the authors clearly describe and justify the methodology used in the data analysis? Do the authors communicate any assumptions or limitations of their methodologies?
[X] Results: Do the authors clearly communicate their findings through writing, tables and figures?
[X] Conclusions: Are the conclusions presented by the authors correct?
[X] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
[X] Writing quality: Is the writing of good quality, concise, engaging?

Estimated hours spent reviewing: 1.5 hours

Review Comments:

Positives:

The data chosen is good and an unique one. The model developed along with the future improvements mentioned in the project report could potentially be a life-saving solution.
The results achieved and improvements to be made are transparently described.
Efforts towards the feature selection such as using RFECV to reduce the total number of features and addressing the class imbalance using undersampling are commendable.

Improvements:

The modularity of the code can be improved by breaking the large code chunks into functions to promote reusability and readability. I checked the box for modularity above because the project has separate files for different functionalities but I would recommend adding more functions.
I would recommend adding a few more plots to describe different aspects of the data if this recommendation makes sense to your problem statement.
Adding a few simple assert statements inside the scripts will ease debugging during development.

Attribution

This was derived from the JOSE review checklist and the ROpenSci review checklist.

Data analysis review checklist

Reviewer: @jo4356

Conflict of interest

[x] As the reviewer I confirm that I have no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the MDS code of conduct.

General checks

[x] Repository: Is the source code for this data analysis available? Is the repository well organized and easy to navigate?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?

Documentation

[x] Installation instructions: Is there a clearly stated list of dependencies?
[x] Example usage: Do the authors include examples of how to use the software to reproduce the data analysis?
[x] Functionality documentation: Is the core functionality of the data analysis software documented to a satisfactory level?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Code quality

[x] Readability: Are scripts, functions, objects, etc., well named? Is it relatively easy to understand the code?
[x] Style guidelides: Does the code adhere to well known language style guides?
[x] Modularity: Is the code suitably abstracted into scripts and functions?
[ ] Tests: Are there automated tests or manual steps described so that the function of the software can be verified? Are they of sufficient quality to ensure software robsutness?

Reproducibility

[ ] Data: Is the raw data archived somewhere? Is it accessible?
[x] Computational methods: Is all the source code required for the data analysis available?
[x] Conditions: Is there a record of the necessary conditions (software dependencies) needed to reproduce the analysis? Does there exist an easy way to obtain the computational environment needed to reproduce the analysis?
[ ] Automation: Can someone other than the authors easily reproduce the entire data analysis?

Analysis report

[x] Authors: Does the report include a list of authors with their affiliations?
[x] What is the question: Do the authors clearly state the research question being asked?
[x] Importance: Do the authors clearly state the importance for this research question?
[x] Background: Do the authors provide sufficient background information so that readers can understand the report?
[x] Methods: Do the authors clearly describe and justify the methodology used in the data analysis? Do the authors communicate any assumptions or limitations of their methodologies?
[x] Results: Do the authors clearly communicate their findings through writing, tables and figures?
[x] Conclusions: Are the conclusions presented by the authors correct?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
[x] Writing quality: Is the writing of good quality, concise, engaging?

Estimated hours spent reviewing: 1.5 hours

Review Comments:

I wasn't able to download the data. Seems like the data is removed?
The EDA is very detailed and nicely explained. You clearly explained the thought process of going from one part of the EDA to the next.
The eda.py script has everything inside if __name__ == "__main__":. It might be a bit more readable and consistent with a main function, since all of your other scripts have a main function.
I agree with the previous reviewer that the code can be broken down into smaller functions, maybe based on different steps of the analysis.
Adding to the previous point, having doctrings to the smaller functions, and moving all the comments inside the code to the docstrings, would make the code much more readable.
The final report is well organized, and I like that you described how you can further improve on your model.

Attribution

This was derived from the JOSE review checklist and the ROpenSci review checklist.

Data analysis review checklist

Reviewer: @AraiYuno

Conflict of interest

[x] As the reviewer I confirm that I have no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the MDS code of conduct.

General checks

[x] Repository: Is the source code for this data analysis available? Is the repository well organized and easy to navigate?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?

Documentation

[x] Installation instructions: Is there a clearly stated list of dependencies?
[x] Example usage: Do the authors include examples of how to use the software to reproduce the data analysis?
[x] Functionality documentation: Is the core functionality of the data analysis software documented to a satisfactory level?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Code quality

[x] Readability: Are scripts, functions, objects, etc., well named? Is it relatively easy to understand the code?
[x] Style guidelides: Does the code adhere to well known language style guides?
[x] Modularity: Is the code suitably abstracted into scripts and functions?
[ ] Tests: Are there automated tests or manual steps described so that the function of the software can be verified? Are they of sufficient quality to ensure software robsutness?

Reproducibility

[ ] Data: Is the raw data archived somewhere? Is it accessible?
[x] Computational methods: Is all the source code required for the data analysis available?
[x] Conditions: Is there a record of the necessary conditions (software dependencies) needed to reproduce the analysis? Does there exist an easy way to obtain the computational environment needed to reproduce the analysis?
[ ] Automation: Can someone other than the authors easily reproduce the entire data analysis?

Analysis report

[x] Authors: Does the report include a list of authors with their affiliations?
[x] What is the question: Do the authors clearly state the research question being asked?
[x] Importance: Do the authors clearly state the importance for this research question?
[x] Background: Do the authors provide sufficient background information so that readers can understand the report?
[x] Methods: Do the authors clearly describe and justify the methodology used in the data analysis? Do the authors communicate any assumptions or limitations of their methodologies?
[x] Results: Do the authors clearly communicate their findings through writing, tables and figures?
[x] Conclusions: Are the conclusions presented by the authors correct?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
[x] Writing quality: Is the writing of good quality, concise, engaging?

Estimated hours spent reviewing: 1.5 hours

Review Comments:

Positives

I do like the fact that all the scripts were written in python, or python code within jupyter notebook. This reduces the level of complexity during the process of the initial project setup.
Although there seem to be some lengthy chunks of codes in different scripts, there are simple but descriptive comments that helped me understand what the code is intended to do.
In the final report, how further improvements could be made are described. I think it is an excellent way of building your analysis in an iterative approach.

Improvements

There seems to be a "figures" folder inside results folder, which I assume that it is meant to include all the png files. However, I see that some .png files (figures) are in results folder. Just to check out the figures, I directly went into the rusults/figures folder and missed the ones in results folder. In fact, the figures inside rusults/figures folder is older than the ones in results folder. I am not sure which ones are the valid ones. I could only figure them out by looking at the code.
Some basic tests could be added in order to ensure that each script behaves as expected. For example, I see that train.csv and test.csv are supposed to be saved as outputs of clean_split_data.py script. Using assert to check whether the file actually exists would prevent someone to accidentally remove code that generates output files.
I am not sure if this was intended, but .gitignore seems to be extremely long. It consists of many wildcards and file types that I don't see a point in including. Perhaps, this file was grabbed from somewhere else. In that case, a reference should be provided just like any other files. I think this .gitignore file's source is perhaps here! https://github.com/github/gitignore/blob/main/Python.gitignore.

Overall, I would give 9.8 out of 10!! Near perfect repo! Great work team 21!

Attribution

This was derived from the JOSE review checklist and the ROpenSci review checklist.

Thank you everyone for the great feedback! We've addressed many of the points provided, including the following:

In regards to comment number 2 of @gfairbro's review (an issue which was also mentioned in subsequent reviews)we addressed this by linking to another URL containing the same data as per Tiffany’s directions: 503c540.
In regards to comment number 4 of @gfairbro's review we added a line in the usage section of our README where it explains that windows users may have to run an additional command prior to running the make file in order to properly render PNG's: 630613e

We also added it to our Dockerfile so that it runs properly: 4d287a5

In regards to comment number 3 of @gauthampughaz's review we added assert statements to all of the scripts that did not already have some form of testing: c1731a3 677fad9 45414ce 3465f96
In regards to comment number 3 of @jo4356's review we addressed this here: 75d1262

UBC-MDS / data-analysis-review-2021