Submission: Group 16: Beijing Air Quality Analysis

Submitting authors: Jacqueline Chong, Junrong Zhu, Macy Chan, Vadim Taskaev

Repository:https://github.com/UBC-MDS/DSCI_522_Beijing_Air_Quality Report link:https://ubc-mds.github.io/DSCI_522_Beijing_Air_Quality/ Abstract/executive summary: This analysis project aims to answer whether the levels of PM2.5 air pollution in Beijing, China has improved between 2013 and 2017. To do so, we performed a difference in medians hypothesis test between two intervals, time_A (March 2013 - February 2015) and time_B (March 2015 - February 2017), and concluded that no statistically significant decrease in PM2.5 particulate measurements can be detected.

Editor: @flor14 Reviewer: Michelle Wang, Siqi Tao, Hu Jiwei and Wang Shi Yan

[x] I agree to abide by MDS's Code of Conduct during the review process and in maintaining my package should it be accepted.

Reviewer: Michelle Wang @michelle-wms

Conflict of interest

[x] As the reviewer I confirm that I have no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the MDS code of conduct.

General checks

[x] Repository: Is the source code for this data analysis available? Is the repository well organized and easy to navigate?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?

Documentation

[x] Installation instructions: Is there a clearly stated list of dependencies?
[x] Example usage: Do the authors include examples of how to use the software to reproduce the data analysis?
[x] Functionality documentation: Is the core functionality of the data analysis software documented to a satisfactory level?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Code quality

[x] Readability: Are scripts, functions, objects, etc., well named? Is it relatively easy to understand the code?
[x] Style guidelides: Does the code adhere to well known language style guides?
[x] Modularity: Is the code suitably abstracted into scripts and functions?
[x] Tests: Are there automated tests or manual steps described so that the function of the software can be verified? Are they of sufficient quality to ensure software robustness?

Reproducibility

[x] Data: Is the raw data archived somewhere? Is it accessible?
[x] Computational methods: Is all the source code required for the data analysis available?
[x] Conditions: Is there a record of the necessary conditions (software dependencies) needed to reproduce the analysis? Does there exist an easy way to obtain the computational environment needed to reproduce the analysis?
[x] Automation: Can someone other than the authors easily reproduce the entire data analysis?

Analysis report

[x] Authors: Does the report include a list of authors with their affiliations?
[x] What is the question: Do the authors clearly state the research question being asked?
[x] Importance: Do the authors clearly state the importance for this research question?
[x] Background: Do the authors provide sufficient background information so that readers can understand the report?
[x] Methods: Do the authors clearly describe and justify the methodology used in the data analysis? Do the authors communicate any assumptions or limitations of their methodologies?
[ ] Results: Do the authors clearly communicate their findings through writing, tables and figures?
[x] Conclusions: Are the conclusions presented by the authors correct?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
[x] Writing quality: Is the writing of good quality, concise, engaging?

Estimated hours spent reviewing: 1 hour

Review Comments:

This is a really interesting topic and the report was overall well written and analyzed to give some great (albeit slightly disappointing, one would've thought their quality will be better by now) conclusions about Beijing's air pollution improvement over time. Some detailed comments can be found below.

Codes are well-organised and easy to read, and functions were written with proper roxygen-style documentation.
Well-explained background and justification to give readers context of what the significance of the test is, by highlighting the harmfulness of PM2.5 out of all indicators present in the dataset, which is the focus of study.
Tests were written to ensure proper inputs for functions, and that functions work as expected (for example, calculation of medians).
EDA report did not display the respective plots at the bottom, making it a little challenging to follow along with the conclusions drawn. Although upon clicking Final Report link, they were correctly displayed.
More explanations can be made as to why the 2 specific time periods were chosen for hypothesis testing. You could maybe also look at the year which certain policies were implemented and compare using those time periods instead for hypothesis testing, which may be more meaningful to see correlation of impact of policies on change in air pollution.
The raw data files showed aggregation of data of several different areas of Beijing, and the final analysis seems to be based upon all these sites. Perhaps some further EDA could be done on the air pollution levels of these sites and see if they vary alot, and if they do, then zoom in further on the sites that showed more significant levels of air pollution.
The boxplot and violin plots are showing basically the same things, so just keeping one would suffice (perhaps violin one, which shows distribution as well).
The question that was proposed to be answered was "whether air pollution has shown improvement" but the conclusion was more towards "China has much to grow in terms of economic development" -- further explanation can be made as to why this is the case, and what link it has with the initial question of air pollution improvement.

Attribution

This was derived from the JOSE review checklist and the ROpenSci review checklist.

Hi @michelle-wms. Thank you for your comments.

We looked through the comments, and agree that we need to tighten up our conclusion to link it to our initial question before exploring the implications of our results.
We also take your suggestion that we did not clearly define why we chose to divide the data into 2013-2015, and 2015-2017. Our reasoning was that we wanted a balanced data set. Moreover, our main goal was to see if there was a change in air pollution level, and not necessarily how it has changed based on the government policy or international regime (regime not in the sense of an authoritarian regime, but more of how the world views this issue collectively) changes.
For our EDA, our group wanted to focus on the question at hand and look at the overall change in Beijing. As such, we chose to be more targetted in our approach, instead of exploring all the possible questions.
As for the question of the boxplot and violin plot. They are actually very different plots. The boxplot was from our EDA, while the violin plot is plotted after we have completed our hypothesis testing. It is meant to help us visualise the change, and explain why we could not reject our null hypothesis. However, if you have suggestions on how we can improve our violin plot for our hypothesis test, we would really appreciate it!

Thanks again Michelle for your prompt response!

Cheers!

Data analysis review checklist

Reviewer: Shi Yan Wang @sy25wang

Conflict of interest

[x] As the reviewer I confirm that I have no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the MDS code of conduct.

General checks

[x] Repository: Is the source code for this data analysis available? Is the repository well organized and easy to navigate?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?

Documentation

[x] Installation instructions: Is there a clearly stated list of dependencies?
[x] Example usage: Do the authors include examples of how to use the software to reproduce the data analysis?
[x] Functionality documentation: Is the core functionality of the data analysis software documented to a satisfactory level?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Code quality

[x] Readability: Are scripts, functions, objects, etc., well named? Is it relatively easy to understand the code?
[x] Style guidelides: Does the code adhere to well known language style guides?
[x] Modularity: Is the code suitably abstracted into scripts and functions?
[x] Tests: Are there automated tests or manual steps described so that the function of the software can be verified? Are they of sufficient quality to ensure software robsutness?

Reproducibility

[x] Data: Is the raw data archived somewhere? Is it accessible?
[x] Computational methods: Is all the source code required for the data analysis available?
[x] Conditions: Is there a record of the necessary conditions (software dependencies) needed to reproduce the analysis? Does there exist an easy way to obtain the computational environment needed to reproduce the analysis?
[x] Automation: Can someone other than the authors easily reproduce the entire data analysis?

Analysis report

[x] Authors: Does the report include a list of authors with their affiliations?
[x] What is the question: Do the authors clearly state the research question being asked?
[x] Importance: Do the authors clearly state the importance for this research question?
[x] Background: Do the authors provide sufficient background information so that readers can understand the report?
[x] Methods: Do the authors clearly describe and justify the methodology used in the data analysis? Do the authors communicate any assumptions or limitations of their methodologies?
[ ] Results: Do the authors clearly communicate their findings through writing, tables and figures?
[ ] Conclusions: Are the conclusions presented by the authors correct?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
[x] Writing quality: Is the writing of good quality, concise, engaging?

Estimated hours spent reviewing: One and a half hours

Review Comments:

Overall, I think the project is well structured with clearly stated research question. The README.md provides a good background and clear instructions on the project. All files are put in a reasonable place, and they are identifiable. I see that the team also numbers files sequentially in the doc folder, which is a great practice to follow. The final report is easy to follow, supported with visualizations.

Some potential improvements may be as follows:

In the EDA analysis, the axis label for figure 2 can be adjusted from 1 to 12 to reflect month
The EDA analysis can be more concise, and the position of the figures can be rearranged. For example, the last paragraph under "Study the data" section can be dropped. Also, the team can consider moving the last two paragraphs ('We created Table 3 and Table 4...Looking at the statistic values on Table 3...') right after Table 3 and Table 4 for better reference.
It is interesting to see how the team tries to relate the question with Kuznets curve. However, it does not help the research question on if Beijing has shown any improvements on PM2.5 measurements.
In the final report, both boxplot and histogram PM2.5 distribution are included in one figure. It seems a duplicate of the information. Either one of them should be sufficient.
The project aims to investigate long-term improvement, however, it is questionable that the project only divides the data into two time-session of 2-year each. It is also questionable why the median is chosen as the estimator for long-term performance evaluation. At this stage of the project, I don't think it is reasonable to test other estimators or split the data into more groups, but it might still be helpful to address somewhere that the team is aware of the concerns.
The conclusion in the final report refers back to the industrialization and the economy performance of China, but the focus for the project should be on PM 2.5 measurements in Beijing. It is also questionable that the PM 2.5 performance of Beijing can represents the overall situation of China. The team may consider to revise the conclusion of the report.

Attribution

This was derived from the JOSE review checklist and the ROpenSci review checklist.

Data analysis review checklist

Reviewer: hjw0703

Conflict of interest

[x] As the reviewer I confirm that I have no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the MDS code of conduct.

General checks

[x] Repository: Is the source code for this data analysis available? Is the repository well organized and easy to navigate?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?

Documentation

[x] Installation instructions: Is there a clearly stated list of dependencies?
[x] Example usage: Do the authors include examples of how to use the software to reproduce the data analysis?
[x] Functionality documentation: Is the core functionality of the data analysis software documented to a satisfactory level?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Code quality

[x] Readability: Are scripts, functions, objects, etc., well named? Is it relatively easy to understand the code?
[x] Style guidelides: Does the code adhere to well known language style guides?
[x] Modularity: Is the code suitably abstracted into scripts and functions?
[x] Tests: Are there automated tests or manual steps described so that the function of the software can be verified? Are they of sufficient quality to ensure software robsutness?

Reproducibility

[x] Data: Is the raw data archived somewhere? Is it accessible?
[x] Computational methods: Is all the source code required for the data analysis available?
[x] Conditions: Is there a record of the necessary conditions (software dependencies) needed to reproduce the analysis? Does there exist an easy way to obtain the computational environment needed to reproduce the analysis?
[x] Automation: Can someone other than the authors easily reproduce the entire data analysis?

Analysis report

[x] Authors: Does the report include a list of authors with their affiliations?
[x] What is the question: Do the authors clearly state the research question being asked?
[x] Importance: Do the authors clearly state the importance for this research question?
[x] Background: Do the authors provide sufficient background information so that readers can understand the report?
[ ] Methods: Do the authors clearly describe and justify the methodology used in the data analysis? Do the authors communicate any assumptions or limitations of their methodologies?
[x] Results: Do the authors clearly communicate their findings through writing, tables and figures?
[ ] Conclusions: Are the conclusions presented by the authors correct?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
[x] Writing quality: Is the writing of good quality, concise, engaging?

Estimated hours spent reviewing: 1 hour

Review Comments:

Overall, I think it is a nice report with a good structure and convincing content. The only things that I can think of that might improve the quality are the following:

The question here is: "Does PM2.5 measurement in Beijing, China collected between 2013 and 2017 show any sign of improvement?", why do you divide the interval into two groups with 2 years interval for each group rather divide it into four groups with 1 year interval for each group? Since the mean of PM2.5 dropped in 2016 but increased in 2017 and on average it is the same as in 2013-2015 as shown in Table 4 in the EDA part. I think you could probably just compare year 2016 and 2017.
As it is shown the data is skewed in distribution, I suggest using logarithmic scale when doing the hypothesis test because t-statistic might not working well in severe skewed data.
Instead of mentioning the Environmental Keznets Curve Hypothesis economic model in the conclusion, I think you could say something about how you could improve the inference in the future, like what if you could have more recent data on Beijing's air quality.
PM2.5 is not the only thing that matters to the air quality, maybe if you have time, you could look into other features in the data set like PM10.
Maybe you could illustrate how you get the p-value in the results and discussion part, is it from student t-statistic or from bootstrap sampling distribution and give the exact values of these statistics, like is the p-value equals 1 or 0.99.
If you choose median as the test statistic, you could include median in the EDA part as opposed to mean.

Attribution

This was derived from the JOSE review checklist and the ROpenSci review checklist.

Data analysis review checklist

Reviewer: SiqiTao

Conflict of interest

[x] As the reviewer I confirm that I have no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the MDS code of conduct.

General checks

[x] Repository: Is the source code for this data analysis available? Is the repository well organized and easy to navigate?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?

Documentation

[x] Installation instructions: Is there a clearly stated list of dependencies?
[x] Example usage: Do the authors include examples of how to use the software to reproduce the data analysis?
[x] Functionality documentation: Is the core functionality of the data analysis software documented to a satisfactory level?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Code quality

[x] Readability: Are scripts, functions, objects, etc., well named? Is it relatively easy to understand the code?
[x] Style guidelides: Does the code adhere to well known language style guides?
[x] Modularity: Is the code suitably abstracted into scripts and functions?
[x] Tests: Are there automated tests or manual steps described so that the function of the software can be verified? Are they of sufficient quality to ensure software robsutness?

Reproducibility

[x] Data: Is the raw data archived somewhere? Is it accessible?
[x] Computational methods: Is all the source code required for the data analysis available?
[x] Conditions: Is there a record of the necessary conditions (software dependencies) needed to reproduce the analysis? Does there exist an easy way to obtain the computational environment needed to reproduce the analysis?
[x] Automation: Can someone other than the authors easily reproduce the entire data analysis?

Analysis report

[x] Authors: Does the report include a list of authors with their affiliations?
[x] What is the question: Do the authors clearly state the research question being asked?
[x] Importance: Do the authors clearly state the importance for this research question?
[x] Background: Do the authors provide sufficient background information so that readers can understand the report?
[x] Methods: Do the authors clearly describe and justify the methodology used in the data analysis? Do the authors communicate any assumptions or limitations of their methodologies?
[x] Results: Do the authors clearly communicate their findings through writing, tables and figures?
[x] Conclusions: Are the conclusions presented by the authors correct?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
[x] Writing quality: Is the writing of good quality, concise, engaging?

Estimated hours spent reviewing: 1 hour

Review Comments:

This is a very meaningful topic! The content is well structured and codes are organized. Some of my thoughts and suggestions over this project are listed below:

In the methodology section of the final report, the x-label of boxplot in Figure 2 could be better specified. Also, there is a grey block to the right of the figure that I think should be removed.
When talking about air quality, it would be more convincing if we also consider other pollutants other than PM2.5, as PM2.5 not the only factor that affects the air quality.
Since the air quality fluctuates across different months of a year and this can be an unseen periodical variation of the data, I would suggest looking into a certain range of time of each year (for example, October to February) instead of all 12 months.
The length of time interval between two time frames may influence the result as well (the two frames used in the project are consecutive, which looks a bit too close to me), may be you can try time frames with a bigger interval(eg: 2013-2014 and 2016-2017), if sample size permits, and see if it makes a difference.
The reason why median is chosen could be better explained (to my understanding, median was chosen over mean because the data is heavily skewed), and it should also be included in the EDA.

Attribution

This was derived from the JOSE review checklist and the ROpenSci review checklist.

Thank you michelle-wms, sy25wang, hjw0703, and SiqiTao for your peer review feedback.

We have made changes in relation to the following four comments:

Based on michelle-wms's last comment, we made changes to the Results and Discussion section of the Final Report to ensure the scope of the implications of our findings remains in line with out project analysis objective. More specifically, we re-stated the implications of our analysis as relating China's economic development to its environmental policies, rather than directly relating it to the Kuznets Curve, which we mention briefly throughout our project. (commit: 35b4e95)
Based on SiqiTao's first comment, we made changes to the formatting of Figure 2 in the Methodology section of the Final Report. (commit: 1628a64)
Based on sy25wang's first comment, we adjusted the formatting of Figure 2 of the EDA document to replace the x-axis labels numeric labels with their month string label equivalent, resulting in improved readability. (commit: c5e3bfb)
Finally, based on hjw0703's fifth comment, we modified rounding of the p-value result as stated in the Results and Discussion section of the Final Report. This is a subtle change, where we increased the rounding of the p-value to 7 decimal places, but a meaningful one for improved interpretability, as the previous rounding to fewer decimals (to a value of 1.0) raised unnecessary suspicion. (commit: 0bdf7a0)

UBC-MDS / data-analysis-review-2021