Task: Examining correlation between existing CEJST thresholds

[giannatparisi]

Describe the task

So far I have made a correlation matrix for the CEJST thresholds and have begun the process of collapsing thresholds by category.

• correlation matrix: *analysis: The following analysis operates under the assumption that a Pearson coefficient (R) value above .7 is strongly correlated. However, when observing the relationships in the correlation matrix, the highest R value is ~.568 (showing the correlation between medianincome and poverty). While this may indicate overall low correlation values, there are several factors to consider when deciding the extent to which given values are correlated. One of those factors is the variance of the dataset. The Pearson coefficient value operates under the assumption that the dataset contains continuous values with normal variance. In the CEJST dataset, however, the values are discontinuous (with 0 representing FALSE values and 1 representing TRUE values), and at least 90% of the values in this matrix are 0. This has led to low correlation coefficients regardless of the 1-1 relationship between values. With this in mind, it might be useful to take this analysis with a grain of salt, as conducting a Jaccard similarlity test (which measures the similarities between binaries) will allow us to draw more concrete conclusions. Instead, we can use this as a way to connect assess how certain variables compare to others, and the cohesiveness of their correlations throughout the matrix.

The NRI dataset – agloss, poploss, buildloss – seems pretty disjoint from the other indicators compared to what it “picks up”. This is true, but to a lesser extent, for the pollution indicators, like RMP sites. They don’t seem to be strongly correlated outside of other environmental indicators. This is not the case for health, T&WD, etc. indicators

Additionally, there are no mildly or strongly significant negative correlations. This is as expected -- we are working with environmental and economic indicators here that are proven to be significantly positively correlated.

Acceptance Criteria

• I'm stuck on how to proceed with the next step in this task. screenshot of my code so far here:
• assist with next steps/whether my code is even correct/whether I'm still misunderstanding the task.

[emma-nechamkin]

thanks so much, Gianna! I'll take a look at this tomorrow.

[emma-nechamkin]

Gianna, really nice work to start!

For the next task, I think you want to construct a NEW field rather than mapping the max. Using dplyr, I think you will want something like:

df %>% 
    rowise() %>% # you could also groupby census tract, which makes less intuitive sense but is faster
    mutate(max_var_in_this_group = max(var1, var2, var3,..., var_n)

You might have to do something for rows with all nan(or you can drop them).

This will produce a new variable that is the maximum value across all included variables.

The code you wrote above does not create a new variable.

[adwel81]

This is great! Curious if it's possible to include the two "screening" indicators in here too - Low Income (% below 200% FPL) and Higher Educational Enrollment. Also, is your analysis at the national or state level? I wonder if correlation results may be different at the state level. I was looking at the relationships between Educational Enrollment, Educational Attainment and Age for New York State this morning and found interesting relationships between the age distribution and educational enrollment, and between educational enrollment vs. High School attainment, posted in https://github.com/usds/justice40-tool/issues/1509 Seems super valuable to keep looking - Thanks!