single geog: percentage point difference not all always graying

[EricaMaurer]

https://nyc-factfinder.planninglabs.nyc/profile/735/social?charts=false&comparator=SI07&reliability=true

[allthesignals]

Possibly related to #548

The percentage point estimate and moe columns should be gray (insignificant ) and they'ren ot

[allthesignals]

So this is related to percentage point difference significance, but your PR added code for the significance calculation which is different. Gonna look at the SQL and create a test

[EricaMaurer]

Ok cool. #528 was for change, this issue is difference

[allthesignals]

I think I meant this PR https://github.com/NYCPlanning/labs-factfinder-api/pull/31/files

[allthesignals]

What are the inputs for this significance calculations? Trying to double check formulas.

[EricaMaurer]

percentage point est and MOE

[EricaMaurer]

https://github.com/NYCPlanning/labs-factfinder-api/pull/31/files was for special variables

[allthesignals]

Alright, I have a failing test for this.

The math for this is as follows:

WHEN ABS(
  SQRT(
    POWER(coalesce(percent_m, 0) / 1.645, 2) 
      + POWER(coalesce(comparison_percent_m, 0) / 1.645, 2)
  ) * 1.645 ) > ABS(comparison_percent - percent)
THEN 
  false
ELSE true

I think what's happening is that comparison_person is null, so it can't perform math on an unknown value. That pollutes the entire calculation because it can't do boolean logic on a null value.

This is a tricky thing in some languages - JavaScript is fast and loose with nulls and embraces the "truthiness" concept where null is considered "falsey" and therefore evaluatable in boolean logic.

In our SQL script this seems like a bigger issue of how to deal with nulls and 0s mid-calculation - sometimes we should assume null is 0 for additional calculations mid-stream, but those values shouldn't be null when they are served out to the app.

PR incoming.