2016 Datasets

[anderoy]

Looks like we will have the following datasets to upload:

• [x] student_count_2016
• [x] esd_hs_2016
• [x] esd_k8_2015_r1
• [x] k12_supplemental_2016
• [ ] new_school_2016

I will upload each file here with any important notes as I receive them in their final form. The analysis team keeps punting the final data down the road, but the supplemental data file will be ready next week.

[anderoy]

It looks like GitHub isn't accepting csv files, so I am adding the excel file.

This student_count file is the same structure as the former MEAP_20xx files EXCEPT that it has fewer fields. It is basically just deleting the columns to the right of all the enrollment data.

2014-15 Student Count with Detroit City Totals.xlsx

[anderoy]

@bnchdrff still pressing to get all of these datasets. I am waiting to make sure there won't be any changes to them.

Also (perhaps another issue) on Friday I will be working on documenting all of the live site changes I made so they can be merged in with the rest of the updates.

Thanks for being patient.

[anderoy]

The K8 and HS files are done. There are new fields at the end of each but the structure/fieldnames/column order should be the same as last year.

I notices that the csv export was not included in the "Views" last year. It would be nice if that could be included when building the schema so that it can be easily available for our data team should they need to check the underlying data that is showing up on the Scorecard.

Here is the location of the files. I will put everything else there as well.

https://drive.google.com/open?id=0B9LDi9SHMwnrTHVmeHdTa3I2TnM

[bnchdrff]

Good to hear, but that goodrive folder is empty :grin:

[anderoy]

Hmm, I see two CSV files in it.

https://drive.google.com/folderview?id=0B9LDi9SHMwnrTHVmeHdTa3I2TnM

[bnchdrff]

When importing, add esd_schid col just in case scores need to be overridden.

[anderoy]

Is this ready for Alec to use on the test site?

[bnchdrff]

Not quite yet -- i've got hs & k8 done but am working on the rest! Will get back to it this evening.

[anderoy]

Is it at all possible to push the hs and k8 files? Those are the ones that we need to start reviewing asap on the test site to make sure we are displaying everything properly.

[bnchdrff]

Yup, pushed: https://50-proximity-3mhev6qb5ihtc.us.platform.sh/api/1.0/schools.json?flatten_fields=true&includes=esd_hs_2016,esd_k8_2016

[anderoy]

Thanks!

This doesn't need to happen right away (take a look tonight), but we need to make sure that all fields accept NULL values. Some of the frac_prof and other "percentage" decimal fields seem to be replacing NULL with 0.000000. This causes a major problem on the front end, which depends on the NULL values to turn off displaying a value for a given field.

[bnchdrff]

Can you cite examples? I've done a spot check & have found plenty of null values in many fields at https://50-proximity-3mhev6qb5ihtc.us.platform.sh/api/1.0/schools.json?flatten_fields=true&includes=esd_hs_2016,esd_k8_2016

On Mon, Mar 14, 2016 at 10:11 AM, Eric Anderson notifications@github.com wrote:

Thanks!

This doesn't need to happen right away (take a look tonight), but we need to make sure that all fields accept NULL values. Some of the frac_prof and other "percentage" decimal fields seem to be replacing NULL with 0.000000. This causes a major problem on the front end, which depends on the NULL values to turn off displaying a value for a given field.

— Reply to this email directly or view it on GitHub https://github.com/ESDet/esdportal_profile/issues/52#issuecomment-196324160 .

[anderoy]

Ok, you are correct, I must have been experiencing shellshock at the alarming number of 0 values that were highlighted by the multiple decimal values. All good, sorry for the alarm.

Boggs School is an example that should and does have NULL values in many of the percentage fields.

{"type":"esd_k8_2016s","id":"2073","bcode":"2073","schoolname":"The James and Grace Lee Boggs School","total_ltrgrade":null,"total_pts":null,"prof_cat_points":"3.16666667","growth_cat_points":null,"net5e_cat_pts":"4.00000000","pr2_math_pts":"2.00000000","frac_profmath":"0.19047620","pr2_sci_pts":null,"frac_profscience":null,"pr2_ss_pts":null,"frac_profsocstud":null,"pr2_ela_pts":"5.00000000","frac_profela":"0.52941180","pts_progress_math":null,"mean_pctl_math":null,"pts_progress_read":null,"mean_pctl_reading":null,"cat5e":"4.00000000","mstep_mathpts":"2.00000000","mstep_mathprof":"0.13500000","mstep_elapts":"4.00000000","mstep_elaprof":"0.34200000","mstep_scipts":"3.00000000","mstep_sciprof":"0.09100000","mstep_socpts":"3.00000000","mstep_socprof":"0.15400000","scatter_elaprof":null,"scatter_elagrowth":null,"scatter_mathprof":null,"scatter_mathgrowth":null,"esd_schid":"\r"}

[anderoy]

@bnchdrff any news on this front? Want to get this piece and one more issue locked up and pushed to the live site as soon as possible so that we can check it on the staging site with the live portal data before he takes off for Hawaii.

The other issue is the addition of one field to both the K12 School and Early Childhood Program taxonomy.

• [x] Boolean field (yes/no) called "Enroll Detroit Participant" field_ed_participant default to no or off value.

Can you do that with your push or should I add it to the live site?

[anderoy]

Adding a second field to the K12 school taxonomy.

• [x] Boolean field (yes/no) called "2016 Recommended School" field_2016_recommended default to no or off value.

[bnchdrff]

Worked on it last night & will continue tonight.

Just to make sure, the 2016 supp dataset is missing a couple fields that were present last year:

     'full_attendance_eqvl_all' => array(
        'type' => 'numeric',
        'size' => 'normal',
        'precision' => '6',
        'scale' => '2',
        'unsigned' => TRUE,
        'not null' => FALSE,
        'description' => 'Full Attendance Equivalent N',
      ),
      'full_attendance_eqvl_ca' => array(
        'type' => 'numeric',
        'size' => 'normal',
        'precision' => '6',
        'scale' => '2',
        'unsigned' => TRUE,
        'not null' => FALSE,
        'description' => 'Full Attendance Equivalent of Chronic Abs',
      ),

[anderoy]

Yep, we removed those and added a few more. Good eye. Thank you, thank you.

[bnchdrff]

count_2016 & k12_supplemental_2016 are on the dev server!

[bnchdrff]

(i called it count_2016 instead of student_count_2016, hope that's ok)

[anderoy]

Sent along updated k8 and hs files to be uploaded to production. Then we can close this issue and wrap up the spring 2016 round of updates (barring some emergency)!!