july 2016 db updates

[bnchdrff]

replace data in k8 & hs tables & create new table for 2016 5e

for 5e:

So I realized that the U Chicago group had finally betrayed their promise to me to keep the format consistent year over year.

So in lieu of the full dataset, we have joined the 2015 and 2016 data and only included the minimal required fields for the Scorecard in a way that should create the least amount of difficulty for everyone.

This should be the same fields and field names as fiveessentials_2015 with the following exceptions:

• There is a new field Year at the end
• Most fields have been removed

portal-dataset-2016-updates.tar.gz

[bnchdrff]

@anderoy there are some bad data in esd_hs_2016:

• for records 1620, 720 and 7680 a column has a value of #REF!
• many rows have empty but not-null values for enrmnt_2013
• record 8888 has many empty but not-null values
• record 8471 has two cols with a value of #VALUE!

[bnchdrff]

esd_k8_2016 is clean though! :cake:

[bnchdrff]

How will queries against fiveessentials_2016 work? Should the api consumer get both 2015 & 2016 rows when they do a query for a certain bcode?

[anderoy]

Yes. The queries will typically be by bcode...which should be unique (I hope). No school should have both 2015 and 2016. If they do, then I'll just need to fix it. OR the API could just return the newest record if there are records for both years (in the same dataset).

On Sat, Jul 16, 2016, 17:33 Benjamin Chodoroff notifications@github.com wrote:

How will queries against fiveessentials_2016 work? Should the api consumer get both 2015 & 2016 rows when they do a query for a certain bcode?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESDet/esdportal_profile/issues/57#issuecomment-233152142, or mute the thread https://github.com/notifications/unsubscribe-auth/AAk1nNo5Tj5Qg35-B-CXl9_4nJAh4Ok3ks5qWU4NgaJpZM4JOFrD .

[bnchdrff]

@anderoy after our chat today I modified esd_hs_2016 to convert REF/VALUE errors and blank cells to NULL: esd_hs_2016_v10.csv.gz

[bnchdrff]

@anderoy There is a duplicate bcode in fiveessentials_2016:

Detroit Edison Public School Academy - PreK-2 42564 8651 4 2 53 73 77 35 81 2016
Detroit Edison Public School Academy 3-8 42564 8651 4 2 53 73 77 35 81 2016

[anderoy]

You can get rid of the PreK-2 entry.

On Mon, Jul 18, 2016, 21:48 Benjamin Chodoroff notifications@github.com wrote:

@anderoy https://github.com/anderoy There is a duplicate bcode in fiveessentials_2016:

Detroit Edison Public School Academy - PreK-2 42564 8651 4 2 53 73 77 35 81 2016 Detroit Edison Public School Academy 3-8 42564 8651 4 2 53 73 77 35 81 2016

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESDet/esdportal_profile/issues/57#issuecomment-233508907, or mute the thread https://github.com/notifications/unsubscribe-auth/AAk1nMb22NtAfX9eU11zgCztsHuyGZgdks5qXCz6gaJpZM4JOFrD .

[bnchdrff]

Also there's this row:

Detroit Christian Schools of Excellence 42546 820102276 4 4 NULL NULL NULL 99 91 2015

The bcode is 820102276 -- is this intentional?

[anderoy]

Man...I can't trust anyone at esd to give me a clean dataset. Should be 2276

On Tue, Jul 19, 2016, 18:10 Benjamin Chodoroff notifications@github.com wrote:

Also there's this row:

Detroit Christian Schools of Excellence 42546 820102276 4 4 NULL NULL NULL 99 91 2015

The bcode is 820102276 -- is this intentional?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESDet/esdportal_profile/issues/57#issuecomment-233782049, or mute the thread https://github.com/notifications/unsubscribe-auth/AAk1nA5eAdyoVBRsEVAkneLAnC8a1Kt8ks5qXUtGgaJpZM4JOFrD .

[bnchdrff]

Ready for review at https://2016-july-updates-3mhev6qb5ihtc.us.platform.sh/

[bnchdrff]

Deployed in 4f036e37e2ca787518ed498c8d756608eee27e74