search

18F / data-act-pilot

This small DATA Act pilot contains code that translates agency data to a uniform DATA act format.
Other
21 stars 14 forks source link

177 fix extra column error #196

Closed bsweger closed 8 years ago

bsweger commented 8 years ago

This fixes #177 and introduces two things:

  1. Addresses some of the brittleness around csvs with different newline formats. Normally I'd just open the .csv in universal-newline mode, but in this case, that would require a hack around the Flask internals that stream the file. Instead I used pandas, which I eventually was going to incorporate anyway to reduce some of the column name hardcoding. This means we'll have to figure out why the Pandas install doesn't work on the Vagrant box.
  2. Ignores extra columns in the validations. For this prototype, agencies may be submitting files that contain the larger universe of required data elements, whereas the scope of this pilot was smaller (just the original 83 element). To make this easier, we're validating the elements in our scope and just ignoring anything else on the .csvs.
bsweger commented 8 years ago

@kaitlin thanks for the review--the function still returns a list of dicts, so I removed the text that says the list is created by csv.DictReader