search

18F / data-act-pilot

This small DATA Act pilot contains code that translates agency data to a uniform DATA act format.
Other
21 stars 14 forks source link

Validation from front end #137

Closed msecret closed 9 years ago

msecret commented 9 years ago

Still not working. The DictReader in the validator is not working with the uploaded file for some reason. I suspect its something to do with the uploaded files format.

bsweger commented 9 years ago

Hi Marco,

Using the files in the dummy data folder, I was able to run validator.py from the command line (got the massive error output as expected). I did notice 2 bad column names and some issues with int data in awards.csv, so I uploaded a new copy of that file to the dummy data folder on Google Drive.

Also, SBA doesn't have all of the TAS components, so to cut down even more error noise, I think we can set the following fields to note required in the appropriations, object class, and award_financials.csv:

If you're still getting errors, can you let me know and send the output? Thanks (and thanks for your patience)

msecret commented 9 years ago

So the main problem I'm having is when you upload a file through the website. So I removed the open call in validator.load_data on line 53 and just passed the file data straight from the POST request. When I do this, the DictReader returns this for all the rows:

[{'': '', '249000000': '197000000', '73': '73', '256000000': '0', '100': '400', '361000000': '0'}]

So something is going on with it not being able to parse the data when it's passed directly from the POST request rather then opening a file. When coming from a POST request the file is a werkzeug FileStorage object, which should have similar methods to an opened file. I can't figure out what sort of transformation it needs in order to work with the csv DictReader.

I also wanted to avoid saving the file when it's uploaded because I think that could affect our ATO process. I was even hesitant to use python's temporary file functionality.