search

frictionlessdata / goodtables-web

[DEPRECATED] Please use http://try.goodtables.io/
MIT License
15 stars 6 forks source link

Don't swallow all errors in PipelineBuild Error #44

Closed pwalsh closed 9 years ago

pwalsh commented 9 years ago

At least make sure that specific error messages are shown to the user.

rufuspollock commented 9 years ago

@pwalsh what do we need to fix this - about 50% of files i test from UK spend files get this error and it ain't very informative ;-)

pwalsh commented 9 years ago

It shouldn't be much work (a few hours including tests). From my experience, the UK data is the encoding issue, as I mentioned here: https://github.com/okfn/goodtables-web/issues/41#issuecomment-106196131

rufuspollock commented 9 years ago

@pwalsh great. Let's do it. Can we detect encoding in goodtables better? (or given this come from spend dashboard should we compute encoding when we scrape files and be using that in the url).

Also is this something for Artem or just you?

pwalsh commented 9 years ago

Encoding is either chardet (https://github.com/chardet/chardet) for automatic, or an explicit string from the user. Yes, we could probably get it from the headers on the data.gov.uk server, presuming they are correct, as a third option.

Artem is full this week: I could do it or he could. There are a few good tables issues now - it is just a matter of priority. I need all day tomorrow on census, and some OS. I can do a round on Good Tables on Thursday, first doing the UI of results, and then this. WDYT?

rufuspollock commented 9 years ago

Thursday sounds good. I'd say this + GET item is more important than the UI item which we can do last.

pwalsh commented 9 years ago

I've fixed this. eg: http://goodtables.okfnlabs.org/reports?data_url=https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/299476/December_2013__25k_Transparency_Data.csv&format=CSV&encoding=&schema_url=https://github.com/okfn/goodtables/blob/master/examples/test_schema.json

pwalsh commented 9 years ago

Another good example: http://goodtables.okfnlabs.org/reports?data_url=http://www.walesoffice.gov.uk/files/2010/11/Wales-Office-December-2012-spend-over-500-22-01-2013.csv&format=CSV&encoding=&schema_url=https://raw.githubusercontent.com/okfn/tabular-validator/master/examples/hmt/spend-publishing-schema.json