A error-filled SW file, please

[cjhendrix]

We discussed this a while back, but it would be good for us to have a ScraperWiki CSV zip file filled with the kinds of errors we would like to be able to catch during validation.

I'm not sure what the best way is to build something like that, but for sure we would want a checklist of errors that we should catch so we know when we are successfully catching them all.

[rosnfeld]

I think this could be built manually, perhaps by modifying an existing SW file.

A few ideas to start with:

• wrong number of columns/fields in a row
• garbage data in each field (e.g. have a row for each of the dsID, indID, region, value, period fields that is otherwise good but has random characters in that field)
• only half of the regions populated for a given indicator (a problem that has really happened)
• values that have the right type but are outside configured ranges (e.g. "-1" disasters in a given year)

I realize only the last example is something we talked about explicitly during our validation talk a month ago, as that's more the flavor of things I had been looking at.

[cjhendrix]

Yes, that's the sort of stuff I was thinking of. I think at the moment that we would do pretty well against this list except for the 3rd item.

[rosnfeld]

(The 3rd item unfortunately continues to happen - the most recent SW data seems to have only a small portion of regions covered by World Bank data)

[cjhendrix]

Yes, but at least in that case, the server is alerting us that something went wrong with the scraper. @JavierTeran , I wonder if we shouldn't ask Dragon about that one. It seems to have persisted for several days.

[JavierTeran]

Sure, I am writing to him now. Thanks

[JavierTeran]

And for the dummy file, sometime ago I started preparing one, we talked about it. I will have it ready by the end of next week.