search

ThreeSixtyGiving / datatester

Scripts to asses the quality of data from http://data.threesixtygiving.org
MIT License
3 stars 2 forks source link

We don't seem to clean up properly between loop iterations #36

Open robredpath opened 5 years ago

robredpath commented 5 years ago

Recently, we had an issue downloading one file for the daily data test. Some information (eg file_type, min/max award dates) was duplicated down from the row above the row that had the error. Presumably this should be blank.

I've saved the data as a named version in the daily data test - https://docs.google.com/spreadsheets/d/1iRH0N07Fi-XM6HcZLSR688EiPA4EQGc5wSP1hJIx3L4/edit#gid=0 -

robredpath commented 5 years ago

See also https://opendataservices.plan.io/issues/21460

robredpath commented 5 years ago

@KDuerden this should fix the Lankelly/LandAid issue in the DDT. @mrshll1001 can you make sure that you update datagetter before the next GN load? Ping me if you'd like a hand with that :)

KDuerden commented 5 years ago

@robredpath thanks for the heads up. The changes seem to have knocked 360Insights offline, as it expects different behaviour (a file size value every time). @drkane is on holiday this week so we need to find a way to keep 360Insights up as much as possible until he's back. I'm rebuilding the registry now, having removed a few Lankelly/Landaid files to see if that gives a clear result.

robredpath commented 5 years ago

@kindly @michaelwood could we (temporarily or permanently) make this give a filesize of 0 to help get Insights working?

robredpath commented 5 years ago

Or, would the correct behaviour be to drop the entry if we can't get the file? @KDuerden ?

KDuerden commented 5 years ago

@robredpath I'm not qualified to say.

drkane commented 5 years ago

Omitting the record entirely is probably the safest quick fix (I'm not sure if there are further properties in the same template that would broke, so just adding a dummy file size isn't guaranteed to fix it).

Other option is just to revert to the previous state and then reapply this patch later on.

When I'm back behind a computer I can change insights to check for the right properties and we can revert to the current state.