Improve checking

[dwr-psandhu]

Checking relies on parsing log. It would be better to separate it the failures to another file format and use those in checking as well as retry/restart logic

[water-e]

Mostly I rely on directory level diffs for checking. I have added the ability to filter out known repeat known misses on the downloads. I'm not sure retry logic would depend on parsing the logs, so closing the issue. We need to make the logs more useful, but for that it might be more useful to think of the current logs as detritus from development, consider rooting them out and decide what we want to log.