search

mysociety / alaveteli

Provide a Freedom of Information request system for your jurisdiction
https://alaveteli.org
Other
391 stars 197 forks source link

Merging changes from external databases #65

Closed bjh21 closed 1 week ago

bjh21 commented 13 years ago

There are several external databases from which WhatDoTheyKnow could import lists of public authorities if it had a way of keeping its database in sync with the external one after the import. I think the high-level requirement is for an import and re-import mechanism that can detect changes made on WhatDoTheyKnow that conflict with changes in the external database and present these for manual intervention. The mechanism should also flag bodies that appeared in the last import but not in this one, since these may need to be marked as defunct.

sebbacon commented 13 years ago

Out of interest, what are the databases? Do they do CSV format?

bjh21 commented 13 years ago

The extant databases I was thinking of were EduBase and the Scottish Information Commissioner's list of public authorities. The Scottish IC list (which we've already imported) is in CSV format. I don't know about EduBase.

I also have half a plan to scrape district councils' lists of parishes in ScraperWiki, which would produce CSV. I'd expect most other sources to be able to be coerced into CSV.

bjh21 commented 13 years ago

Oh, and my general assumption is that whatever format we get the database in, we'd massage it into some kind of standard CSV import format before feeding it to Alaveteli. But this massaging is much easier if it can be stateless, with the difficult (and non-source-dependent) merge logic living in Alaveteli.

HelenWDTK commented 1 week ago

This issue has been automatically closed due to a lack of discussion or resolution for over 12 months. Should we decide to revisit this issue in the future, it can be reopened.