Work out how importing from CSVs should work

[TomSteinberg]

Talk to people - ask them

[tmtmtmtm]

I would suggest we solve this outside the UI first — i.e. with a standalone command-line tool for converting CSV to Popolo-format JSON, that could then in turn be bulk-imported. This is orthogonal to what the format of that CSV should be, etc., but doing it this way lets us decompose the problem, get a couple of other tools in the process[1], and make the CSV upload a fairly thin wrapper around those.

[1] And, with the right prodding, we might be able to get someone else entirely to write or maintain those.

[mhl]

@chrismytton, @tmtmtmtm @duncanparkes and I had a conversation about this just now.

We suspect that those people who had previously reported bugs and problems with the migration tool has since moved on or found other ways to do what they wanted, so we're mostly thinking about the users as being Tony or people wanting to contribut to open parliaments.

Our consensus that the configurable mapping of columns to Popolo fields should be abandoned or hidden, and instead we should tell users what columns they needed to provide in their spreadsheet - the reasoning is that the current system is confusing to people, has some bugs and doesn't deal with memberships at all. Most people who are preparing lists of politicians in a spreadsheet, on the other hand, should be comfortable with changing the heading of one of those columns.

This should at least support an import that includes columns for:

• A mandatory ID field (so that the import can be re-run with an updated spreadsheet)
• Columns that can be mapped to Popolo Person fields, including primary name, email, links and identifiers
• A party name, which can be used to create an organization of that name (if it doesn't already exist) and a membership of that organisation
• A title, like "Member of Parliament for Cambridge", used for creating the post
• A classification of that title, e.g. "Member of Parliament", used for the classification field of that post
• Optional start date (for their membership of that post)
• Optional end date (for their memership of that post)

This would allow you to create a PopIt instance with a basic set of people, organizations (parties), posts and memberships, I think.

If someone wants to supply additional information about the organizations, there could be an optional second file upload field for an "organization" CSV, which could contain details of the parties - the names of parties in the basic CSV should match the names in the organization CSV, of course.

[chrismytton]

The migration tool was removed in https://github.com/mysociety/popit/pull/371. The plan is to point people who want to import from CSV to http://everypolitician.org. Therefore closing this ticket.