search

ushahidi / platform

Ushahidi Platform API version 3+
http://ushahidi.com
Other
680 stars 506 forks source link

Admin can import a spreadsheet / CSV of reports #602

Closed ushbot closed 8 years ago

ushbot commented 9 years ago

Created by shadowhand on 2014-05-21 05:45:51.

Imported from https://phabricator.ushahidi.com/T272

ushbot commented 9 years ago

Comment by shadowhand on 2014-05-21 05:51:36:

Sara, see the parent task for how export will be implemented functionally for exporting. Is it safe to assume that we can get IFES to upload CSV files exclusively, or are they going to want support for other formats?

Henry, focus on export for now, but keep this task in mind as a follow up.

ushbot commented 9 years ago

Comment by bodacea on 2014-07-12 05:56:05:

Only need CSV for IFES. V2.7 had CSV and XML as options - might need to ask @Angela to check who in the community uses the XML options.

ushbot commented 9 years ago

Comment by bodacea on 2014-08-22 07:31:32:

Have mapped CSV formats between V0, V1, V2 and V3; coding to upload V0/1/2 files into V3 isn't complicated. Mapping is:

|V3|V2|V1|V0| |ID|#|SERIAL|“"| |PARENT| | | | |USER| | | | |FORM|FORM #| | | |TITLE|INCIDENT TITLE|INCIDENT TITLE| | |CONTENT|DESCRIPTION|DESCRIPTION|DESCRIPTION| |TYPE| | | | |STATUS| | | | |SLUG| | | | |LOCALE| | | | |CREATED|INCIDENT DATE|INCIDENT DATE|INCIDENT DATE, INCIDENT_TIME_H, INCIDENT_TIME_M, INCIDENT_TIME_AM| |UPDATED| | | | |TAGS|CATEGORY|CATEGORY| | |SETS| | | | | |LOCATION|LOCATION|LOCATION, NEAREST_CITY| | |LATITUDE|LATITUDE|LATITUDE| | |LONGITUDE|LONGITUDE|LONGITUDE| | |FIRST NAME| | | | |LAST NAME| |SENDER_NAME| | |EMAIL| | | | |APPROVED|APPROVED| | | |VERIFIED|VERIFIED|VERIFIED| |CUSTOM VALUES|CUSTOM VALUES| |riots,deaths,property,govt_forces,civilians, looting,rape,peace,idp,name_1,name_2,name_3,name_4,video_link,news_link, image_1,image_1_t,image_2,image_2_t,image_3,image_3_t,image_4,image_4_t, image_5,image_5_t,video,sender_location,active,date_add,deleted|

ushbot commented 9 years ago

Comment by @rjmackay on 2014-12-09 15:33:51:

I'm going to de-assign this and move to backlog for now.. since work isn't actively getting done.

ushbot commented 9 years ago

Comment by @rjmackay on 2015-01-15 16:27:44:

@anarghya can this move to Feb?

ushbot commented 9 years ago

Comment by anarghya on 2015-01-16 13:26:36:

Can you please help me understand the time cost for building this? It is a critical feature that will enable our launch partners to test so I would hesitate to push it out. Is this something camelcase could build?

Thanks!

ushbot commented 9 years ago

Comment by @rjmackay on 2015-01-28 12:04:51:

Theres a few different moving parts

  1. handling file uploads - we don't really do this much at all yet
  2. handling the data import on the backend
  3. adding UI for importing

My main drive to push this to Feb is that its not an MVP1 feature.. and we haven't got MVP1 done yet. And it doesn't block anything else. Camelcase only worked on backend features with limited scope. We don't have any kind of architecture plan for handling data import yet.. so it seems like a bad candidate. Unless @aMoniker could write a solid spec for it?

I started on some basic data import work for v2 imports.. which might be reusable.

ushbot commented 9 years ago

Comment by bodacea on 2015-01-28 15:10:40:

Users (e.g. IFES) will also eventually need to csv import/export object other than reports (things like lists of reporters, for instance). The V2 CSV files weren't easy to adapt to non-reports; ir would be good if V3's csv code started with these other possibilities in mind.

ushbot commented 9 years ago

Comment by @rjmackay on 2015-03-12 09:47:41:

@brandonrosage @sophshep CSV import is planned for early April.. we can just do a rough UI an improve later. However if you have ideas on how/where this should fit in that could help. At the moment I guess it could just live under tools somewhere? Given we need 2.x imports and other data import things later maybe we have a section for that somewhere?

ushbot commented 9 years ago

Comment by @sophshep on 2015-03-12 11:03:14:

I think a global import/export button would be in tools.

Would a user ever need to export a CSV of a set? If so, we'll need to account for that in the sets UI.

ushbot commented 9 years ago

Comment by @rjmackay on 2015-03-12 11:05:19:

Would a user ever need to export a CSV of a set? If so, we'll need to account for that in the sets UI.

I think so. Certainly when NGOs use V2 as an internal tool they've wanted to allow anyone with access the ability to download the data as a CSV (either a Set or just a one of search result).

Import is definitely admin only though - so that's in Tools.

ushbot commented 9 years ago

Comment by @rjmackay on 2015-08-06 08:54:15:

CSV import is already doable via the CLI. (landed with D874) It takes a CSV file + json map of source to destination fields + json set of fixed values (ie. form_id : 2 to set which form we're using)

Remaining

We'll need to batch import later, but just get something working first.

ushbot commented 9 years ago

Comment by @rjmackay on 2015-08-06 09:03:55:

You can have a look at the CLI command code to see how the import use case works: https://github.com/ushahidi/platform/blob/master/src/Console/Import.php#L70

for a first pass I'm for you to just copy/paste and then populate data from the API request instead of the commandline. Whatever works

ushbot commented 9 years ago

Comment by @rjmackay on 2015-08-11 14:12:39:

@evansims how's this going? Can you give an ETA? This really needs to go out this week, so let me know ASAP if its not going to be ready.

ushbot commented 9 years ago

Comment by @evansims on 2015-08-13 13:38:50:

@rjmackay Had a hardware failure over the weekend that has set me back on things. Had to get my environment set back up, but I'm up and running now. Will get this to you as soon as possible.

ushbot commented 9 years ago

Comment by @rjmackay on 2015-08-13 17:06:42:

Ok. Thanks for the update. We're launching v3 publicly next Wednesday.. so we really want this in by then.

rjmackay commented 9 years ago

@brandonrosage The backend for importing CSVs into V3 already exists but we keep putting off building a UI. Could you put together a design for this in the next ~week? or are you slammed already?

The way this works in technically is a user can specify

brandonrosage commented 9 years ago

I've staged two layouts to meet these needs:

Take a look, @rjmackay, and let me know if you need anything else to complete this feature.

jshorland commented 9 years ago

This looks great, Brandon! I really like the matching field in CSV to field in V3 post type. Two cases I think we’re missing —1). if there are errors in import, for both individual rows in the CSV and the CSV as a whole, and 2). what if the field itself doesn’t match the format required for the field in the post type?

On Nov 3, 2015, at 7:23 PM, Brandon Rosage notifications@github.com wrote:

I've staged two layouts to meet these needs:

Import http://github.ushahidi.org/platform-pattern-library/assets/html/5_layouts/settings-import.html: The second-level menu item where we present all import options. Of course, for now, it assumes just the one "CSV" option, with controls for the post type and choosing the CSV file. Import CSV http://github.ushahidi.org/platform-pattern-library/assets/html/5_layouts/settings-import-model.html: The view generated from the previous CSV submission, where we display the total number of posts, a control to indicate whether they should all be verified (if the post type includes verification), and a listing of the CSV file's columns paired with controls to indicate which post type field they should map to. Take a look, @rjmackay https://github.com/rjmackay, and let me know if you need anything else to complete this feature.

— Reply to this email directly or view it on GitHub https://github.com/ushahidi/platform/issues/602#issuecomment-153404188.

rjmackay commented 9 years ago

Wow! That was super fast. Looks really good.

I think I failed at explaining the setting fields to a fixed value. "Mark all posts as Verified" was just an example.. it applies for every field in the post type. Here's a few more examples:

.. maybe I'm overthinking this though and users should just add that info to their CSV before they upload?

There are a couple of special cases we should do:

... I can think of a bunch of other possible options but I'm going to resist and keep to the minimal working version.

brandonrosage commented 9 years ago

@rjmackay Thanks for the clarification. I've staged a third layout to meet these needs. Here is the roundup:

Take a look and let me know if you need anything else to complete this feature.

rjmackay commented 9 years ago

"Assign value to unused post fields" <- I like this.. hopefully ends up way simpler for most users. Thanks for the super fast turn round. I think this is everything we need!

rjmackay commented 9 years ago

Taking myself off this. I've attached it to the Responder milestone so it'll get schedule for that soon.

rjmackay commented 8 years ago

Closing in favour of #844