Map HXL hashtags to fields during Export

willdoran commented 6 years ago

Overview

Users should be able to define all mappings for all Fields for a given Survey through a single interface.

HXL Tags and attributes will be available and grouped correctly.

HXL Tags and attributes will be selected by the user and mapped to the corresponding fields

Design prototypes: https://xd.adobe.com/view/2690f082-d88d-4788-5db7-b04c9474a404-50a1/?fullscreen

Requirements

[x] Users can see all surveys and their fields in the export screen
[x] Users can assign mappings between Survey Fields and hxl hashtags
[x] Users can save mappings by requesting an HDX or CSV HXL Export

This issue does NOT include the following:

Users can subsequently edit mappings. This will be covered in a separate issue.

Dependencies

2515
2520

caharding commented 6 years ago

@Shadrock can you scope this out more

Shadrock commented 6 years ago

There are three issues under the HDX epic that relate to the use of the humanitarian exchange language (HXL) and how it might be applied to the data structure of an Ushahidi/COMRADES deployment: #2518, #2519, #2520. I view all of these issues as, more or less, the same and will attempt provide information here to help respond to all of them. This has already been sketched out, to some degree, in the COMRADES / HDX Integration doc here under the ‘user stories’ section.

Very briefly, HXL is a lightweight data standard designed to improve data sharing (see the HXL website for more information). Essentially, HXL is used as a way to denote commonly sought after attributes in a standard way. The primary focus of HXL is tabular-style data such as spreadsheets or API output from database tables, which represent the vast majority of the operational data collected in the humanitarian sphere.

A typical case is a particular variable, such as the number of people affected by a natural disaster, appearing in a multitude of ways across a variety of organizational data sets. It could be rendered as: “Number affected”, “Affected”, “People affected”, “# de personnes concernées”, “Afectadas/os”, or “عدد الأشخاص المتضررين.” Regardless of how the data provider has labeled this variable, humanitarian software needs to be able to recognize the figures. To accomplish this, HXL can be added (usually as part of a manual process) to the second header row in a spreadsheet to make the data machine readable and allow multiple data sets to be concatenated on common variables.

As a data provider, taking this step ensure that your data have operational value that is quickly apprehended by other (generally formal) humanitarian organizations working on the same crisis. Moreover, ensuring that your data are marked-up with HXL allows you to take advantage of an emerging suite of online tools that help produce rapid data visualizations based on HXL.

So, the question for this bundle of issues - as I understand it - is, “how can we ensure that the survey fields in Ushahidi deployments map to standard HXL tags?” The short answer to this is, “I don’t know.”

In some cases, there are relatively standard fields that could be mapped automatically: those concerning a date, a location, etc. (see the HXL attribute dictionary here). However, because we allow users to create their own surveys, it may be that we want to provide them with some tool to assign HXL tags to specific fields. For example, it’s typical for surveys to include a variety of fields referring to the location of something. This can include the latitude and longitude fields that can be manually entered but are more often automatically derived from the manual placement of a pin on the map. When viewed as a .csv export from a deployment, these fields are usually titled LAT and LON in the header. The HXL standard for these fields would be #geo +lat and #geo +lon.

I feel that there are a handful of fields that we could automatically map to an HXL standard. From there, the question about how to map fields becomes more nuanced. Should we push the user towards creating fields that respond to HXL? Doing so would certainly make our deployer’s data more meaningful to others and is something that I would personally advocate. Can we allow users to map extant HXL tags to their own survey fields? I have a variety of not-very-well-articulated thoughts about how this might be accomplished and what it might mean for some form of UI, which I'm happy to elaborate on more as needed.

Shadrock commented 6 years ago

See additional discussion in this Gdoc.

jrtricafort commented 6 years ago

@Erioldoesdesign I clarified the scope of this issue and removed some other issues that were duplicative of this one. This issue is now specifically about supporting HXL Mappings on Export (not specific to CSV or API export, just export in general). I know we have designs for this since you showed them to us a few times. Can you reference your designs here, and then move it to Ready For Spec? @rowasc can take it from there :)

Erioldoesdesign commented 6 years ago

Pretty sure this is final designs ready for platform meeting tomorrow: https://xd.adobe.com/view/2690f082-d88d-4788-5db7-b04c9474a404-50a1/?fullscreen

rowasc commented 6 years ago

Can we move this to done now or is it going to change?

Erioldoesdesign commented 6 years ago

@rowasc Pretty sure this is done as it seems to describe the design task and research. Ticked the boxes in the tickets description and I feel that this is now broken up into individual tickets so perhaps this is a 'done' rather than 'ready for spec'

ushahidi / platform