hotosm / OpenMapKitServer

OpenMapKit Server is the lightweight server component of OpenMapKit that handles the collection and aggregation of OpenStreetMap and OpenDataKit data.
http://openmapkit.org
BSD 3-Clause "New" or "Revised" License
4 stars 9 forks source link

Add 'Download Filtered Attachments' function #84

Closed dakotabenjamin closed 5 years ago

dakotabenjamin commented 5 years ago

from #83 cc @willemarcel @mataharimhairi

I made this a new issue because I have some questions about functionality.

image

In the filters, the page size/number are outside, however both csv and json downloads will only include the current page (in code this is called limit and offset). Is this desired functionality? Should we move the pagination options above the filter button to reduce confusion?

Another question about the set of files/attachments. Are they always going to be recorded in the data.json somewhere? For the most part it seems there are images and some xml files. Are other file formats common? This may not be an issue but I want to make sure.

There seems to be some discrepancy with filtering dates. The json aggregator uses the start/end dates but the ui uses submission dates. Is it ok that I stick with the start-end dates? It may lead to users not getting the intended data.

mataharimhairi commented 5 years ago

@dakotabenjamin I would agree regarding moving the filter button below the options, so it a little clearer that it needs to be pressed to actually filter the data. It isn't that intuitive at the moment, and I did it a couple times thinking it automatically filtered the data once I had selected an option.

As for the filtering dates, to me as a data collector and manager - it would make more sense to filter by the data by submission dates, as start-end dates could mean that a surveyor accidentally started a survey on a feature, but stopped because it wasn't correct and continued again a day or two later.

@Bhawak What would make the most sense to you and your team and you are currently in the process of using it?

Bhawak commented 5 years ago

@dakotabenjamin since the pagination is not for downloading the csv,json,xml etc file, I believe re-positioning the button make sense or adding a line separator and changing the text (eg. currently showing 'x' items of page 'y').

@mataharimhairi In the context of our survey, submitting the survey data on the same date might not be possible at all times. This is because we have connectivity issues in Nepal. I am ok with the filter that we have for downloading data.

If the data collected from the survey is very large (eg. more than 1GB/day), the problem still exists even if we filter through the submission date. From our field experience, it would be easier for us to validate data if we could download the media attachments individually. For eg in kobo, if we want to download the image for the image id: 1538997477257.jpg then we can download via. https://kc.humanitarianresponse.info/attachment/original?media_file=kll/attachments/1538997477257.jpg and also this is also helpful to create a batch file.

willemarcel commented 5 years ago

Some details:

@Bhawak We have the option to download one single image of a submission, but it doesn't work if the form has multiple images or if it has other attachments saved in a form field other than image. We could have the option to list and download all attachments of a submission, but we would need to define a pattern on the forms and use the same field name for all attachments.