#2 Download

[meomancer]

The current download of data is very slow, because the download file needs to be created for every data request, depending also on the permission level of the user. Since all well and monitoring data will be open (see previous section), there is an opportunity to facilitate and speed up the download of data. The idea is to prepare the download file in advance, i.e. whenever the database is edited. It could be one single file, or it could be one file per country. There are two advantages in having one download file per country:  users wouldn’t need to download one heavy file if they don’t need to,  it would speed up the creation of the download files whenever the database is edited, i.e. no need to rewrite the entire file, only the files of the countries where edits have been made would be rewritten. The user would have the following download options (in a drop-down menu): select one, multiple or all countries (see mock-up below).

@Kartoza: Is it possible to rewrite the download file whenever the database is edited? This could become tricky when several edits are made in a row.

If not possible, it would be acceptable to write the download files once per day (during the night). Edits to the database would only be available for download the day after, but it would be easier for the system.

Before downloading the data, the user should identify him/herself and explain what the data will be used for. This could be through open fields or drop-down lists (e.g. “Research institution”, “Consultancy”, “Water management authority”, “NGO”, etc.). The users should also provide an email address where the download file must be sent. In case of registered users, the identification part wouldn’t be necessary (data would be retrieved automatically), they would only be asked what the data will be used for. This short form will be preceded by a few words of explanations from IGRAC, as of why it is important to know who downloads the data and why.

These data submitted for each download session should be saved in the Admin section (Django). It should be possible to export the data , in order to make yearly reports on the download of well and monitoring data.

[meomancer]

Note for mapstore later:

Download data button should be just redirect to https://ggis.un-igrac.org/groundwater/record/download/ without filters (not POST)

[SterckxArnaud]

Hi Irwan, we laid out several options for downloading the data, which one have you chosen?

[meomancer]

hi @ArnaudIGRAC

The solution is:

• We provide downloaded file per country
• The file will be generated nightly
• There is form for user to fill it

The user would have the following download options (in a drop-down menu): select one, multiple or all countries (see mock-up below). Before downloading the data, the user should identify him/herself and explain what the data will be used for. This could be through open fields or drop-down lists (e.g. “Research institution”, “Consultancy”, “Water management authority”, “NGO”, etc.). The users should also provide an email address where the download file must be sent. In case of registered users, the identification part wouldn’t be necessary (data would be retrieved automatically), they would only be asked what the data will be used for. This short form will be preceded by a few words of explanations from IGRAC, as of why it is important to know who downloads the data and why.

[SterckxArnaud]

There are 2 issues:

• you still need to register in order to download the data
• once registered, I can't see any download menu

[meomancer]

Hi @ArnaudIGRAC

you still need to register in order to download the data

User that are not logged in will not be able to access the url https://staging.igrac.kartoza.com/groundwater/record/download-request Could you access it without login?

once registered, I can't see any download menu

I can see it after registration What user did you use?

[SterckxArnaud]

Hi Irwan,

Well and monitoring data are downloadable by anyone as per #487. No need to register anymore. We might well want to ask the user a few information (in cases on unregistered users),. We will propose a template for that later on.

Why do you need to go to to the top right drop-down menu to download data, while there is already a button for that ?

[meomancer]

Ah, that button is not been updated yet Currently the download is through : https://staging.igrac.kartoza.com/groundwater/record/download-request Because we need information there is form for the user information

We will fix and re-release this for next updates:

• [x] Update the above button to : https://staging.igrac.kartoza.com/groundwater/record/download-request - Need @dimasciput helps for this
• [x] Make form to be able to accessed by non registered user

[meomancer]

This has been updated @ArnaudIGRAC Please check it

[SterckxArnaud]

Thanks Irwan, here are a few comments:

1. The drop-down list under Type of Organization should be as follow:
   • National institution
   • NGO
   • United Nations
   • Other international organization
   • Private company
   • Education and research institution
   • Data and research institution
   • Consultancy
   • Media
   • Civil society
2. We need a field "Country", where the user indicates the country where he is based.
3. We don't need the field "Position"
4. Let's be consitent with the sign-up page (https://staging.igrac.kartoza.com/account/signup/). I suggest the following structure

[SterckxArnaud]

5. I see that the data request page is identical whether you download GGMN data or bulk well and monitoring data. I just want to make sure that the system makes the distinction between the two datasets

[meomancer]

Hi @ArnaudIGRAC I'm sorry but how to we distinct between ggmn and "well and monitoring" data?

• Currently ggmn is the wells that has measurement data on it
• For "well and monitoring" data, basically all of well regardless it has measurement or not

Or is it not correct?

[SterckxArnaud]

GGMN data are monitoring data submitted by country organisations (not individuals). They are available here https://ggis.un-igrac.org/view/ggmn Well and monitoring data are basically all the data in the database, including the GGMN data. They are available here https://ggis.un-igrac.org/view/well-and-monitoring-data

[meomancer]

Ah, ok @ArnaudIGRAC I will update the download to distinct between them

[meomancer]

Hi @ArnaudIGRAC The download request form has been updated Also user can select between Well and Monitoring Data and GGMN

Currently the cache files are generating, so it will be empty when download the data now I will check it until tomorrow and notify you the progress of generating cache file

[SterckxArnaud]

Hi Irwan, I am not sure to understand the cache issue but we need to move the staging version to production now, remember we have our second work meeting today at 12.45 UTC+1

[SterckxArnaud]

Download doesn't seem to work. I tried to download data from Austria but the zip files are empty.

[meomancer]

Hi @ArnaudIGRAC I'm still working on new version of download Please wait for the new version to be been deployed (my target is this week)

[SterckxArnaud]

Hi Irwan, clicking on the data download button brings me to this

[meomancer]

Hi @ArnaudIGRAC Could you retry it again? It was because of our server was out of space Now it is working fine

[SterckxArnaud]

Hi Irwan, it doesn't seem to work. All the data files I receive are empty. I also noticed a typo in the download request page: it should be "for which"

[meomancer]

hi @ArnaudIGRAC How did you do to download?

It is still on the staging https://staging.igrac.kartoza.com/groundwater/record/download-request

I haven't released it yet to production

After it is working fine and the files are correct on staging, i will release it to production

[SterckxArnaud]

My bad, I was looking at the production site. Download works fine on staging. The typo I mentioned before is also on staging, please make the edit. I will now have a look at the download files and let you know

[SterckxArnaud]

I downloaded the data from the Gambia, and there are some issues:

1. The "wells" spreadsheet is empty. It should contain the coordinates of the wells, purpose, status, etc.
2. Some files are empty although the well does have monitoring data. See for instance well NWP62001

I also have a question: what does the number in parenthesis mean in the names of the excel files? see for instance

[meomancer]

Hi @ArnaudIGRAC i have removed the parenthesis in name, it was id of well Could you try it again?