Userstory Upload Data Tool - Deltares

lornalittle commented 8 years ago

As a hydrological scientist, I want to upload a single data file with research results to the SWITCH ON server, so that I can share it with my research colleagues.

lornalittle commented 8 years ago

Further userstory based on todays discussions: As a hydrological scientist, I want to upload Multiple data files with research results to the SWITCH ON server, so that I can share it with my research colleagues.

lornalittle commented 8 years ago

As a hydrological scientist, I want to get a link/URL for my uploaded data so that I can copy it into the metadata tool

lornalittle commented 8 years ago

Another userstory: As a hydrological scientist, I want to get a DOI connected to my uploaded dataset, so that I can share it with my research colleagues and use it in citations.

lornalittle commented 8 years ago

Regarding combining the upload function and metadata tool: As a hydrological scientist, I want to use the upload tool and metadata tool together, so that my data is not uploaded without accompanying metadata.

beekhuiz commented 8 years ago

As a GIS scientist, I want to attach a hash-code or checksum to my uploaded data, so that I can know when the data is changed.

beekhuiz commented 8 years ago

As a GIS scientist, I want to upload my grid datasets to an OPeNDAP server, so that the data can be retrieved with web services and users don't need to download the whole dataset.

beekhuiz commented 8 years ago

As a GIS scientist, I want to upload my vector data to a Geoserver, so that the data can be served by web services such as WMS and WFS.

lornalittle commented 8 years ago

Feedback on tool:

it does exactly what we asked (it uploads files, single or multiple).
- its not clear how to collect the url for the uploaded dataset, or even if we can collect it at this stage.
- it is very easy to remove files in the current version - perhaps adding some kind of authorisation so that files are not so easily taken down?

AnnaKue commented 8 years ago

As a hydrologist scientist uploading multiple files for the same dataset (for example GIS data) I would like to be able to create a directory when uploading the files, or to select multiple files and upload them directly as a unique compressed file. (Of course it's possible to preliminary create a zip file and then upload it but it would be convenient to be able to do it while uploading: one step less to do)

beekhuiz commented 8 years ago

Here is a first mockup of a more complete tool:

mockup

beekhuiz commented 8 years ago

The proposed method for uploading data works as follows:

(left side of the screen): the user gives a name for the dataset to be uploaded.
The 'storage type' is chosen; for some datasets, such as NetCDF and shapefiles, specialized software exist to disseminate the data with web services. These are THREDDS and Geoserver.
Click on 'Create directory'. The right side of the screen will now show the 'Upload dataset files' menu.
Click on 'Add files' to select the files on your computer for upload.
Click 'Start upload' to upload all the files. A progress bar is shown for each upload and the total upload time.
(optional): Make a .zip file of one or more uploaded files. The files to be zipped can be selected with the checkboxes on the right. After zipping, the original files are deleted (optional?) and the .zip file is added to the list.
Finally, click on 'Store metadata' to save all information of the dataset files and return to the 'metadata tool'. If the data is stored on e.g. a THREDDS server, the dataset files contain urls to various web services, such as WCS and WMS. These offer many advantages compared with the regular upload.

p.s. the 'Data Upload Tool' is started from the 'metadata tool'.

beekhuiz commented 8 years ago

Concerning the authentication and authorization (for removal of files, creation of dataset folders, etc.): this should be dealt with together with the metadata tool. Ideally, the user logs in at the metadata tool, and when uploading the files with the data upload tool, the same login is used so no second authentication is required.

lornalittle commented 8 years ago

The look and feel of the data upload tool should be harmonised with the metadata tool and the BYOD - matching fonts, colours and layout where possible

lornalittle commented 8 years ago

The data upload tool should include all the appropriate logos

jfkeppel commented 8 years ago

As a hydrological scientist when uploading data I want to get an estimate and or continuous feedback on the duration of the upload so I can plan this action/work more efficiently.

jfkeppel commented 8 years ago

As a hydrological scientist I want to be able to resume an interrupted upload, especially for large data sets so that I don't have to start over when an upload is interrupted unexpectedly.

jfkeppel commented 8 years ago

As a hydrological scientist when uploading data I want my data to be validated against standards (syntax and semantics) so that my data is much more valuable and more easy to reuse for others

lornalittle commented 8 years ago

Add a third sentence at the top: Name should be as specific as possible. Name should contain variable and source of data (in that order). Create dataset folder button – should be called NEXT: data upload **Need a function to find if same name has been previously used.

lornalittle commented 8 years ago

Please remove the start and cancel button on the right hand side of files (unecessary).
Store dataset info button - Call it Next: Create metadata
Need a button to go back on this page (if we need to make changes).
There should be a progress bar and switch on logo – all pages should have the same frame as in metadata tool. Also add the green info box (as in metadata tool) with info instead of ‘hover over’ function for information.

lornalittle commented 8 years ago

Overall comments: impressed, it is a nice tool and it works well

lornalittle commented 8 years ago

When considering the upload data tool connected to the metadata tool:

Linked from upload: Should not be able to change the name of dataset if you came from the upload tool.
Name and link and file box should be greyed out/deactiviated if you come from the upload tool.
File type came up incorrectly from previous tool. Default should be ‘please choose’.
Link from Deltares tool was not valid.
Should be a link back to upload tool from metadata tool

AnnaKue commented 8 years ago

Hi ! Here are some remarks after testing the tool which looks very promising.

At the first step you chose a name for the dataset, I think this name should appear somewhere (as a title) at the next step (when you are uploading files) so you know where you are.
The button to go directly to the upload metadata tool is very good, but if you click on it and start uploading metadata you may forget to finalize the upload (by clicking on “start upload”). Maybe there should be at least a warning to incite you to finish the upload before going to the other tool.
It’s not clear where is the data going and how to find it later (what is the final URL actually). The name of the file becomes a link when it’s uploaded so I think that’s the final link, but maybe there should be a button “copy URL to clipboard” or something like that. The URL for the whole uploaded dataset (i.e. the folder) should also appear somewhere.

lornalittle commented 8 years ago

When I click on the 'upload new dataset' button, the page doesnt open the upload page.

beekhuiz commented 8 years ago

(i can't seem to reply in github so I add a new comment..). @lornalittle : did you type in a name for the dataset? Only then can you start uploading a dataset. There should be an indicator that first a dataset name need to be given, see image below. If you did type in a name and the button doesn't work there is something else going on.

lornalittle commented 8 years ago

@beekhuiz Yes, I typed in Test_data_lorna and clicked upload new dataset - I come to a page saying website not available.

beekhuiz commented 8 years ago

@lornalittle Apparently the proxy server is not accessible anymore from outside the Deltares network.. (I just tested it from an outside connection.. from inside Deltares it works well). No idea how that happened. I will contact our ICT department to solve this ASAP. Thanks for noticing this error.

rcapell commented 8 years ago

As discussed during Skype call (2016-05-25), users should have an option to upload combined shape and data file datasets. The choice should be clear from the "Select storage server" step in the upload tool (http://dl-ng003.xtr.deltares.nl/?datasetname=test), in a way so that it becomes clear that the geoserver can take a shape-data combination as well as shape-file-only uploads.

From my user-perspective, it would be easiest to use if one does not have to choose server types, but just get a choice of data types to upload, i.e. three upload boxes on one page: "upload text files, upload NetCDF files, upload shape files", and the tool chooses the appropriate server depending on the combination of file types uploaded by the user.

lornalittle commented 7 years ago

As a hydrologist/water scientist, I want to have a DOI automatically generated and added to datasets I upload using the data upload tool so that they can be found and referred to by others and are sustainable.

https://www.software.ac.uk/blog/2016-09-26-making-code-citable-zenodo-and-github

lornalittle commented 7 years ago

As a hydrologist, I want to upload scripts so that other scientists can manipulate my data

lornalittle commented 7 years ago

As a hydrologist/water scientist, I want a DOI automatically generated for existing datasets already uploaded (eg. Root zone) using the data upload tool so that they can be found and referred to by others and are sustainable.

p-a-s-c-a-l commented 7 years ago

We use the Zenodo Service to obtain a persistent Digital Object Identifier (DOI). Zenodo is a research data repository. It was created by OpenAIRE H2020 Project and CERN to provide a place for researchers to deposit datasets. For further information please consult Zenodo's Terms of Use and Policies.

The process of obtaining a Digital Object Identifier is as follows:

1) User checks "Get Digital Object Identifier" in Open-Data Registration Tool 2) User has to upload a new Dataset. "Official" DOIs are generated by a DOI Registration Agency (e.g. zenodo.org) for datasets uploaded to their servers. 3) The Data Upload Tool stores the new dataset on deltares and on zenodo servers. 4) Zenodo prereserves a preliminary DOI for the new dataset. 5) User has to complete all steps of the Open-Data Registration Tool. User inputs that are required for the DOI (e.g. creators) are mandatory. 6) Open-Data Registration Tool sends meta data to SWITCH-ON Meta-Data Repository and to zenodo 7) Zenodo publishes the DOI. 8) DOI is shown in BYOD

Request a DOI: 010-get-digital-object-identifier