As a researcher, I want to enter or upload the provenance for my file so that I can provide other researchers with additional information about the history of my data.

[djbrooke]

The mockups supporting this user story are available here:

https://drive.google.com/file/d/1AO_L0utJwjHEBxpj5b2xNiVkOjOG545D/view

This story covers the submission and validation of a JSON file, and the submission of manually entered provenance.

[pdurbin]

In cc09abf I stubbed out the validation that the prov file is JSON and can be parsed an a JSON object rather than a JSON array.

[dlmurphy]

In our usability testing of our Provenance mockups yesterday, we had one major piece of UI feedback that applied to this mockup:

When you upload a provenance file, the "Upload Provenance File" box is still displayed and active. This led the participant to think that it was possible to upload more than one provenance file. Because he is very familiar with provenance, he was startled to think that we'd messed this up and allowed multiple file upload.

An easy fix is to simply remove the file upload box once a provenance file has been uploaded, and only display it again if the user clicks the "X Remove" button.

[pdurbin]

As of 7948579 we are storing both types of provenance data:

• "Bundle File" (PROV-JSON, a W3C standard): stored as an "auxillary" file for a datafile (like thumbnails)
• "Description" ("free-form", a text box): stored to a new provfreeform column on the filemetadata table

To be clear, this is where the two types will go after you click "Save Changes". There's still staging work to do for the UI. Clicking "Cancel" means nothing that's been staged will be persisted at all.

[dlmurphy]

Usability testing yesterday also got us feedback on some of the instructional text for this popup:

Based on our participant's feedback, I recommend changing this message:

"Please add any information required to accurately document the history of this data file, including how it was created and how it was transformed."

TO

"In addition to a bundle file, or in place of one, you may add any information required to accurately document the history of your data file, including how it was created and how it was transformed."

I also recommend changing Description to Provenance Description.

[dlmurphy]

Lastly, @mheppler pointed out during usability testing that we need a "Cancel" button added to this popup modal:

[mheppler]

What we are building

Include recommendations from testing as outlined above by @dlmurphy and ok'd by @taniaschlatter and @djbrooke and the following:

File Upload Pg

• Edit button for each file with dropdown menu for Provenance (combine "Edit Tags" button as a dropdown option, as well as Delete, Restrict and Unrestrict)
• New Edit dropdown button in the table header, consistent with the Edit File dropdown button in the files table header on the dataset pg (the Provenance option will not be linked from this button, we are not supporting batch/multi-file editing for Provenance)

Provenance (Edit) Popup

• New Provenance popup will provide help text and form input for the "Provenance Bundle" and the "Provenance Description", along with Save Changes and Cancel buttons
• Provenance Bundle form input is a file upload widget that validates JSON format (a nice to have is to validate against the W3C schema)
  • Upon uploading valid bundle, the upload widget is hidden, and a Remove and Preview button are displayed next to the bundle file name
  • Remove button removes uploaded bundle file, the upload widget is then displayed again
  • Preview button opens Provenance graph in new window
• Provenance Description form input is a textarea component
• Save Changes stages changes as part of the File Upload workflow
• Cancel deletes changes

Provenance (Confirmation) Popup

• Warning text explaining user must preview the Provenance info they added before publish, because you can not change it after you publish your dataset
• Continue button brings user back to File Upload workflow
• Cancel button brings user back to Provenance (Edit) popup

[TaniaSchlatter]

When we have a better sense of the graph, it may make more sense and streamline the flow to include the Preview in a modal.

[mheppler]

Mockup of the Edit dropdown button in the table header for File Upload pg, as referenced in checklist above.

This is intended to bring the functionality of the same Edit Files dropdown button on the dataset pg to the File Upload pg.

[matthew-a-dunlap]

Sorry, meant to delete a comment not close the story!

[matthew-a-dunlap]

Questions:

• The Provenance (Confirmation) Popup happens when clicking "Save Changes" on the file upload page? Or does it happen when clicking save changes on the provenance (Upload) popup?
• The notes on this popup says "explaining user must preview the Provenance". Are we enforcing this? In other words, are we requiring they view the graph and only allowing publication if that has happened?

[matthew-a-dunlap]

One more:

• On the file upload page, do we want to change the buttons for the multi-file restrict and delete? Not sure if it needs it but it may be good to re-align it with the new per-file dropdown.

[mheppler]

@matthew-a-dunlap

• Confirmation popup is after save on the Provenance popup, before the user is returned to the File Upload pg
• There is no enforcing of a preview
• Yes, we want consistency with all our Edit buttons as scalible solutions going forward for all the actions related to files

[matthew-a-dunlap]

High level To-Do:

• [x] Freeform provenance commands: read, delete, update
• [x] JSON-based provenance commands: read, delete, update
• [x] Wire up file page for provenance editing: option in dropdown and popups
• [x] Wire up upload page for provenance editing: updating file table with per file dropdowns, edit global dropdown and and popups
• [x] Provenance main popup aesthetics
• [x] Provenance confirm popup aesthetics
• [x] Error dialog on save failure
• [x] Code cleanup
• [x] API tests for Provenance read/delete
• [x] Ensure normal upload work

Known bugs and issues:

• [x] Cannot save blank freeform provenance info via UI, likely due to nature of freeform text box.
• [x] Save after removing provenance json via popup does not complete. Pretty sure it worked before adding the confirm popup. This also causes the popup to get into a state where it won't reopen
• [x] Are the update commands needed? it just calls remove followed by create, and should probably be taken out of the codebase.

[mheppler]

Cleaned up the layout of the new Edit button in the table header on File Upload pg.

NOTE: Discovered that the file upload does not work. Once you select a file from your computer, it does not display in the file table, and there are no errors logged by glassfish. I assume that @matthew-a-dunlap is aware of this, based on the lengthy to-do checklist in his comment above.

Also, merged develop branch and resolved conflicts in structure.css in order to get bug fixes from @sekmiller in the work on #4051.

[dlmurphy]

I've finished the parts of the Provenance user guide that I can write from just the mockups. Once this reaches a point where I can demo the feature, I'll want to go back and make a few additions.

[matthew-a-dunlap]

Working on the upload functionality now and thinking about the flow. Here are some questions I have in the same vein:

• If a user has uploaded provenance data to a file the past and goes to look at provenance for the file, how should that be represented and how should the popup look?
• If its an uploaded provenance file, should it be shown similar to https://github.com/IQSS/dataverse/issues/4343#issuecomment-350377415? Does there need to be some difference to communicate that this file is already saved)?
• If a user goes to this popup and doesn't input any changes, should we still say "save changes" (which may confuse the user that they have made changes that they haven't)?

[TaniaSchlatter]

Hi Mathew,

Comments inline below:

On Jan 9, 2018, at 7:41 PM, matthew-a-dunlap notifications@github.com wrote:

If a user has uploaded provenance data to a file the past and goes to look at provenance for the file, how should that be represented and how should the popup look?

If its an uploaded provenance file, should it be shown similar to #4343 (comment)? Does there need to be some difference to communicate that this file is already there?

It should look like the mock-up in the comment. It shows a provenance file (test-prov.json) has been uploaded and it there. The upload box should not show, however, as per Derek’s comment.

If a user goes to this popup and doesn't input any changes, should we still say "save changes" (which may confuse the user that they have made changes that they haven't)?

It should say “save changes” as shown in the mock-up in the comment. The user can cancel if they don’t put in any changes.

Thanks!

• Tania

  — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[matthew-a-dunlap]

@TaniaSchlatter The problem that I see is that there are two different states:

1. A user has uploaded a provenance file but not saved it
2. A user previously uploaded a provenance filed, saved it, and is now looking at the file through this screen

Is our intention to have the same visual state (as shown in the mockup) for both? Won't our users be confused if they say... upload the file, click save, reopen the popup to be sure and see the same UI as before they clicked save?

[TaniaSchlatter]

The same visual state works for both. We could say “save” on the button in both rather than “save changes,” but I don’t think it is necessary, and would not want to do that without checking for consistency of wording on save-type buttons in general.

• Tania

On Jan 9, 2018, at 8:18 PM, matthew-a-dunlap notifications@github.com wrote:

@TaniaSchlatter The problem that I see is that there are two different states:

A user has uploaded a provenance file but not saved it A user previously uploaded a provenance filed, saved it, and is now looking at the file through this screen Is our intention to have the same visual state (as shown in the mockup) for both? Won't our users be confused if they upload the file, click save, reopen the popup to be sure and see the same UI as before?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[matthew-a-dunlap]

I have updated my checklist above with the state of this story. All the main functionality works and is wired up. The confirm popup is not committed but mostly done on my local branch. Hopefully will be able to wrap this up on Tuesday.

[mheppler]

@matthew-a-dunlap in your to-do above, you ask, What should happen if either form of provenance data fails to save?

I suggest using an error message from the bundle, like we do for Solr, that reads something like, Due to an internal error, your provenance information was not saved.

[matthew-a-dunlap]

My most recent commit was not tagged correctly to show up here, but was committed to the branch ( https://github.com/IQSS/dataverse/compare/4343-prov-upload-valid ). The confirm popup is now functional.

I have also updated the list above with a few more bugs/issues that need to be worked on. None of them look major at all.

[mheppler]

Cleaned up UI of Provenance workflow on File Upload pg. Checked off the following to-do items:

• Provenance main popup aesthetics
• Provenance confirm popup aesthetics

Was not able to replicate the following bug:

• If provenance popup is closed, cannot be reopened without refreshing page

[matthew-a-dunlap]

This branch is ready for code review. I decided not to expose the provenance get commands via api at this point as I think there is still some flux in that space (though likely it would be just returning the json file each time). I have added a note to the relevant provenance story.

[matthew-a-dunlap]

@dlmurphy was looking this over for documentation and noticed an issue where the provenance data will not save correctly when editing is accessed from the file upload and create dataset pages. I had lost track of this need when programming, so I am pulling this back into development.

This also made me realize another issue: on the edit files page: it is confusing what all the "Save Changes" buttons are doing. In the provenance popup, should clicking "save changes" here save the provenance data to the database?

or should it just stage the changes until save changes is then presssed again here?

If we want the provenance data to not be committed until the Edit Files Page save changes button is clicked, it still feels to me to be a bit ambiguous.

[matthew-a-dunlap]

I discussed the above issue with @mheppler , the save should happen when the page (not the popup) is saved. We will be changing the popup "save changes" to "continue" as clarification.

Digging into this deeper, because of the misunderstanding of where we were saving, the code was designed to only stage the provenance metadata for one file at a time. I will have to rewrite the code to stage multiple files provenance data and ensure the popup is refreshed correctly when switching between files. My hopes of having this done today are highly unlikely.

[matthew-a-dunlap]

A second clarification from @mheppler :

"When you open the Provenance popup, that form should still have a Save Changes button. The button text that I suggested changing was on the confirmation popup that warns you about previewing before publishing. That confirmation popup button text should be changed Continue."

We also discussed confusion around all the metadata editing paths saying "Save Changes" even tho these changes are not actually saved until the page is saved. This will be reviewed at a later date (5.0?).

[matthew-a-dunlap]

I need to tweak how one of our commands is called and do code cleanup. Otherwise its good.

I deployed the current state to https://apitest.dataverse.org , which is close enough to be looked over @dlmurphy @mheppler @TaniaSchlatter

Note: The alert is also non functional, will fix 2moro

[matthew-a-dunlap]

Had a discussion with the design team this morning. They've identified the need to add the popup to the file page. This is not as simple as just wiring the popup up for another page, as the popup in this page needs to trigger saving within itself. Its not that much work in the end, but I bet it won't be done today.

I pointed out a design issue I saw, that the saving of provenance changes is inconsistent depending on the page. In the file page saving happens in the popup while on the other pages users can complete the popup and leave without their work being saved. I also pointed out that our approach of wiring up these interactions in many places introduces complexity to the codebase which overall can impact the ability to work with the code.

[matthew-a-dunlap]

This branch is complete besides one minor issue due to how we calculate DataFile equality. We check if DataFile.id is equal, but because DataFiles can have null as their id this check will return true for different DataFiles. I could work around it but I think it is best to talk to @scolapasta on Monday if we could expand on the equals method.

[mheppler]

Comment by @jacksonokuhn in #4381

On the upload page, can we refer to the provenance as "Documents" rather than "Bundles"? We're storing the data as bundles on the backend, but they're gonna be uploaded as standard PROV-JSON docs. The current wording may confuse people.

[dlmurphy]

After discussing with Jackson, we're going with "Provenance File" as our terminology. I've updated the UI to reflect this, and I'm currently updating the documentation.

[matthew-a-dunlap]

There are a few "outstanding issues" related to the popup work:

• [x] The provenance popup rendering is not tied to any sort of set-up variable to inform it whether the prov system exists for the installation. There are open questions about whether the prov system will be more required or optional.
• [x] Saving provenance data should only be possible if the file in question did not have prov data in previous published version. Currently one can go and change the provenance after publication.
• [x] Provenance json api upload does not take into account provenance already existing.
• [x] Maybe move provenance save functionality to before ingest

These may require pulling the popup code back into development, but I think the code in https://github.com/IQSS/dataverse/pull/4422 should be reviewed as-is for other issues first.

[matthew-a-dunlap]

Had discussion with @scolapasta & @ekraffmiller around equality and entity objects today. Tomorrow I plan to move forward on those.

[matthew-a-dunlap]

I have rewritten functionality for the provenance popup to better handle issues around testing equality of datafiles. Also, provenance dropdown options and popup will only show up if a provenance address has been set via rest call. I plan to ask around tomorrow about how best to check if a datafile in a previous version had provenance and tie that into render logic.

I have turned off rendering of dropdown elements for tags/restrict/unrestrict/delete that were added to the editDataFiles.xhml on the branch. The work to make these functional is non-trivial, especially due to the sprawling nature of the file pages. Even with restrict/unrestrict/delete, where there are already methods to do an action based on selected checkboxes (and we are looking to allow a user to directly act on one file) I still feel this work should be handled in its own story. We have discussed that for #3488 we may want to refactor the tags functions, and a similar discussion should be had for restrict/unrestrict/delete.

[matthew-a-dunlap]

This is the current look of the provenance popup after the provenance json has been sent to CPL and can no longer be edited. If anything, we should probably use different wording for the freeform string @dlmurphy @mheppler

I was unsure whether any of the prov json options should be shown. I went with no because it seemed confusing to allow the user to preview the original prov file after there have been new additions. Let me know if I should take another approach.

[dlmurphy]

@mheppler and I have put together some screenshots depicting new messaging for the provenance popup, along with what it should look like in the use case @matthew-a-dunlap just mentioned. Please excuse any minor layout weirdness in these mockups, the only real difference here is the text. Also, the "User Guide" text should be a hyperlink to the Provenance section of the Dataset + File Management page of the user guide.

[dlmurphy]

Just collaborated with @TaniaSchlatter to rewrite the text in that yellow warning popup. Here's a mockup:

And here's the text for easy copy/paste:

Once you publish this dataset, your provenance file can not be edited or replaced.

Select "Cancel" to return the previous page, where you can preview your provenance file to confirm it is correct.

[jacksonokuhn]

I'm realizing that we need some more text on the upload screen :/ Basically we need to tell users that the title they give their datafile should be the same as the title it has in the prov they're uploading. The other option (which is way more of a pain in the butt) is to actually add another input box where they can give us the name of their datafile in the uploaded provenance. We need this in order to connect the prov getting uploaded to the prov in our system.

[matthew-a-dunlap]

@jacksonokuhn We can get the name of their datafile from the db and pass it in the rest call when we send the provenance to CPL. Is there some reason we need the user to specify it instead?

[sekmiller]

@jacksonokuhn - instead of relying on the name being the same, which could be problematic across datasets and installations, can we use the file DOI/PID?

[jacksonokuhn]

either way works actually, so long as we make it clear to the user

On Mon, Feb 12, 2018 at 11:50 AM, Stephen Kraffmiller < notifications@github.com> wrote:

@jacksonokuhn https://github.com/jacksonokuhn - instead of relying on the name being the same, which could be problematic across datasets and installations, can we use the file DOI/PID?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IQSS/dataverse/issues/4343#issuecomment-364985805, or mute the thread https://github.com/notifications/unsubscribe-auth/ANf1lMBWRoCngMBwyM5r-XFDqOPmlkcpks5tUGvxgaJpZM4Q1aPb .

[matthew-a-dunlap]

This is being pulled back into development as we need the user to provide us extra information for use when sending their upload prov.json to CPL. Specifically, we are unable to tell in many cases which file mentioned in the provenance json relates to the DataFile we are sending provenance for. This can often be inferred by comparing file names but if names have been changed for some reason the user should still be able to work with CPL.

To solve this, when the user uploads their provenance json we will mine the filenames out of the json and use them to help the user tell us which file relates to the DataFile they are on. This interaction will happen under the file upload dialog in the main popup. At a later date we may also show the user a preview of their provenance graph to assist them, but this enhancement can be done during/after work to interface with the visualization library in #4346.

There are some outstanding questions on this work:

• The number of file options presented to the user may be very large, too large for a dropdown. How are we going to help the user with this selection? An autocomplete text box has been suggested.
• If the names match, it would be best to present that option to the user first. Is this a hard requirement? If so, how do we do this alongside the selection process above.
• UI interaction is one thing, but how do we do this via command line? The easiest option is to require API users to provide the correct name along with their json in the call.

[matthew-a-dunlap]

ToDo list to tackle the json parsing, dropdown work, and code review from before this was pulled back into develop.

UI:

• [x] Make dropdown required if file is uploaded. Currently throws error to main page after popup saved instead of stoping saving when button clicked.
• [x] Add dropdown label
• [x] Needed: Dropdown description text and error message for not selecting an element from the dropdown.
• [x] If a user uploads a json file, removes it and uploads a different one the UI does not show the 2nd uploaded file, even though the upload is successful behind the scenes. For some reason the second time through it looks like the update in fileUpload either breaks silently or or happens before the upload is finished.
  • May be related to miuse of the EntityManager
• [x] If a user attempts to remove a json file without selecting the bundleNode, a error message is shown and remove does not take place
• [x] Representation of the selected dropdown entity can be improved. At least, if there is no name it should not be blank.
• [x] Hitting enter in the prov node search on the file page triggers file delete. May also happen in other places.
• [x] Popup UI update weird. Can be seen by uploading a broken prov file, closing the popup and reopening.
• [x] UI changes from review, including dropdown sorting (reverse alphabetical based on entity name)

Behind:

• [x] Solve bug with calling two commands that both call ctxt.files().save(dataFile), with the second one erroring saying "The object cannot be merged because it has changed or been deleted since it was last read."
• [x] Solve other context bug with save on some pages causing duplicates of the DataFile.
• [x] Fix ProvIT tests now that storing ProvJson requires extra user input
  • Also now that the ProvJson command returns a datafile not the json. The API endpoint can still return json if we want.
• [x] Fix fragment tests, maybe write more as well
• [x] Ensure that the json string variable being thrown around in the fragmentbean always has the content expected.
• [x] Fix api calls to make sure they use the call that saves the context
  • Need to make sure saving works along with other edits at same save
• [x] After deleting json in file page and saving, reopening the popup before refresh causes null pointer (does not break interaction)
  • This is because the context is bad after saving... I need to understand EntityManager better to solve multiple save issues
• [x] Some json will cause the parsing code to seize up and not complete, causing the ui to not finish updating (prov-sample.json)
• [x] Tabular ingest may be broken. "buttons were locked, but there was no ingest banner for the file, just a download btn" (Pretty sure this was broken do to json parsing issues that are fixed)
• [x] Create a method for actually returning search results. The autocomplete should return a subset of the entries depending on the user inputted string.
• [x] If an error happens during json upload and parsing, inform user after rollback.
• [x] Make doubly sure everything works from the create dataset page, that page has not been used heavily during the development.
• [x] Code/comment cleanup

[matthew-a-dunlap]

The current state of affairs (before my long weekend): No code has been started for this story but after discussing the need in our meeting yesterday I investigated the use of an autocomplete dropdown for the task. The code for the autocomplete in permissions was able to copy-pasted into the popup fragment cleanly and is a good starting point. The next step is understand the w3c provenance spec a tiny bit better and parse the json. The basic need is to parse all the sub-tags to the activity tag and present those as options using the name tag inside each sub-tag if it exists ("*:name" to be specific, for example "rdt:name"). I was also going to check to see if the DataFile Name matches the sub-tag name and present it first in the suggestion list.

With this information in hand, the last piece will be to store the needed name to be sent off at pubish. I'm not quite sure how we should save this, and I half think the best option is to store it as part of the prov.json filename just to keep them bundled together.

There is still an open question of what to do if two nodes share the same name. My idea for an approach was present them both as options, but maybe try to parse out some other identifying piece of info. Doing this in a way that is useful generically is likely challenging. Aside from this edge case I do not see this work as taking that much time, tho I have not had to do much json parsing in java (I have done xml parsing and remember dealing with those libraries to be more painful than I expected).

[matthew-a-dunlap]

@mheppler This is ready for you to look over. There is remaining work, but its either in the backend or API work. The one exception to this is the UI issue related to upload => remove => upload again. I'm pretty stumped on that one.

[matthew-a-dunlap]

@mheppler @dlmurphy Here is the error dialog that happens when there is an error during uploading and parsing of the provenance json. Let me know what you think of the wording. We will also need another error (or warning?) for when the provenance uploaded by the user does not have any entities (files) that can be parsed out for the user to select as connected to their DataFile.

[matthew-a-dunlap]

Here is a sample of the provenance autocomplete for selecting the entity @dlmurphy @jacksonokuhn

[matthew-a-dunlap]

All that looks to be left is fixing the IT tests and maybe writing another test or two for the parser code!

[matthew-a-dunlap]

Deployment notes:

• Run db updates (4.8.5 to 4.9)
• Make sure this system setting is in place (The documentation is is #4364). curl -X PUT -d 'http://localhost:7777' http://localhost:8080/api/admin/settings/:ProvServiceUrl
  • The url contents do not matter for this branch, just its existence is needed to enable provenance

Testing notes:

• Provenance popup can be accessed from these locations: Dataset creation, file upload, edit files (off main page), file page. These all need to be tested, especially file page as its flow is the most different.
• Its possible there is an edge case I've missed when adding and removing the prov json from the popup. Test adding, removing and readding while the popup is up, or adding, removing, closing and reopening. Etc etc.
• You will need prov json examples with entities in them. I pulled mine from https://github.com/CamFlow/cytoscape.js-prov/tree/master/json (elisabeth & example look good)