Replace BrowseEverything

[tpendragon]

We're running off of a branch which is a major version that may never get released, and the strategy of copying files locally during the browser session is resulting in being unable to upload large files from the Isilon.

Look at this when we remove it: https://github.com/pulibrary/figgy/pull/5315/files#diff-c1fd91cb1911a0512578b99f657554526f3e1421decdb9e908712beab57e10f9R61-R63

[hackartisan]

Implications of this change:

• Users will no longer be able to ingest from Google Drive. We aren't aware of anyone ingesting this way any longer.
• We'll need an alternate implementation for fetching server directory locations / contents

[hackartisan]

This issue blocks migration away from webpacker because of an implementation detail in the browse everything react code

[hackartisan]

A first step might be removing the google drive feature.

[hackartisan]

Datadog keeps logs for 2 weeks. There are no logs showing an attempt to use google drive in the past 2 weeks: https://app.datadoghq.com/logs?query=service%3Afiggy%20%20%2Fbrowse%2Fproviders%2Fgoogle_drive%2Fauthorize&cols=host%2Cservice&index=&messageDisplay=inline&stream_sort=time%2Cdesc&viz=stream&from_ts=1663699135003&to_ts=1666291135003&live=true (vs file system: https://app.datadoghq.com/logs?query=service%3Afiggy%20%2Fbrowse%2Fproviders%2Ffast_file_system%2Fauthorize&cols=host%2Cservice&index=&messageDisplay=inline&stream_sort=time%2Cdesc&viz=stream&from_ts=1663699135003&to_ts=1666291135003&live=true). Note that this path, I believe, shows that someone selected that file source from the dropdown, not that they necessarily uploaded something with it. But it's the best I could find.

We lost the google analytics tracker in early August, but even when it was in place I don't think these browse everything routes render the layout so they don't get logged. I looked at may-june and I'm not seeing any stats on /browse.

I don't really think 2 weeks is long enough to conclude the feature's not in use.

[hackartisan]

@kelea99 says we can go forward with removing google drive. It has never worked very well and she's not aware of anyone using this ingest workflow. Kim will announce in the digital_library channel.

[hackartisan]

Replacements proposed:

• Uppy, Shrine
  • pros
    • we could also move away from dropzone and have a more consistent experience
  • cons
    • These are both fully-featured upload tools; they're going to provide a lot more than we need.
    • neither has a plugin for selecting a file that's already on the server.
    • even if we write our own plugin it will probably be much more complex logic than if we write something specifically to our own use case
• Vue finder, tool mentioned during structure manager design discussion
  • pros
    • simple tool, feature set matches our needs pretty well -- show files, navigate them, some event handling
    • small, well-organized code base
    • good documentation
    • some keyboard navigation possible
  • cons
    • single developer, however they are currently actively engaged
    • doesn't yet support vue 3, although there is a pre-release
• jstree
  • pros
    • This is the tool we use for pulfalight, so some consistency in dependency choice
    • Similar to the current UI, some consistency for users
    • has a fuzzy search feature
  • cons
    • a jquery plugin is probably not as good a fit as a vue component
    • solo developer
    • more features than we likely need

[hackartisan]

re: vue finder, this looks relevant: https://github.com/jledentu/vue-finder/issues/190

[hackartisan]

Looking more at https://vue-finder.netlify.app/examples.html#basic-example. Maybe we can use the expand event instead of the selected event. The expand event does emit on leaf nodes. We still want to be able to select multiple files, so we would enable select on leaf nodes.

We could combine this with an info pane that autofills to ensure user has clarity on what they're about to ingest. Also use multiple hidden inputs so multiple file selection submits as an array.

Alternately. What if we write our own thing. We don't have a dependency on an external library that could cause us update headaches down the road. What if, for every node, you could define the label, who the children are, and whether it's selectable. This means ruby can say "this one will not ingest correctly"

Pros of writing our own thing:

• We can enforce element-by-element selectability
• We can ensure it's fully accessible
• We can change the UI based on user feedback without going back to find a new tool.
• Generally, it will serve our use cases and only our use cases.
• Trey has already done some proof of concept work.

decision: use our own vue component.

UI decisions still TBD. We know that the current UI has caused confusion and errors:

• In bulk ingest UI, I pick a directory that's not valid for ingest
• I chose a folder in the file manager. what's about to happen?

Some UI options we've discussed:

• keeping a tree view, making it look more like the structure manager mockups
• Using the horizontal view like a mac / Vue Finder -- may reduce the number of little mouse targets
• Having an info pane to summarize what's will happen on submit.

Particular UI considerations:

• How to make a directory expandable but not selectable.
• How to select only one directory
• How to allow multiple select on files (within a single directory?)

[hackartisan]

Mockups:

• simpler mockup with most updated visual elements: https://docs.google.com/drawings/d/1tq2JOyoO4_xf_ORGUf39tByk9_0VOUpO0GECC-tyD-Y/edit
• More comments on this mockup: https://docs.google.com/drawings/d/1oa_CxDqRZK4jlkPJA9x3uyolQUYkdBTlF_p9hPlB5hM/edit

Implementation notes:

• We need a concept of "list-focus" that's different from "selected" or "expanded"
• We need both "selectable" and "expandable." Figgy will mark every directory as "expandable."
• We need to figure out how to get the data from the modal into the form
  • Use a 4th component that lives on the page and renders the modal and the other components, populates the values when the modal is closed.
• We need 3 vue components, one container to hold the data and other two, then the directory pane and the list pane.
  • parent pane holds which directory is the "list-focus" item, when that data variable has a value the list pane populates.
  • initializes the directory pane with everything that's expandable.
  • In the directory pane, if you click an item it sends a list-focus event to the container.
  • If you click "select all" the list pane fires a selection event.
  • If you click a directory name in the left does it expand it?

All data is held in the root/container. The child components fire events up and the container component manages the data updates.

Some outstanding questions:

• If I click the title of a directory, does it expand or just list-focus?
  • in drive, it does not expand. just list-focus.
• If I expand a directory, does it also list-focus that directory?
  • in drive, no.

[hackartisan]

ruby side: don't provide hidden files via api endpoint.

[tpendragon]

Current state:

Most of above is done. I did a regular JSON endpoint instead of GraphQL because it was just really, really easy. Bulk ingest is fully replaced.

There's a lot of styling that could be done and the file ingest portion needs done, which is the most complicated part.