Usability issues when both Controlled Vocabulary and Configurable Entities enabled on single field (e.g. Author)

[tdonohue]

Describe the bug First reported in https://github.com/DSpace/dspace-angular/pull/751#pullrequestreview-476572310

If you enable Controlled Vocabulary (i.e. author-based authority control) and Entities on the same Author field, you'll get a view that looks like this

The field is extremely confusing to interact with, as it has too many buttons, and the meaning behind the buttons is entirely unclear.

Expected behavior

1. It should be possible to enable controlled vocabulary and Entities on an Author field , as it allows you to more easily add Authors that either (a) are already added to other Items (i.e. via authority control indexing of dc.contributor.author), or (b) have an Author Entity (and "profile page") in the system.
2. This expected behavior was previously discussed (with proposed mockups) in the Entities Working Group:
   1. Initially discussed in meeting on Aug 27, 2019. See notes and comments on that page
      1. Mock-ups were initial presented (see pages 13-18 of that PDF specifically for Authority Control + Entities on Author field)
         2. Follow-up discussions occurred in the meeting on Sept 17, 2019. In that meeting, the Entities WG clarified that the mockups were not final, but the final solution should allow for both to be enabled on a single field (if a site chose to enable both).

The solution to this ticket should allow for both Controlled Vocabulary and Entities to be enabled on an Author field. Ideally, both searches should appear in the same pop-up modal window. This would ideally mean there's a single (shared) "Lookup" button which is able to search across (a) just Controlled Vocabulary, (b) just Entities, or (c) both.

UPDATE: In 7.0 beta 5, we applied a workaround to this issue by switching the Author submission field to the onebox type. See https://github.com/DSpace/DSpace/pull/3115 This is a valid workaround because the usability issue ONLY occurs when using the name type for the Author field. As noted in comments below, there is some question about whether Authors should just be entered in a onebox at all times. Therefore I've moved this to 7.2 for further analysis/discussion...at that point, we may want to determine whether the name type should be permanently removed, or if this issue ticket should be fixed.

[artlowel]

Current state

The current behavior for an authority controlled field is:

• you can type in a value and click the plus button to add it as plain metadata
• if you click search instead you get a dropdown, showing local authority matches
• If you select a match and click edit, you can change the value shown for the item being submitted, while keeping the same authority key

The current behavior for a relationship field is:

• you can type in a value and click the plus button to add it as plain metadata
• if you click lookup instead you get a modal window, showing local entity matches in the first tab, and matches in external sources in the others
• if you select a match, and name variants are enabled for the field, you can change the value shown for the item being submitted while keeping the relationship with the entity

These goals both are trying to achieve are nearly identical. The dropdown used currently for authority control has the advantage of being less disruptive than the modal. Although I think it could be improved by removing the need to click a button first, and just starting the search after the user has typed a few characters. That way, submitters unaware that something they're entering might be in the local authority would still see suggestions.

The modal has the advantage that it searches across many sources so if you don't find what you're searching for as a local entity, you'll see results in ORCID or Pubmed as well and import it from there. I also prefer the combobox to enter a new name variant or choose from existing ones to the process with the edit button that authority control uses. It wasn't clear to me what the purpose of the edit button was at first

Proposal

I propose we combine the advantages:

We add the best match from local authority and local entity results to the dropdown, it will become search-as-you-type, no need for a button. If there are results in other external sources, the final result in the dropdown will be e.g. "51 results in ORCID, 3 results in Pubmed", where if you select it you open the lookup-modal with the query pre-filled and on the right tab

We'll also add a local authority tab to the lookup-modal, that will allow you to search for, select and add name variants for authority controlled fields in exactly the same way it works for related entities and external sources.

I made the mockup below last year, for one of the entities working-group meetings, as mentioned by @tdonohue above:

The name variant combobox will be added to selected entity and authority control results straight in the submission form as well, to make it easier to change the name variant after you've selected it.

All of this can be implemented in an estimated 48h

[tdonohue]

@artlowel : The overall concept here sounds good to me. However, I'm slightly worried about the performance of the "External Sources" selection in that search-as-you-type mockup. It seems (to me) like searching those external sources is unlikely to be able to keep up with 'as you type' searching -- it's very possible you'll send off the next search before all the previous ones return.

However, I think that's easily solvable by simply not showing dynamic results in that selection. Essentially, you end up with three options: "Local Authority" (which searches as you type), "Entities" (also searches as you type), and "Or, search external sources" (which shows no results, but lets you know that you can click there if you do not see a local authority or entity that matches the value you are typing).

In other words, I think the search-as-you-type is a great option for local-based searches, but not great for external searches (which are highly dependent on performance of those external APIs). So, we may want to treat these differently & make external sources only searchable from the popup modal.

I'll keep the 48hours estimate for now (unless you feel it's drastically simplified by this change). But, I'd recommend we consider rethinking that "External Sources" in that search-as-you-type mockup.

[tdonohue]

In today's meeting, @atarix83 mentioned it might be possible to update configuration in 7.x to simply avoid this issue by configuring the Controlled Vocabulary settings to not display the Lookup button, but instead use a "onebox". (@atarix83 could you add an example here?)

In the same meeting, the Atmire team noted this might be worth keeping in Beta 5 as the usability of Controlled Vocabulary & Entities needs improvement anyways -- the buttons are confusing, especially the "Edit" button. So, delaying this may be confusing to users & we should consider at least fixing some of the known usability issues.

[atarix83]

@tdonohue my suggestion to temporary resolve the issue with lookup field with authority control, is to configure of the relation-field in the submission-forms-.xml file by adding the type onebox as the following example

                <relation-field>
                    <relationship-type>isAuthorOfPublication</relationship-type>
                    <search-configuration>person</search-configuration>
                    <repeatable>true</repeatable>
                    <label>Author</label>
                    <hint>Add an author</hint>
                    <linked-metadata-field>
                        <dc-schema>dc</dc-schema>
                        <dc-element>contributor</dc-element>
                        <dc-qualifier>author</dc-qualifier>
                        <input-type>onebox</input-type>
                    </linked-metadata-field>
                    <required>At least one author (plain text or relationship) is required</required>
                </relation-field> 

In this way this is the result during the submission :

[hardyoyo]

I like the onebox approach, because it very elegantly skips past the classic issue documented by this notorious posting: https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/ As long as we can document how to configure validation of the field, I think this is a good solution.

[abollini]

@tdonohue the code for the regex validation seems to be inplace at least on the rest side, see https://github.com/DSpace/DSpace/blob/main/dspace-api/src/main/java/org/dspace/app/util/DCInput.java#L209

https://github.com/DSpace/DSpace/blob/main/dspace-server-webapp/src/main/java/org/dspace/app/rest/submit/step/validation/MetadataValidation.java#L76

https://github.com/DSpace/DSpace/blob/main/dspace-api/src/main/java/org/dspace/app/util/DCInput.java#L532

it should be enough to add a <regex>.+, .*</regex> in the submission-form.xml https://github.com/DSpace/DSpace/blob/main/dspace/config/submission-forms.xml#L61 if we want to force to have a value with a comma followed by a space in it.

[atarix83]

I can confirm it works also on agular side, for example the following configuration works

                <field>
                    <dc-schema>dc</dc-schema>
                    <dc-element>identifier</dc-element>
                    <dc-qualifier>citation</dc-qualifier>
                    <repeatable>false</repeatable>
                    <label>Citation</label>
                    <input-type>onebox</input-type>
                    <hint>Enter the standard citation for the previously issued instance of this item.</hint>
                    <required></required>
                    <regex>\w+(, )+\w+</regex>
                </field>

[abollini]

BTW I have just checked that Zenodo and Dataverse both follow the approach "onebox" using an hint to suggest to write Family name, given names or organisation, see the screens below Zenodo:

Dataverse:

[tdonohue]

@abollini : thanks for adding those screenshots. Those help prove that the approach of a single author textbox (with validation obviously) is a reasonable solution to this problem.

[tdonohue]

I've created a small PR which provides an example of how we could re-configure this Author field to simply be a onebox instead of a name field, therefore bypassing the most severe usability issues. The PR includes an example <regex> validation configuration as well, but is commented out by default: https://github.com/DSpace/DSpace/pull/3115

[tdonohue]

UPDATE: In 7.0 beta 5, we applied a workaround to this issue by switching the Author submission field to the onebox type. See https://github.com/DSpace/DSpace/pull/3115 This is a valid workaround because the usability issue ONLY occurs when using the name type for the Author field. As noted in comments above, there is some question about whether Authors should just be entered in a onebox at all times. Therefore I've moved this to 7.2 for further analysis/discussion...at that point, we may want to determine whether the name type should be permanently removed, or if this issue ticket should be fixed.

(I've copied this comment into the description above as well to make it easier to find)