Add support for unit-specific configuration files

[brent-hartwig]

Problem Description: As part of #73, we will need the ability to create and serve up unit-specific configuration files, specifically for search terms, related lists, and advanced search. This is under the premise that not all search terms (which impact all of the aforementioned configuration files) will apply to all units. Take PhysicalFeature for example: only (full) LUX and YPM will care about it.

Expected Behavior/Solution:

• [ ] Introduce means to restrict search terms (if not also search scopes*) by unit.
• [ ] Modify the remaining search term generator to apply unit-specific settings and produce unit-specific output.
• [ ] Given some search terms are generated from facetsConfig.mjs, TBD whether we also need to restrict those by unit. This may be the tipping point to fold facetsConfig.mjs into searchTermConfig.mjs.
• [x] Modify remaining generators to create unit-specific output that is restricted to unit-specific search term configuration.
• [x] Modify endpoints to use the correct configuration.

Scope includes being able to restrict search terms by unit but not identifying which ones to omit by unit.

* Anticipated to be straightforward within the search term configuration file and generated outputs. TBD if searchScope.mjs needs to become unit-aware.

Requirements: See above

Needed for promotion: If an item on the list is not needed, it should be crossed off but not removed.

~- [ ] Wireframe/Mockup - Mike~ ~- [ ] Committee discussions - Sarah~ Approved by Jeff on 26 Jun 24. ~- [ ] Feasibility/Team discussion - Sarah~ No known technical challenges.

• [x] Backend requirements - See above.
• [x] Frontend requirements - Only submit search criteria that is valid for the unit the portal is for (e.g., a YCBA portal should not attempt to search for PhysicalFeatures.
• [x] Middle tier requirements:
  • Use the correct service account.
  • Only submit search criteria that is valid for the service account being used.
• [ ] Are new regression tests required for QA - Amy
• [ ] Questions
• List of questions for discussions. Answers should be documented within the issue.

UAT/LUX Examples:

• By unit, demonstrate the ability to return different advanced search configuration from /ds/lux/advancedSearchConfig.mjs.
• By unit, demonstrate the ability to apply different related list configurations via /ds/lux/relatedList.mjs.
• By unit, demonstrate the ability to return different search information from /ds/lux/searchInfo.mjs.
• By unit, acceptable search criteria will vary between units when their search term configuration varies. Endpoints include:
  • /ds/lux/facets.mjs
  • /ds/lux/search.mjs
  • /ds/lux/searchEstimate.mjs
  • /ds/lux/searchWillMatch.mjs

It is the responsibility of the frontend and middle tier to only submit search criteria that is valid for the associated unit; invalid search criteria will result in the endpoint returning a 400 status code, meaning "bad request".

Dependencies/Blocks:

This ticket depends on the security portion of #73 which has already reached production.

This ticket does not presently block other tickets.

Related Github Issues:

73 is the main data slice / unit portal mostly research plus security implementation ticket.

Related links: YPM Portal REQ: https://docs.google.com/spreadsheets/d/1eXi-jiBs9cokZF4yQ8z9XmTs3YLdoyZ8nVBTOEEQI7s/edit?gid=698332853#gid=698332853

Wireframe/Mockup:

N/A

[brent-hartwig]

PRs:

• https://github.com/project-lux/lux-marklogic/pull/279
• https://git.yale.edu/lux-its/ml-cluster-formation/pull/49

[brent-hartwig]

@roamye and @clarkepeterf, I believe we have some choices when it comes to this ticket's UAT. The most involved that comes to mind would have us:

1. Deploy to DEV's LUX by Unit with:
   • endpointAccessUnitNames=ypm
   • Modify searchTermsConfig.mjs as Peter and I did when we tested.
2. Ensure https://lux-byunit2.collections.yale.edu/ is still wired up to the LUX by Unit tenant and uses YPM's service account.
3. Verify advanced search aligns with edits made to searchTermsConfig.mjs.
4. Verify excluded (in searchTermsConfig.mjs) related lists on entity pages do not appear.
5. Ensure https://lux-byunit1.collections.yale.edu/ is still wired up to the LUX by Unit tenant and uses the main LUX service account. Or deploy to and use https://lux-front-dev.collections.yale.edu/.
6. Verify everything is business as normal when using the main LUX service account.

At this point, I do not know of a list of search scopes or search terms that we should exclude from YPM. Should such lists exist, we could easily make those configuration changes as part of this or another ticket. At that point, I think it will become important to determine and set frontend expectations with YPM while they continue to use a copy of LUX's frontend and middle tier, such as:

1. If a search scope is excluded, I would expect a) the search tab to still be present, b) simple search to fail, and c) do not know how the frontend would react to not having advanced search configuration for the tab's scope.
2. If a related list is excluded, would the control never appear, would an error reach the user, or something else?
3. If excluded search terms would have otherwise been part of a related list's configuration, the associated relationships would not longer appear.
4. If an entity page's HAL links reference an excluded search term, we can expect those search inquiries to fail.

I would expect the frontend and middle tier YPM develops to account for the above, but that we'll be in a state of limbo between excluding any search scopes or terms and YPM developing those tiers.

I should add that the backend may have the option to handle invalid search requests more gracefully, such as removing invalid search criteria.

cc: @azaroth42, @jffcamp, @prowns

[clarkepeterf]

@prowns @jffcamp Do we know what the requirements are around excluding certain scopes/fields or making specialized fields for unit slices?

This will help to understand what functionality should be supported in ML, the middletier, and the frontend.

[prowns]

That list has not yet been developed.

cc: @roamye, @garymotz

[clarkepeterf]

Things to consider:

• Are there search fields/scopes that will be used for slices that are not used for the main lux?
• Are there search fields/scopes that will be not be used for slices that are used for the main lux?
• If an entire search scope is not used for a slice - should the frontend hide that tab?
• If a search field/scope is disabled in a slice, should HAL links that use that field be disabled?

[roamye]

@brent-hartwig - couple of questions, the requirements listed above for the FE/MT, specifically related to the ones about searches, are worded in the way of a test case.

Are what is listed intended to be verified? or is "submitting search criteria" defined as something the FE/MT will push into the unit portal/service account? Such that, it is required for them to submit it in order for the process to be complete?

[roamye]

@clarkepeterf - I listed your consideration points, under the tab "Categories" in this spreadsheet here: https://docs.google.com/spreadsheets/d/1eXi-jiBs9cokZF4yQ8z9XmTs3YLdoyZ8nVBTOEEQI7s/edit?gid=698332853#gid=698332853

I also listed @brent-hartwig questions/statements from this comment into that tabs.

Requirements listed within the issue have been placed into the Requirements tab of the spreadsheet and UAT information has been listed under the Test Cases tab.

I have attached the link to the spreadsheet under the related links section of this issue as well.

We can continue developing/modifying as further discussion is had.

cc: @prowns, @garymotz

[brent-hartwig]

@roamye, how the frontend and middle tier react to configuration dropped for specific units is not in scope of any ticket yet. My concern is setting expectations with YPM if a) they were to continue their testing using a vanilla frontend and middle tier and b) they ask us to exclude search scopes or terms. As for testing this specific ticket, I offered an approach --one that could create distractions because we do not yet know how the frontend and middle tier will react to excluded configuration. One thing is for sure: everything should be business as usual when a middle tier authenticates into MarkLogic with a "full" LUX service account.

[roamye]

I am unsure how to verify the UAt examples listed above in the issue & from this comment: https://github.com/project-lux/lux-marklogic/issues/277#issuecomment-2296865989. Is there some place I can go to verify the UAT? Or is this something only you two @clarkepeterf & @brent-hartwig can verify?

[brent-hartwig]

@roamye and @clarkepeterf, I can verify this one. I'll temporarily created a YPM service account in Blue to ensure LUX and YPM are both receiving configurations.

[brent-hartwig]

The base roles that #227 removed the ML Gradle files for are still in DEV and Blue. The base-endpoint-consumer role's presence is in conflict with #277's implementation and needs to be removed. If/when all DEV tenants receive Backend v1.21 or later, the base roles may be deleted from DEV. I have asked Jeff and Sarah if I may do so now in Blue.

@clarkepeterf, we also have the option to go back to the initial implementation that split on dash and required at least four values in the array. It would have avoided this particular conflict, but could still be prone to others.

[brent-hartwig]

Brent to finish verification in Blue after Blue switches from PROD to TST, which is presently scheduled for 9 Sep 24.

[roamye]

This ticket will remain open per the teams thread comments here and here.

[brent-hartwig]

Verified in Blue today after deploying release1.24. Closing.