New Way To Handle Data Constants (from 1156)

[gigamorph]

Problem Description: Generating data constants will allow us to have regular dataset updates. Additionally the front end still needs IRI's, which a data constant function can translate.

Expected Behavior/Solution: Data Constants will allow regular data set updates and fix the IRI issue.

NOTE: This ticket is dependent on the work defined in #55

Requirements:

• Concepts (LUX URI) mapped to Getty AAT
• AAT ids more stable than LUX URI
• Process needs to be changed to reference AAT directly

Needed for promotion: If an item on the list is not needed, it should be crossed off but not removed.

[ ] Wireframe/Mockup - Mike

• [x] Committee discussions - Sarah
• [x] Feasibility/Team discussion - Sarah
• [ ] ~Backend requirements - TBD~
• [ ] ~Frontend requirements- TBD~
• ~[ ] Questions~
• ~List of questions for discussions. Answers should be documented within the issue.~

UAT/LUX Examples:

• Example endpoints from LUX to use for development and testing, if applicable.

Dependencies/Blocks:

• Blocked By: ~new data set~.
• ~Dependent on: #55~ Blocking:
• https://github.com/project-lux/lux-middletier/issues/18 , which is blocking
• https://github.com/project-lux/lux-frontend/issues/102

Related Github Issues:

• Issues that contain similar work but are not blocking or being blocked by the current issue.

Related links:

• Associated Teams conversation

Wireframe/Mockup: Place wireframe/mockup for the proposed solution at end of ticket.

[jffcamp]

@clarkepeterf - Requires front-end ticket Requires middle-tier ticket

[roamye]

this issue needs to be updated to include AAT rather than data constants. (once the new data set is available). blocked by new data set. will hold off on updating/working on issue till then.

[brent-hartwig]

WRT the backend...

The database index configuration would need to be updated to use the AAT values for data constants, starting in the online spreadsheet. Changing to AAT values is expected to reduce the need to check and update the index configuration when the dataset changes.

But the backend also generates and relies on /lib/dataConstants.mjs. It includes some data constants provided by the data pipeline (pipelineDataConstants.json), some data constants derived by querying the data (queries), and accessor functions that other backend code uses. The primary use is within model.mjs (e.g., get the primary name from a document.)

Data constants that can have an AAT value should be updated. That'll probably be true for all of the ones presently provided by the data pipeline as well as at least 21 constants derived via query. Whether we end up with a manually maintained dataConstants.mjs file or not is TBD.

facetsLib.mjs uses a different kind of data constant when deciding whether to reject a facet request: the number of values in an index. Index value counts became data constants as it can take seconds to determine the number of values in large indexes --we didn't want to add that to a facet request's time. I do not believe this is grounds to continue updating the data constants when the data changes --they would play catch up the next time the backend code is deployed. If we wanted to replace the generator with a manually maintained dataConstants.mjs file, we could come up with a way to make the facet determination without incorporating index value counts.

After the AAT change, would we still need the data constants endpoint?

I do not see anything else unique in the backend's other uses of data constants.

[roamye]

Per conversation in FE#102 https://github.com/project-lux/lux-frontend/issues/102#issuecomment-2047927696 this ticket can be unblocked as the data set is no longer a blocker.

[azaroth42]

Propose critical

[roamye]

From IT meeting notes this ticket will need to be rescoped to include the use of AATs rather than data constants.

[clarkepeterf]

@azaroth42 @kkdavis14

Sort Names don't have Equivalent AATs in their referencing documents. This will prevent completion of ML 54

Example: West Virginia has a name classified as Sort Name (https://lux.collections.yale.edu/data/concept/31497b4e-24ad-47fe-88ad-af2007d7fb5a), but there is no equivalent given.

[clarkepeterf]

Data unable to be updated this release, moving to next release

[roamye]

@clarkepeterf can you please find any applicable UAT links for this ticket? thank you.

[roamye]

@clarkepeterf @kkdavis14

is this record a possible UAT link for this issue: https://lux-front-tst.collections.yale.edu/view/visual/b50fb93f-dc1d-4e49-a3b8-acc1cfd4a174 from BH: https://www.bugherd.com/projects/284041/tasks/2297 ?

Note: record fails :/

[roamye]

Approved by UAT

Note: UAT is unreviewable.