Update how ML handles dates (fixes date sorting)

[clarkepeterf]

Problem Description: Date facets sort in the wrong order because they are sorted as strings. Strings require more conversion to/from epoch times in both ML and the frontend.

Example of bad sorting Expected Behavior/Solution: Dates are sorted in the correct order

Marklogic will continue to return dates in string format. Full ISO String for years [-9999:9999]. Just the year outside of that range.

Agreed to the following on 2024-08-14: - MarkLogic will return date facets as epoch dates in seconds before/after 1970-01-01 - MarkLogic will require date search fields to use seconds before/after 1970-01-01 - Frontend will support full dates in the years -9999 - 9999 - Frontend will support just years outside of the range -9999 - 9999

Requirements: List of details required for the completion of the issue or requirements for the feature/bug. This can also include requirements that lie outside of the teams such as new design docs or clarification from an outside source.

Needed for promotion: If an item on the list is not needed, it should be crossed off but not removed.

• [ ] ~Wireframe/Mockup - Mike~
• [ ] Committee discussions - Sarah
• [ ] Feasibility/Team discussion - Sarah
• [ ] Backend requirements - TBD
• [ ] Frontend requirements- TBD
• [ ] Are new regression tests required for QA - Amy
• [ ] Questions
• List of questions for discussions. Answers should be documented within the issue.

UAT/LUX Examples:

• Bad sorting (old way in PRD): https://lux.collections.yale.edu/api/facets/agent?q=%7B%22AND%22%3A%5B%7B%22recordType%22%3A%22person%22%7D%2C%7B%22startDate%22%3A%22-1%22%2C%22_comp%22%3A%22%3C%22%7D%5D%7D&name=agentStartDate&sort=desc&page=1
  • This is bad for two reasons:
    1. The dates are supposed to be sorted in descending order. This means getting smaller (more negative). However for the most part they are getting larger (less negative). (Look at the Values under the OrderedItems)
    2. Dates show up out of order (e.g. goes from year -55 to -521 to -50)
• Good sorting (new way in DEV): https://lux-front-dev.collections.yale.edu/api/facets/agent?q=%7B%22AND%22%3A%5B%7B%22recordType%22%3A%22person%22%7D%2C%7B%22startDate%22%3A%22-000001%22%2C%22_comp%22%3A%22%3C%22%7D%5D%7D&name=agentStartDate&sort=desc&page=1
  • This fixes the above problems:
    1. Dates are in descending order (getting more negative)
    2. No dates show up out of order

Dependencies/Blocks: N/A

Related Github Issues: N/A

Related links: N/A

Wireframe/Mockup: Place wireframe/mockup for the proposed solution at end of ticket.

[brent-hartwig]

@clarkepeterf, consider naming the endpoint more generic (/range) and initially only supporting date indexes. Consumer to specify the same name parameter value as the /facets endpoint. Later on, this may expand into providing the range of other indexes, numeric or otherwise.

[clarkepeterf]

DIscussed with @kamerynB - we will continue with just using the /facets endpoint, no /range endpoint. The range can be achieved using two parallel facet requests for sort=asc&page=1&pageLength=1 and sort=desc&page=1&pageLength=1.

Date facets will switch to return epoch (float) times instead of the string dates

[clarkepeterf]

@kamerynB is it easier for you to pass in the epoch time when using search fields? If so, I can set it up to work that way. Let me know.

[clarkepeterf]

@roamye this will have to be done next sprint, as it needs to be made in line with https://github.com/project-lux/lux-frontend/issues/340

[roamye]

@clarkepeterf - this also skipped the UAT process so that work out! moving it back to prioritization review.

[clarkepeterf]

@roamye ok - I already have changes related to this issue pushed to DEV

[roamye]

From ITS Team Meeting 8/23: It would be easier for ML to pass ISO Strings. This will work for the FE as the FE will also pass the ISO strings into ML.

There is a challenge with preceding extra 0's. Which @kamerynB will look into. This could potentially be affected by dinosaur dates. Need input from @azaroth42

From TF meeting 8/26: This was moved to backlog as it now has UAT examples.

[roamye]

Approved by UAT 9/12

[roamye]

Looks good in prod. Closing.