Call number browse is broken

[ahafele]

Unsure if it is a local problem or community issue.

Relevant slack convos - https://folio-project.slack.com/archives/C20V5L40P/p1694096746131329?thread_ts=1694086346.601559&channel=C20V5L40P&message_ts=1694096746.131329

https://folio-project.slack.com/archives/C20V5L40P/p1655823670836369

There is a summary doc here. For certain call number ranges the browse just seems to stop. We know the records exist (because a call number search finds them) but ranges of them seem to be missing from the call number browse. I don’t know if it is somehow related to how the optimization works or perhaps similar to this orchid bug (see last comment).

In the meantime, staff are having to use SearchWorks for shelflisting which is not optimal. https://docs.google.com/document/d/1P8We-Ts773hORt-rp9MCFClzAUH-kIIWr_8y668sZKA/edit?usp=sharing

[shelleydoljack]

I turned on query logging in folio dev to see what the queries are that mod-search sends to ES and correlated the responses to those queries from port-forwarding ES service and posting the queries individually to ES. I can't really make out what's going on. In the case for the browse query reported in the google doc, the response from ES was a script error (maybe because dev doesn't have the record?):

{
  "error": {
    "root_cause": [
      {
        "type": "script_exception",
        "reason": "compile error",
        "script_stack": [
          "def f=doc[itemEffectiveShelvingOrde ...",
          "          ^---- HERE"
        ],
        "script": "def f=doc[itemEffectiveShelvingOrder];def a=Collections.binarySearch(f,params[cn]);if(a>=0) return f[a];a=-a-1;f[(int)Math.min(Math.max(0, a),f.length-1)]",
        "lang": "painless",
        "position": {
          "offset": 10,
          "start": 0,
          "end": 35
        }
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "cluster_instance_sul",
        "node": "jJmk5ciyQt2dTREW37pXww",
        "reason": {
          "type": "script_exception",
          "reason": "compile error",
          "script_stack": [
            "def f=doc[itemEffectiveShelvingOrde ...",
            "          ^---- HERE"
          ],
          "script": "def f=doc[itemEffectiveShelvingOrder];def a=Collections.binarySearch(f,params[cn]);if(a>=0) return f[a];a=-a-1;f[(int)Math.min(Math.max(0, a),f.length-1)]",
          "lang": "painless",
          "position": {
            "offset": 10,
            "start": 0,
            "end": 35
          },
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "cannot resolve symbol [itemEffectiveShelvingOrder]"
          }
        }
      }
    ]
  },
  "status": 400
}

I tried a different record that was in dev, but the call number isn't an LC type, so it was probably not a good second example. In any case, I'm inclined to wait until we upgrade since we are at mod-search 1.8.2 and mod-search is already 2 major versions up from where we are.

We could try disabling optimization with mod-search env var CN_BROWSE_OPTIMIZATION_ENABLED: false by using kubectl patch deployment/mod-search (seems an easier way than updating the helm chart and adding the env var to the override resources.yaml file).

[shelleydoljack]

@ahafele I updated the mod-search deployment on folio dev to have CN_BROWSE_OPTIMIZATION_ENABLED: false. I'm not sure if the data there needs to be re-indexed; it doesn't seem to be the case. I tried browsing HV640.5.S97 and DK508.9.V65 (from the google doc) and it seems to be dropping the user in the right place. It's hard to tell if it works comparing the browse in folio dev to searchworks since dev only has 1 million records. Should we try disabling the optimization on folio-test?

[ahafele]

Thanks @shelleydoljack! Yes, let's try on folio-test.

[shelleydoljack]

@ahafele I updated the mod-search deployment on folio-test to have call num browse optimization turned off.

[shelleydoljack]

Browsing DK508.9.V65 returns same 0 results and takes about 15s longer to retrieve results versus on folio-prod. Browsing DK508.9 returns same list as on prod and takes about 15-20s longer. Scrolling down and clicking next, you have to wait until more results are retrieved and it stops at DK508.9.C47 H67 2016. The user can start at that call number and browse again and it doesn't drop you in the correct place on the shelflist. Maybe turning this off isn't really great after all. :shrug:

My guess is that the browse in our version of mod-search simply can't handle the number of records. I wonder what LoC's experience is.

[shelleydoljack]

Going to move this ticket back to the backlog. Folio and folio-test have been reverted back to the optimization turned on.