Investigate and resolve issues with suppressed records appearing in Library Search

[lovinscari]

Stakeholders are reporting issues with suppressed records displaying in Library Search. Specifically,

1. From Jenny Vitti, "In Alma, the holding record for the only physical copy of this item is suppressed because it's UNIV XL/permanently lost. In discoverE, bib records are suppressed from display when all physical holdings are suppressed (even if the bib record itself isn't suppressed). However, in Library Search, this item shows up as an empty result with blank "Access & Availability" instead. It might be worth checking UNIV XM and other suppressed collections as well."

2. From a listening session with Elizabeth Russey Roke, MMS ID - 990022481590302486.

It is assumed suppressed records should not be showing in Library Search so the preference would be not to show suppressed records. If for some reason there is not a way to suppress the records in Library Search then those records should be updated with a Real Time Availability status for contact us (details of that status will be forthcoming in a separate ticket).

[bwatson78]

For 990022481590302486: LEADER 03370nas a2200481Ia 4500 is coming through from the OAI. The sixth-positioned character is n, meaning New. Per the MARC documentation, there isn't an indicator for that character space indicating Stolen or Permanently Lost.

05 - Record status
    a - Increase in encoding level
    c - Corrected or revised
    d - Deleted
    n - New
    p - Increase in encoding level from prepublication

Moving to the Availability API response:

<record>
<leader>03370nas a2200481Ia 4500</leader>
<controlfield tag="001">990022481590302486</controlfield>
<controlfield tag="005">20160111225006.0</controlfield>
<controlfield tag="007">hd afb012 bacp</controlfield>
<controlfield tag="008">080515c19809999cauqr p a 0 a0eng d</controlfield>
<datafield ind1=" " ind2=" " tag="035">
<subfield code="a">(Aleph)002248159EMU01</subfield>
</datafield>
<datafield ind1="9" ind2=" " tag="035">
<subfield code="a">u3134335</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="035">
<subfield code="a">TGPSM06-31</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="035">
<subfield code="a">(Sirsi) a3134335</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="040">
<subfield code="a">TGPSM</subfield>
<subfield code="c">MNW</subfield>
<subfield code="d">UtOrBLW</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="049">
<subfield code="a">CUY</subfield>
</datafield>
<datafield ind1="0" ind2="0" tag="245">
<subfield code="a">Pagana</subfield>
<subfield code="h">[microform].</subfield>
</datafield>
<datafield ind1=" " ind2="1" tag="264">
<subfield code="a">San Jose, Calif. :</subfield>
<subfield code="b">
Pagan/Occult/Witchcraft Special Interest group of American Mensa,
</subfield>
<subfield code="c">1980-</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="300">
<subfield code="a">volumes :</subfield>
<subfield code="b">illustrations</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="310">
<subfield code="a">Five or six issues yearly</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="336">
<subfield code="a">text</subfield>
<subfield code="b">txt</subfield>
<subfield code="2">rdacontent</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="337">
<subfield code="a">microform</subfield>
<subfield code="b">h</subfield>
<subfield code="2">rdamedia</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="338">
<subfield code="a">microfilm reel</subfield>
<subfield code="b">hd</subfield>
<subfield code="2">rdacarrier</subfield>
</datafield>
<datafield ind1="0" ind2=" " tag="362">
<subfield code="a">Vol. 1, no. 1 (Yule 1980)-</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="500">
<subfield code="a">
Description based on: Vol. 1, no. 1 (Yule 1980); title from cover.
</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="500">
<subfield code="a">ARC Accession no. 455.</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="500">
<subfield code="a">
Issues for 1980-<1990> carry both volume and sequential whole numbering (e.g. Vol. 1, no. 1, #1, v. 2, no. 1, #7, v. 3, no. 1, #12).
</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="580">
<subfield code="a">Merger of: Lunar priestess and: Robin Hood's barn.</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="533">
<subfield code="a">Microfilm.</subfield>
<subfield code="m">
Vol. 1, nos. 1-6 (1980-1981), v. 2, nos. 1-5 (1981-1982), v. 3, nos. 1-4 (1982-1983), v. 4, nos. 1-6 (1983-1984), v. 5, nos. 1-5 (1984-1985), v. 6, nos. 1-9 (1985-1986), v. 7, nos. 1-4 (1986-1987), v. 8, nos. 1-5 (1987-1988), v. 9, nos. 1-5 (1988-1989), v. 10, nos. 1-3, 6 (1989-1990).
</subfield>
<subfield code="b">Woodbridge, Conn. :</subfield>
<subfield code="c">
Primary Source Microfilm, an imprint of Thomson Gale,
</subfield>
<subfield code="d">2005.</subfield>
<subfield code="e">1 microfilm reel ; 35 mm.</subfield>
<subfield code="f">
(American religions collection. Series I. Non-traditional American religions, Pt. 1. Witchcraft, paganism, and magick ; reel no. 61, item no. 8).
</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="533">
<subfield code="a">Microfilm.</subfield>
<subfield code="m">
Vol. 1, nos. 1-6 (1980-1981), v. 2, nos. 1-5 (1981-1982), v. 3, nos. 1-4 (1982-1983), v. 4, nos. 1-6 (1983-1984), v. 5, nos. 1-5 (1984-1985), v. 6, nos. 1-9 (1985-1986), v. 7, nos. 1-4 (1986-1987), v. 8, nos. 1-5 (1987-1988), v. 9, nos. 1-5 (1988-1989), v. 10, nos. 1-3, 6 (1989-1990).
</subfield>
<subfield code="b">Woodbridge, Conn. :</subfield>
<subfield code="c">
Primary Source Microfilm, an imprint of Thomson Gale,
</subfield>
<subfield code="d">2005.</subfield>
<subfield code="e">1 microfilm reel ; 35 mm.</subfield>
<subfield code="f">
(American religions collection. Series I. Non-traditional American religions, Pt. 1. Witchcraft, paganism, and magick ; reel no. 62, item no. 1).
</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="590">
<subfield code="a">
Microfilm contains issues: Vol. 1, nos. 1-6 (1980-1981), v. 2, nos. 1-5 (1981-1982), v. 3, nos. 1-4 (1982-1983), v. 4, nos. 1-6 (1983-1984), v. 5, nos. 1-5 (1984-1985), v. 6, nos. 1-9 (1985-1986), v. 7, nos. 1-4 (1986-1987), v. 8, nos. 1-5 (1987-1988), v. 9, nos. 1-5 (1988-1989), v. 10, nos. 1-3, 6 (1989-1990).
</subfield>
</datafield>
<datafield ind1=" " ind2="0" tag="650">
<subfield code="a">Neopaganism</subfield>
<subfield code="v">Periodicals.</subfield>
</datafield>
<datafield ind1=" " ind2="0" tag="650">
<subfield code="a">Witchcraft</subfield>
<subfield code="v">Periodicals.</subfield>
</datafield>
<datafield ind1=" " ind2="0" tag="650">
<subfield code="a">Occultism</subfield>
<subfield code="v">Periodicals.</subfield>
</datafield>
<datafield ind1="2" ind2=" " tag="710">
<subfield code="a">University of California, Santa Barbara.</subfield>
<subfield code="b">Library.</subfield>
<subfield code="b">American Religions Collection.</subfield>
</datafield>
<datafield ind1="2" ind2=" " tag="710">
<subfield code="a">Mensa.</subfield>
<subfield code="b">Pagan, Occult, Witchcraft Special Interest Group.</subfield>
</datafield>
<datafield ind1="1" ind2="4" tag="780">
<subfield code="t">Lunar priestess</subfield>
</datafield>
<datafield ind1="1" ind2="4" tag="780">
<subfield code="t">Robin Hood's barn</subfield>
</datafield>
<datafield ind1=" " ind2="0" tag="830">
<subfield code="a">American religions collection.</subfield>
<subfield code="n">
Series 1, Nontraditional American religions, Western esotericism from witchcraft to the new age,
</subfield>
<subfield code="p">Pt. 1. Witchcraft, paganism, and magick ;</subfield>
<subfield code="v">reel no. 61, item no. 8-reel no. 62, item no. 1.</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="910">
<subfield code="a">RDA ENRICHED</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="910">
<subfield code="a">MARS</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="940">
<subfield code="a">AMERRELIG</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="945">
<subfield code="a">9DT</subfield>
<subfield code="2">20081022</subfield>
</datafield>
<datafield ind1=" " ind2=" " tag="949">
<subfield code="a">MICFILM4497</subfield>
<subfield code="v">reel 61 no.8-reel 62 no.1</subfield>
<subfield code="w">ALPHANUM</subfield>
<subfield code="m">GENERAL</subfield>
<subfield code="l">MICROFORMS</subfield>
<subfield code="r">Y</subfield>
<subfield code="s">Y</subfield>
<subfield code="t">MICROFILM</subfield>
<subfield code="z">AMERRELIG</subfield>
</datafield>
</record>

There isn't any obvious indicator here stating that the record is lost and/or stolen. But there could be...

Per Temple University Libraries' search, they have created a datafield "ITM" that does contain a subfield calling out whether the Item is lost or stolen. This field seems to be available in the OAI set publishing, not just the Availability API response. If we could do the same, we could add logic to the ingest processing looking for the values in our new subfield, treating those records as suppressed.

Next, I'll look at Princeton's search, just in case they have something different in place.

[bwatson78]

Princeton has a field they pull in from their availability API call that details various item statuses, including whether the item is lost or stolen.

[lisahamlett]

I figured out how to publish the 852HOL for only those locations that are not suppressed. HOWEVER, that means bibliographic records would still come over if the only item on the record is in a suppressed location. You can look at blacklighttest for example.

9936621898602486 has two holdings -- one for OXFD and one for UNIV. UNIV is suppressed so it's only publishing the OXFD one:

<datafield tag="HOL852" ind1="1" ind2=" ">
<subfield code="b">OXFD</subfield>
<subfield code="c">STACK</subfield>
<subfield code="h">823.92</subfield>
<subfield code="i">K9668w</subfield>
<subfield code="8">22343856360002486</subfield>

But 990017011650302486 has only one holding and it is suppressed, so it has no 852.

[lisahamlett]

Alex has confirmed this solution ^^^ will not work because it would prevent bound-with children from publishing.

[lisahamlett]

Attempt 2: Create a HOLSP field that contains suppression information.

Shows holding and then shows holding is suppressed (subfield 8 is holding id):

<datafield tag="HOL852" ind1="0" ind2=" ">
<subfield code="b">UNIV</subfield>
<subfield code="c">XL</subfield>
<subfield code="h"> HD38.7 .W38 2010 </subfield>
<subfield code="8">22212270450002486</subfield>
</datafield>
<datafield tag="HOLSP" ind1=" " ind2=" ">
<subfield code="a">true</subfield>
<subfield code="8">22212270450002486</subfield>
</datafield>

Two holdings, one suppressed, one not:

<datafield tag="HOL852" ind1="0" ind2=" ">
<subfield code="b">UNIV</subfield>
<subfield code="c">XL</subfield>
<subfield code="h">PR6111.U68</subfield>
<subfield code="i">W48 2017</subfield>
<subfield code="8">22328758720002486</subfield>
</datafield>
<datafield tag="HOLSP" ind1=" " ind2=" ">
<subfield code="a">true</subfield>
<subfield code="8">22328758720002486</subfield>
</datafield>
<datafield tag="HOL852" ind1="1" ind2=" ">
<subfield code="b">OXFD</subfield>
<subfield code="c">STACK</subfield>
<subfield code="h">823.92</subfield>
<subfield code="i">K9668w</subfield>
<subfield code="8">22343856360002486</subfield>
</datafield>
<datafield tag="HOLSP" ind1=" " ind2=" ">
<subfield code="a">false</subfield>
<subfield code="8">22343856360002486</subfield>
</datafield>

So we publish everything unless the count of HOL852s is the same as the count of HOLSPa==true

Brad is testing with blacklighttest OAI, which should publish 88 records.

[bwatson78]

The above logic seems to be working on both blacklighttest and blacklightemily OAI sets. I'm moving the PR with the new ingestion logic forward: https://github.com/emory-libraries/blacklight-catalog/pull/1092

[lovinscari]

I will discuss this in standup as I am still seeing the example provided by Elizabeth as appearing with no availability information.

[bwatson78]

The issue with 990022481590302486 is that it hasn't been suppressed in the typical fashion--it has been assigned to a "Temporary Location" which DiscoverE knows to not show as available. @lisahamlett has amassed a list of all temporary locations, so I will create another category of suppression in the ingest logic and test whether all physical holdings exist only in these temporary locations.

[bwatson78]

PR made for temporary location suppression: https://github.com/emory-libraries/blacklight-catalog/pull/1125

[lisahamlett]

Job to add current date to full blacklight set finished 12/09/2021 23:06:12 EST

[lovinscari]

@lisahamlett and @bwatson78 - I am confirming this is currently on blackcat-test? I would like to have Jenny Vitti and Elizabeth do some testing. Thanks.

[bwatson78]

That is currently on Test--true.

[lovinscari]

I have emailed Jenny and Elizabeth and have asked them to also check.

[lovinscari]

Jenny and Elizabeth have confirmed the issue appears to be resolved.