(uber ticket) Newly created collections not showing up in "Set Collection" or "Edit Collection" drop-downs

[blalbrit]

1490 investigation is needed to determine the root cause

Describe the bug Some (all?) newly created collections are not showing up in the list of collections for the Bulk Update "Set Collection" or Item View "Edit Collection" tasks.

User Impact A user (in this case @hannahfrost ) is not able to add a newly created collection to items that were not formerly associated with a collection (see below for details).

To Reproduce Steps to reproduce the behavior (for Bulk Update):

1. Go to https://argo.stanford.edu/catalog?f%5Bis_governed_by_ssim%5D%5B%5D=info%3Afedora%2Fdruid%3Apw482xr1298
2. Click on 'Bulk Update'
3. Click on 'Get druids from search'
4. Scroll down to 'Set Collection'
5. Scroll down to the drop-down list of collections and look for 'Stanford University, Department of Physics, senior honors theses' (https://argo.stanford.edu/view/druid:ds182hd6984)

(for Item View):

1. Go to https://argo.stanford.edu/view/druid:bm962wh8682
2. Click on 'Edit Collections'
3. From the 'Add a Collection' drop-down, look for 'Stanford University, Department of Physics, senior honors theses' (https://argo.stanford.edu/view/druid:ds182hd6984)

Expected behavior The collection 'Stanford University, Department of Physics, senior honors theses' (https://argo.stanford.edu/view/druid:ds182hd6984) should appear in either/both drop-down lists.

Screenshots Let us know if you need screenshots for any of the above.

cc. @hannahfrost

[blalbrit]

See https://app.honeybadger.io/projects/49894/faults/50335590

[mgolson-stanford]

Describe the bug: We recently created a new collection object for the Shikamura collection https://argo.stanford.edu/view/druid:my748dz4281. This newly created collection object is NOT showing up in the collection drop down and we are unable to associate digitized objects with this collection or release them to SearchWorks.

User Impact: A user (@mgolson) is unable to add a newly created collection to items that were not formerly associated with a collection. This is preventing us from associating digital objects with the new Shikamura collection object and releasing this content to SearchWorks.

To Reproduce: (for bulk update)

1. Go to https://argo.stanford.edu/catalog?f%5Bexploded_tag_ssim%5D%5B%5D=DPG+%3A+Shikamura
2. Click on "bulk update".
3. Click on "get Druids from search"
4. Scroll down to "set collection"
5. Scroll down to the drop-down list of collections and look for "Archive of Takeo Shikamura and Hatsu Shikamura" (https://argo.stanford.edu/view/druid:my748dz4281).

(for item view)

1. Go to https://argo.stanford.edu/view/druid:my748dz4281
2. Click on "edit collections".
3. From the "add a collection" drop down list of collections look for ""Archive of Takeo Shikamura and Hatsu Shikamura".

Expected Behavior

The "Archive of Takeo Shikamura and Hatsu Shikamura" should appear or be available in either or both drop down menus.

cc: @mgolson and @cebraj

[oceanofsound]

Collection added, objects closed - not sure why, but I could see the collection in the dropdown.

[oceanofsound]

Sorry, didn't mean to close, this issue may still be a bug, but I was able to fix @mgolson-stanford's objects.

[dinahhandel]

I am having the identical issue as @mgolson-stanford above. Also with an EAL collection, Wang Jingwei and Lin Bosheng photo collection 194?-1944?. The objects I want to add to the collection are here: https://argo.stanford.edu/catalog?f%5Bexploded_tag_ssim%5D%5B%5D=Project+%3A+Wang+Jingwei

@oceanofsound can you try with my collection?

[oceanofsound]

@dinahhandel, sure can - collection added to those objects and versioned.

[dinahhandel]

@oceanofsound you have the magic touch! What's wrong with us over here in Lathrop? 🤔

[peetucket]

Impact: users are unable to add items to collections, either singly or in bulk. This requires either manual intervention from a developer or from a user with Argo admin powers.

[peetucket]

The drop-down menus that show the list of available collections for adding (both on the item detail page and in the bulk actions page) are built from solr, as shown here:

https://github.com/sul-dlss/argo/blob/master/app/models/concerns/permitted_queries.rb#L52-L76

This code filters the collections in solr based on which APOs the logged in user has permissions to access. Some possibilities:

• the indexing is incorrectly identifying APO permissions when creating solr docs
• the indexing is somehow not correctly tying collections to APOs when creating solr docs
• problems with the particular roles certain people are in (i.e. not a bug in the code, but incorrect roles management)
• some bug in the code above that constructs the solr query that is doing the filtering incorrectly

Note that the code that returns a list of collections for the "add collection" drop-down menus is different from the code that returns a list of collection on the item registration page. For item registration, you only see collections governed by the APO selected on the registration page. That code is here:

https://github.com/sul-dlss/argo/blob/master/app/controllers/registration_controller.rb#L35-L64

For collections to show up in this list, they need to be in the administrativeMetadata datastream under the <registration> node.

[ndushay]

Is it possible this is a solr index out of sync error? Does reindexing the new collection object fix it?

[cebraj]

That was my first thought when @mgolson-stanford and I encountered this issue. I thought it could be due to the indexing lag because I checked the APO adminMD and the collection druid we created was listed there, so that was a good sign. But the collection never showed up in any of the dropdown lists or the Collection facet list (until Geoff W. helped out, see above), and I'm pretty sure we tried reindexing.

[justinlittman]

Here's my theory: the users who are encountering are admins (which also explains why they're all in Lathrop, @dinahhandel!).

The list for admins includes all collections: https://github.com/sul-dlss/argo/blob/master/app/models/concerns/permitted_queries.rb#L54

But the result is limited to a 1000: https://github.com/sul-dlss/argo/blob/master/app/models/concerns/permitted_queries.rb#L64

However, we have more than a 1000 collections (1361 to be exact).

Thus, I suspect the list of collections for admins is being truncated. Since the ordering is probably oldest to newest, this means the newest collections are omitted.

[peetucket]

It looks like the sort is by title actually (confirmed in the UI), but oddly enough, seems to be done in Ruby as opposed to asking Solr to do it.

So the quick and dirty fix is to expand the limit (e.g. to 2000). This will eventually become a problem again, though at that point, the list becomes so unwieldy, a bigger change to the UI would be needed.

[peetucket]

Ahh, the title is in sw_display_title_tesim, which is a multivalued field, and thus we cannot ask Solr to sort on that field. Hence why the title sort is done in ruby once the solr results are returned.

[peetucket]

Another non-optimal solution is to inverse sort by date (putting latest at the top), but then finding something that is not recent becomes a chore.

[ndushay]

We believe this is fixed with PR #1528, deployed today. The specific items listed here by Hannah and Michael Olson are visible to me (currently from home, VPNed, but also during meeting in 369 Lathrop).

@andrewjbtw please close this issue if you are satisfied.

[ndushay]

actually, I'm closing this issue; please reopen (or make a new ticket) if you still experience this problem.