Initial list not loading on WLS

[isaacbw]

Could you please have an update by EOD on Friday 7/29? Doesn't need to be solved, but just an update Ben and I can tell her on our meeting with her on Monday.

MCNY pointed out this issue on their WLS:

"On a separate note, I was on our site earlier today, https://mappingnyc.mcny.org/cities/nyc, and content didn't immediately load. When I went specifically to lists, our tour came up but when I would select posts, I would get an error message saying that nothing had been uploaded. I tried this a few times and content loaded eventually, but at the moment nothing is coming through. I just wanted to check if this was a bug or if something else was going on here."

[allthesignals]

Seems to be affecting other sites as well: https://civilrights.urbanarchive.me/cities/nyc, https://pressed.urbanarchive.me/cities/nyc, https://stories.hdc.org/cities/nyc This works: https://janeswalknyc.mas.org/cities/nyc

[allthesignals]

There seem to be a lot of "null" content "edges" in the feed:

My instinct so far is that it expects at least a few pieces of content to come through, which will then start another request for more content?... unclear...

Basically the first page is empty, so it doesn't look for any more.

[allthesignals]

@wllrd hi! Sorry to bother, but wondering if you had insight. Briefly: some of the white label sites are coming up with blank listings:

If you look above, you can see that there are a bunch of null listings:

My assumption is:

My instinct so far is that it expects at least a few pieces of content to come through, which will then start another request for more content?... unclear...

Basically the first page is empty, so it doesn't look for any more.

What could have happened: I'm guessing there are "null" hits for reasons having to do with the inner/outer join logic of graphql (i'm a noob in this area) and that other content in the database was added? Maybe I can futz with the query a bit to see if I sort ascending for truthy results?

What am I asking you?: Does this sound about right? Am I on the right track? Even a thumbsup or down is helpful. Thank you! :)

[wllrd]

@allthesignals no bother at all!

this type of issue is exceedingly common thanks to the elaborate footguns i've put in place.

here's what i'd check:

1. moderation. we designed this for the "instagram" phase of our app but the rest of the app never caught up so it's confusing to pretty much eveyone. "moderation" is basically an association between a piece of content and a whitelabel site (or UA main). for content to show up on the main feed, it typically must be at least "medium" sized. (again, "size" being a holdover from designs that died on the vine.)

2. if moderation appears to be setup properly, you may need to re-index the data. to do this, you'd get into a Rails console and import the appropriate model(s). you can either import everything (e.g. Archive::AudioAsset.import) or a subset (Archive::ImageAsset.where(<something>).import for large models that can take multiple hours otherwise. i also recommend opening a tmux session on the server so you don't have to count on ssh being up for multiple hours if doing anything slow.

hope that helps, and happy to follow up if you have more questions.

[allthesignals]

@wllrd I went down the path of investigating your first suggestion. I observed that sites without the listings don't have an associated collection. Sites that do work correctly and show a listing always have at least one associated collection.

For the whitelabel site in particular, here's what I'm seeing in the UI:

Before I start fussing with the console, I want to make sure things look correct through the admin. Thank you again for any insight you have time to provide! :)

[allthesignals]

Linking chatter/noisy thread for posterity: https://urbanarchive.slack.com/archives/C0305M2E8BW/p1659540899446059

[allthesignals]

Alright, checked moderation and there are plenty of assets with correct Medium level moderation:

Also, theory about missing collections was invalidated by a site that shows the bug AND includes collections. So, I'm going to investigate path 2 (unless some of my reasoning doesn't check out)

[allthesignals]

When I click the dropdown and close out, it seems to load the feed just fine. Interesting!

[allthesignals]

When I click the dropdown, click out, it iniates a successful query that includes a "cursor" in the variables (sounds like a pagination thing):

[allthesignals]

Thanks for that context, @cwvibe/@wllrd!

We investigated moderation setup, and it looks okay.

Next path forward is (#\2) what you mentioned above:

if moderation appears to be setup properly, you may need to re-index the data. to do this, you'd get into a Rails console and import the appropriate model(s). you can either import everything (e.g. Archive::AudioAsset.import) or a subset (Archive::ImageAsset.where().import for large models that can take multiple hours otherwise. i also recommend opening a tmux session on the server so you don't have to count on ssh being up for multiple hours if doing anything slow.

Taking this path (carefully) as a stop gap. I will get into prod rails console and run: Archive::ImageAsset.where(<associated with Museum of the City of New York>).import

(Relatedly: do you have advice on running a whitelabel site locally? I'm rifling through routing configuration to see if I can figure it out myself, since I'm running on localhost).

[wllrd]

those are results that are filtered out by security rules after the results come back. so that's happening in the graphql types and possibly the pundit definitions too.

it's a longstanding issue that i consider a major drawback of using a single search index across multiple sites. (should probably use multiple indexes, but more likely should probably remove the search engine entirely)

(oops, wrong account!)

to run locally you need an uncomfortable number of things:

• database dump -- can be found in render.com's UI
• ES search index dump (or time to rebuild it) -- particularly clunky
• docker images pulled/built
• local networking setup: i test with xxx.archive.local entries in /etc/hosts (i think it's whitelisted in Rails settings?) where xxx is the slug of the whitelabel site.

for the search index, it won't hurt to import multiple times, and is fine if you just import everything in a tmux/screen session and let it do its thing for awhile.

hope this is helpful and not too frightening...

[allthesignals]

@wllrd all good! thanks so much for chiming in. thanks, this gives me some good points. i sort of had a feeling this local setup was looking for some production dumps. seems reasonable to me!

Currently, trying to run #import:

Archive::ImageAsset.where(archive_record_id: mcny_ids.first(10000)).import

Where mcny_ids=Archive::Record.where(organization_id: 1).pluck(:id).

Not sure yet what a positive signal would be.

[wllrd]

so if some results are appearing but are pushed off the first page, it's not necessarily an issue with the index. in that case there's actually nothing that can be done beyond fix the way results are assembled.

[allthesignals]

Gotcha, so my instinct with results perhaps being pushed off the first page seems somewhat correct?

Re-indexing progress:

irb(main):018:0> Archive::ImageAsset.where(archive_record_id: mcny_ids.first(10000)).import
=> 0
irb(main):019:0> Archive::ImageAsset.where(archive_record_id: mcny_ids.last(10000)).import

Well, I'm running it on small cuts of the data so fingers crossed. There are 50k Record records. If it doesn't work with at least 20k, it's not looking too good.

[wllrd]

the import output is typically 0 i think -- seems like it refers to new records, not those that were updated.

part of my motivation for wanting to deprecate ES or split it into multiple indexes is exactly this issue -- it was a minor inconvenience until we started seeing completely non-overlapping data between different sites.

one hack you can consider short-term is to completely customize the homepage feed results -- we do that for a few other sites like VP.

[allthesignals]

Ran it on the full 50k, no luck. I'll look into the customization path. Would it help to create a collection for these assets and just associate that way? No idea if that would help. I wonder if I could try a small collection and see if that helps.

[wllrd]

you could also use a tag, and show only tagged entries -- i seem to recall there's already an instance of this

[allthesignals]

Not sure about that route but I decided to spin this up locally and my current plan of attack has been to get the query to load the correct values the "first" time...

It was a little jank but I got running locally after some tinkering with /etc/hosts and pointing the local app to the production API.

But this is from within bodega! Ooops. The white label sites are handled in grand central.

So I made a small stopgap to increase the page size for the white label sites. This seemed to address the issue of not having enough non-null results to exceed the max to trigger a request for a new page.

But the long-term fix should be to fix the query so that it only includes non-null results... I don't know why it's doing that.

[wllrd]

the issue i'd gotten at before is that a page of results is being returned from the search engine, but then some results are being nullified by the graphql library because they don't pass an access check -- this is its default behavior.

and that's an architectural flaw that requires workarounds for now (like you've done here), often related to page size or loading multiple pages. note that this should probably be limited to specific sites since it'll impact loading perf on any sites with large pages.

one solution might be to use the role based document security features of ES. the one i was heading toward is to remove elasticsearch entirely and lean on activerecord for assembling complex search queries.