Cross-instance user search doesn't return users explicitly searched for

[nietaki]

(First of all sorry for opening up the first issue in this pristine repo - I'm only doing that because I want to make GenServer.social my new home and this feels like a bit of a hurdle in the adoption)

I noticed a strange behavior in the user search. Let's say I stumbled upon James Felton and want to search for him to follow.

I put the following in the search input:

@Jimmfelton@mastodon.social

The search results for People comes up empty

But when I just put @Jimmfelton in the search input, the People results contains exactly one entry, which is @Jimmfelton@mastodon.social. It doesn't matter if I already follow the person I'm seraching for - including the instance name makes sure I don't find them

I tried other formats, but they don't seem to work either:

Jimmfelton@mastodon.social
https://mastodon.social/@Jimmfelton

This isn't really a problem for people with unique usernames, because I do have a workaround, but with people like https://tapbots.social/@paul for example, it's more of a problem - just searching for @paul comes up with results that don't contain the account I'm looking for.

---

I wouldn't be surprised if this should be a bug report for Akkoma instead, so I figured I'd just post it here first, to get some feedback from the admins/community first

[nietaki]

I realised I'm following the suggested Mastodon workflow and it doesn't work:

[piisalie]

Thanks for this, I am always grateful for feed back and raising of issues. :)

So the case with jimmfelton is interesting. I can reproduce it and I get this log which I will look into tomorrow.

Found an old user for Jimmfelton@mastodon.social, but the ap id https://m
astodon.social/users/Jimmfelton is the same as the new user. Race condition? Not changing anything.

[piisalie]

I did a little more digging on this, and for users the server already knows about searching for jimmfelton@mastodon or pineywoozle@masto seems to work well.

Still not sure about the url search / if that's supposed to work (global searching is a no-go in the fedi generally). I'll dig around and report back

[nietaki]

Thanks for looking into it

I suspected that my repro might get out of date soon once the server knows about the user.

The url search seems like the most important aspect of it, since most users are (?) hosted on vanilla mastodon and that's the workflow they recommend.

[nietaki]

I was following some more people today and here's some more observations:

The url search seems to be working well (relatively quick and with the right results) for many cases:

https://infosec.exchange/@codinghorror
https://mathstodon.xyz/@standupmaths
https://infosec.exchange/@SwiftOnSecurity
https://toot.thoughtworks.com/@mfowler
https://blackrock.city/@spolsky

However, for some others, the API request takes exactly 15s + 20ms to respond and returns an empty result as far as people are concerned:

https://octodon.social/@hmason
https://mastodon.xyz/@oldmankris
https://mastodon.social/@omer

The results seem to be consistent - trying it multiple times with the same url yields the same result (so far). I'd be curious to confirm if the result is always the same based the instance being searched through (the two .xyz instances from the example are actually different instances)

---

It seems like there's some underlying timeout issue, or problems parsing the response from the other instance and an empty result after a couple of retries

That gives me hope that the behaviour could be debugged if there's detailed enough logs somewhere or using something like (shameless plug) rexbug

[piisalie]

Thank you for your thoughtful and thorough testing, I was able to combine this with some of the logs and I have an idea of what's happening.

TLDR: Search is crashing in some cases, I was already planning db updates/upgrades this weekend and we can see if that helps. If it doesn't help I can start trying to look into why that query is slow and try to optimize it.

The longer version:

The log I pasted above happens because the search looks for both statuses, and people (and hashtags).

If you pass a string starting with https:// then the status side of things begins trying to fetch the object at that link.

It then tries to either find the object or fetch it remotely.

Additionally a user search is happening by url and apid and such (this raises the database CPU pretty significantly actually)

In the cases you shared, there is already a user in the DB with an apid, but the db request times out looking for them.

Then the status fetcher tries to create an ap object for them, but that fails because they already exist.

The whole thing crashes and you get no results.

[piisalie]

Alright, I am now able to search and get results for your example users (and a few others I was testing with).

It's still little slow (which is maybe an argument to look into elastic search) but it appears to work.

Can you confirm?

[nietaki]

I now tested it with the examples from before and a couple of new ones and it all works without a hitch, thanks!

One thing that surprised me in your writeup is that the DB search for the user would timeout. I don't know how many rows the db has or how inconveniently the data is structured, but I would imagine that even with full-text search it should be a whole lot faster...

I'm going to have to set up Akkoma locally and play around with it one of these days, sorry I can't focus on that right now...

[piisalie]

Great question, there are about 3 million activities and 300k (local and remote) accounts in the db currently.

I think it's less that full text search is not performant and more that there is a lot happening. There are three separate searches that happen: Accounts, Status, and Hashtags. It filters on your users domain blocks and user blocks and keyword blocks and etc etc, so it's a bit more than a generic search. Though I did make a note to myself to go back and do some explains on that query and see if it I could improve it, it is already using tsvecors/rum indexes etc.

Also Akkoma tries to keep timeouts relatively short, and the Repo pool small to reduce database load and the likelihood that something will block something else, so the whole thing is timeboxed at 15s. Even if the accounts search returns results, if posts search takes longer than 15s then the task async kills the whole thing and we get no results. (Which is why I think it may be time to re-consider elastic search).

GenServer Social is accepting hundreds of posts and updates per minute in addition to the "normal usage" of browsing around and posting that we do, so locking or taking out a Repo pool connection for a long time could really jam things up.

I'm gonna close this issue for now but if you wanna go digging:

lib/pleroma/user/search.ex
lib/pleroma/search/database_search.ex

are the main places to start :)

[piisalie]

I believe it worked quickly for some and not others based on how quickly we could fill one page of results.

If they're an active user / mentioned a lot with lots of recent posts then finding 20 things is pretty easy, if they only have one post, then the db will keep searching and searching trying to fill the first 20 results either by proving that nothing more exists (or in our case timing out before it got that far)