Bug: stale feed. Large contact list.

alltheseas commented 1 week ago

Tracking Issus

These are steps needed to fix this issue

What happens Feed seems stuck. Only hours long posts show. Universe view is showing new posts. DMs work. Notifications work.

What I expect to happen I dont expect to see a stale feed.

Link to noteID, npub

reported by elsat npub1zafcms4xya5ap9zr7xxr0jlrtrattwlesytn2s42030lzu0dwlzqpd26k5

Screenshots/video recording https://github.com/damus-io/damus/assets/64376233/95ae8255-4cfb-4be0-8cf0-b56124ebe0d4

a	b

Versions Damus version: 1.9 (1) 90c68fed Operating system version: iOS 17.4.1] Device: e.g. iPhone 13 mini

Steps To Reproduce Steps to reproduce the behavior:

Unclear.

alltheseas commented 1 week ago

For brief moments feed becomes real time - for instance after not having used the app for an hour.

After app close, and open feed is stale oncemore.

alltheseas commented 1 week ago

Stale feed problem persists

universe view is real time

jb55 commented 6 days ago

On Tue, Apr 30, 2024 at 02:08:53PM GMT, alltheseas wrote:

Stale feed problem persists

universe view is real time

can you try reducing your number of relays and see if that fixes anything. I've heard reports of this and I'm not sure why more relays breaks things.

alltheseas commented 6 days ago

reduced relays - does not change stale feed

alltheseas commented 3 days ago

Turned on damus on old phone - same issue present

alltheseas commented 3 days ago

If yall log in with my npub are you able to recreate?

jb55 commented 3 days ago

(this was sent via email but also sent to the github issue as a comment)

Re: [damus-io/damus] Bug: stale feed (Issue #2194) Link: https://github.com/damus-io/damus/issues/2194

On Sat, May 04, 2024 at 07:09:46PM GMT, alltheseas wrote:

reduced relays - does not change stale feed [..] Turned on damus on old phone - same issue present

Your contact list is large and contains lots of hashtags and pubkeys. This may have something to do with it.

We should use these to build some test cases for this to dig down and see what is going on.

I have some suspicions:

Potential problems

Your contact list is about 150KB, subscribing to that 9x is about 1.4MB of upload every time your phone reconnects. The sheer weight of this is likely causing stalling.
Relays may reject large subscriptions like this, leading to empty feeds.
There may be some data encoding problem in the list leading to a rejected subscription, but I did not see any issue on your contact list. I did see some lists like this though.

Options

Outbox model to only send subsets of your list to the sets of relays that your follow list is on. This is probably the best path without writing any new NIPs but may lead to missed messages depending on how damus chooses which relay subsets to read from for each pubkey subset.

If most users have the damus relay on their list we may be back to the same problem. Relay reliability is also a concern in this scenario as well as all the other outbox model issues in Damus which still needs to be resolved by the local relay model[1][2]
Only choose a subset of the relays to send the timeline subscription to. This may result in missed messages.

Workarounds

Since your contact list is so large, you may need to either:

Start going through and unfollowing people who are no longer active

or

Severely reduce your relay list to 1-4 until we have a fix.

This is a huge problem with the protocol at the moment with not many obvious solutions. I try to keep my contact list under 1000 for this reason.

We should definitely run some simulations on how much data the outbox model will save us here, with real world data from real relay and contact lists such as yours.

Next steps

Look into ways for damus to suggest stale accounts to unfollow
Put together potential outbox model subscription selection algorithms and see how much it helps us (may only help with a diverse selection of relays on relay lists)
Use elat's current contact list as a testbed for fixing this issue since it is an example testcase that is not working.
Look into follow caps to reduce the damage of this issue but this may be controversial, perhaps we can find a solution without doing this.

Ideal solution

Look into a potential new NIP for relays. Relays already have your contact list, what if the query was simple as:

{ "authors": ["kind3:"] }

Which would do the same thing as the 150k query except just tell the relay to use the list it has instead of sending it over and over.

Then we could send a few bytes instead of 150k * number of relays

I prefer this approach, not sure if anyone has proposed it yet.

I have a feeling a lot more users are going to be running into this issue so we should starting thinking of solutions soon.

Cheers,

Will

[1] https://github.com/damus-io/damus/issues/2041 [2] https://github.com/damus-io/notedeck/

alltheseas commented 2 days ago

Potential additional test case:

note1v2uzw8f488y29mhuafwjrkn45ss79sx2fydn3sedtxkca46rnjcq6u3g74

npub1vwymuey3u7mf860ndrkw3r7dz30s0srg6tqmhtjzg7umtm6rn5eq2qzugd

alltheseas commented 2 days ago

damus-io / damus