Project replication stuck in a loop

[viraptor]

I've got a seed (commit ce7dc5e69c89845124edc7b01b6c9e6e92c76273) working with latest net-next upstream. I added a single local project and added a remote seed. Then the peer is never registered as online on the seed, but the project transfer is stuck in a loop.

On upstream startup the connection seems to be established:

 INFO  librad::net::protocol::io > discovered; remote_id=hybk76pim74znj9tapyarr3qmxt1qyc9u1rfk9cda63pdsq9rrz8o4
 INFO  librad::net::protocol::io::connections > connect; remote_id=hybk76pim74znj9tapyarr3qmxt1qyc9u1rfk9cda63pdsq9rrz8o4
 INFO  librad::net::protocol::io::connections > establishing connection remote_addr=35.197.169.21:12345

Then the this chunk just keeps repeating:

 INFO  librad::git::refs                           > load; peer=None urn=rad:git:hnrkk458xbhmjopnxs7ezpoprip7tc6nr414o
 INFO  librad::git::refs                           > load; peer=None urn=rad:git:hnrkk458xbhmjopnxs7ezpoprip7tc6nr414o
 INFO  librad::git::refs                           > load; peer=None urn=rad:git:hnrkk458xbhmjopnxs7ezpoprip7tc6nr414o
 INFO  librad::git::refs                           > load; peer=None urn=rad:git:hnrkk458xbhmjopnxs7ezpoprip7tc6nr414o
 INFO  librad::git::refs                           > load; peer=None urn=rad:git:hnrkk458xbhmjopnxs7ezpoprip7tc6nr414o
 INFO  librad::git::replication                    > replicate; config=Config { fetch_limit: Limit { peek: 5120, data: 5368709120 } } remote_peer=hybk76pim74znj9tapyarr3qmxt1qyc9u1rfk9cda63pdsq9rrz8o4 urn=rad:git:hnrkk458xbhmjopnxs7ezpoprip7tc6nr414o
 INFO  librad::git::fetch                          > new; remote_peer=hybk76pim74znj9tapyarr3qmxt1qyc9u1rfk9cda63pdsq9rrz8o4 urn=rad:git:hnrkk458xbhmjopnxs7ezpoprip7tc6nr414o
 INFO  librad::net::protocol                       > open-git-stream; remote_id=hybk76pim74znj9tapyarr3qmxt1qyc9u1rfk9cda63pdsq9rrz8o4
 INFO  librad::git::replication                    > replication; limit=Limit { peek: 5120, data: 5368709120 } remote_peer=hybk76pim74znj9tapyarr3qmxt1qyc9u1rfk9cda63pdsq9rrz8o4 urn=rad:git:hnrkk458xbhmjopnxs7ezpoprip7tc6nr414o
 INFO  librad::git::fetch                          > fetch; fetchspecs=Peek { remotes: {hynmraey7hm8q4g7ipnnejc6oah9fccfnppa49xntoyxo6wgr6y1uh}, limit: Limit { peek: 5120, data: 5368709120 } }
 INFO  librad::net::protocol                       > open-git-stream; remote_id=hybk76pim74znj9tapyarr3qmxt1qyc9u1rfk9cda63pdsq9rrz8o4
 INFO  librad::git::fetch                          > fetch; fetchspecs=Replicate { tracked_sigrefs: {}, delegates: {Urn { id: Oid(ad6cef0f1698344fb75176c1a4ab7b167844d4b5), path: None }}, limit: Limit { peek: 5120, data: 5368709120 } }
 INFO  librad::net::protocol                       > open-git-stream; remote_id=hybk76pim74znj9tapyarr3qmxt1qyc9u1rfk9cda63pdsq9rrz8o4
 INFO  librad::git::refs                           > update; urn=rad:git:hnrkk458xbhmjopnxs7ezpoprip7tc6nr414o
 INFO  librad::git::tracking                       > untrack; urn=Urn { id: Oid(ad6cef0f1698344fb75176c1a4ab7b167844d4b5), path: None } peer=hynmraey7hm8q4g7ipnnejc6oah9fccfnppa49xntoyxo6wgr6y1uh
 INFO  coco::state                                 > creating include file @ '"/home/viraptor/.config/radicle-link/650e67d1-43bb-4291-9258-53ee4bd41455/git-includes/hnrkk458xbhmjopnxs7ezpoprip7tc6nr414o.inc"'

On the seed side the log looks like this:

Mar 17 10:49:59 radicle-seed radicle-seed-node[3601]: Mar 17 10:49:59.523  INFO incoming: librad::net::protocol::io::streams: new ingress bidi stream remote_id=hynmraey7hm8q4g7ipnnejc6oah9fccfnppa49xntoyxo6wgr6y1uh remote_addr=220.235......:50221
Mar 17 10:49:59 radicle-seed radicle-seed-node[3601]: Mar 17 10:49:59.523  INFO invoke_service: librad::git::p2p::server: upload pack ls
Mar 17 10:49:59 radicle-seed radicle-seed-node[3601]: Mar 17 10:49:59.527  INFO invoke_service: librad::git::p2p::server: upload pack ls done

The seed address is hybk76pim74znj9tapyarr3qmxt1qyc9u1rfk9cda63pdsq9rrz8o4@radicle.viraptor.info:12345, and the seeded project is https://github.com/viraptor/arpy/

Removing the store and retrying results in the same behaviour for me every time.

[viraptor]

Then the peer is never registered as online on the seed

That was incorrect. The peer is listed at /peers (with type: "connected"), but not loaded on the website or in /info. Looks like a separate issue.

[kim]

That second snippet looks very strange indeed. Is that completely unattended? Why is there an untrack? /cc @xla @FintanH

[viraptor]

Yes. Completely new setup. I added a new project from existing local repository. And that was the only action.

[FintanH]

That second snippet looks very strange indeed. Is that completely unattended? Why is there an untrack? /cc @xla @FintanH

The untrack there is a call to prune in replication.rs. It's making sure that local PeerId isn't appearing as a remote in the namespace.

The looping is likely due to the sync loop in Upstream, right @xla? I'm presuming you're testing with this branch https://github.com/radicle-dev/radicle-upstream/pull/1463/ @viraptor?

[xla]

The looping is likely due to the sync loop in Upstream, right @xla? I'm presuming you're testing with this branch radicle-dev/radicle-upstream#1463 @viraptor?

Yeah, we aggressively sync at the moment to ensure changes propagate. This is a crutch to keep the "experience" snappy in Upstream. It definitely means we are trading bytes on the wire for this and we might tune it down to be less frequent to not overburden with excessive resource consumption.

[viraptor]

That's the branch. Specifically this report from commit dcb90fe8947f3b9b20e9ea15b9f8aed0f32f1b8b