search

Joystream / joystream

Joystream Monorepo
http://www.joystream.org
GNU General Public License v3.0
1.42k stars 115 forks source link

Storage squid support for Argus and Colossus #4957

Closed kdembler closed 9 months ago

kdembler commented 1 year ago

There is a consensus among SWG, DWG and JSG that we should migrate our Colossus and Argus nodes away from QN and create a storage-focused squid to serve those nodes' data needs.

Some quick notes on the current status:

  1. We want to create a squid and place it in a new repo.
  2. Storage squid should borrow as much code as possible from existing Orion mappings.
  3. We should use the latest ArrowSquid release with focus on long-term maintenance.
  4. One thing I've recently discovered is missing from QN/Orion mappings is some kind of reference to which storage operator accepted the upload of a given object. It would be nice to include that.
  5. @zeeshanakram3 already started work on some basic version
kdembler commented 1 year ago

@zeeshanakram3 btw we already have ArrowSquid archive: https://v2.archive.subsquid.io/network/joystream or lookupArchive("joystream", { release: "ArrowSquid" })

bedeho commented 1 year ago

Are we planning to have all infra use this one source of truth as the default policy?

kdembler commented 1 year ago

I think that makes sense, yes. But only for archive data, all the new processing would be done with RPC. And we can always fallback to only RPC.