Orion V2 initial implementation draft

[Lezek123]

Orion V2 initial implementation draft

The draft implementation I'm describing in this document can be found HERE.

This repository is based on the current squid-substrate-template and contains all the input schemas, refactored Atlas queries, drafts of the custom graphql resolvers etc., as well as some basic setup and reference code to ilustrate some of the ideas of how certain issues can be addressed.

The official Subsquid documentation, which is often referenced throught this document can be found here: https://docs.subsquid.io/

How to run the local setup

I'll be explaining each part of the setup more deeply later in this document, but here is just a quick-start reference:

1. Clone the Joystream repository (if not already done)

git clone https://github.com/Joystream/joystream.git

2. Run the joystream-node service in the Joystream repository

# You can also specify the usual environment variables like `RUNTIME_PROFILE` etc.
export JOYSTREAM_NODE_TAG=$(./scripts/runtime-code-shasum.sh)
docker-compose up -d joystream-node

3. Clone the Subsquid-Orion repository

cd ..
git clone https://github.com/Lezek123/subsquid-orion.git

4. Run the archive (indexer)

cd subsquid-orion/archive
docker-compose up -d

5. Build the processor

cd ..
npm install
make codegen
make build

5. Run and migrate the processor datbase

   make up
   make migrate

6. Run the processor

   make process

7. Run the GraphQL server

   make serve

After performing those steps you should be able to go to http://localhost:4350/graphql and see something like this:

Currently the processor will produce some mock data on each block so you can also test some of the existing queries:

On-chain data indexing and processing

Squid Archive

Squid Arachive is analogous concept to the Hydra Indexer, it uses the Joystream node websocket rpc endpoint to fetch data about on-chain events and extrinsics and store it in a relational database (PostgreSQL).

We can configure the archive via a docker-compose file located in archive/docker-compose.yml.

The current squid archive configuration is using a local Joystream docker node (ws://joystream-node:9944) that runs on joystream_default network as a source.

SubstrateBatchProcessor

SubstrateBatchProcessor is a class we use to instantialize the events processor. As opposed to Hydra, where we would only implement the "mapping" functions (or "mappings"), Subsquid let's us instantialize and programatically configure the processor ourselves (manifest.yml file no longer required), which gives us more controll over its behavior.

SubstrateBatchProcessor is just one of the many processor implementations available in Subsquid, but it's the one currently recommended for processing substrate events and extrinsics. This specific processor implementation queries all blocks along with the events of interest from the Squid Arachive (using the @subsquid/substrate-gateway service). The maximum number of blocks in a single batch currently depends on the @subsquid/substrate-gateway implementation, it's still a little unclear how this will work in the future, but currently there are two main components that affect the batch size:

• the time it takes to read & prepare a batch (by the gateway) is limited to 5 seconds
• the size is limited to "1 MB" (however note that this currently depends on some assumptions about "average" event/call/extrinsic size, which are not very reliable, so the final size may be much greater)

Current processor implementation:

In the current draft implementation:

• the processor is given new TypeormDatabase({ isolationLevel: 'READ COMMITTED' }) instance (which is then used to insert data into the database). As you can see we can easily provide config for TypeormDatabase here, which we take advantage of by specifing isolationLevel: 'READ COMMITTED'. This isolation level reduces the possibility of conflicts, since the database state will be modified by both the processor and through the external api (ie. counting video views, featuring, reporting video/channel etc., as explained further down below);
• I added some code which populates the database with a bunch of "mock" entities on each System.ExstrinsicSuccess event;
• I specified Content.VideoCreated and Content.ChannelCreated among events of interest for ilustration purposes.

This impelementation provides a decent general overview of how the "mappings" are written in Subsquid and how one can extract the events&data of interest from a batch and then perform bulk inserts/updates at the end of processing a batch, which considerably increases the performance.

The API

Input schema

The current input schema files can be found here: https://github.com/Lezek123/subsquid-orion/tree/main/schema I tried to preserve a similar schema to the one we currently use in Hydra, however there are some notable differences:

• I only used entities that are actually of interest for Atlas (so no proposals, working group, forum etc.);
• In some entities I reduced the set of fields to only those that are currently of use for Atlas. For example, since Atlas currently doesn't support channel collaborators, I removed collaborators field from the Channel entity. This is mainly for simplification purposes and to reduce the initial scope of work, they can of course be added later if needed;
• Interfaces are no longer supported in Hydra, however, since unions can be used as an alternative, I refactored the events to use EventData union instead. The result can be seen here. I will explain the differences that come from this change further down below;
• Since deeply nested filters are now supported, as well as nested field queries, I removed some redundant entity relationships etc., which I assumed were mostly serving as a workaround for the lack of those features before;
• I replaced fields like nftOwnerMember, isNftOwnerChannel, nftOwnerCuratorGroup in the NFT entity with a NftOwner union for better clarity
• Unified state: The input schema now also includes entities that were previously only existing in Orion, like ChannelReport etc. Some of those entities will probably be moved away from the input schema, unless we want to take advantage of the autogenerated queries that Subsquid will provide for them. If not - we can just use custom models instead (as described further below).
• Entities like Video and Channel now include new followsNum and videoViewsNum counters, as Atlas relies on being able to execute queries that include sorting based on those values, as well as recieving those values as part of the result set. Having that in mind, I decided that the cost of introducing those additional fields is relatively low compared to the efficiency&simplicity benefits, however some custom aggregation queries (like sorting based on number of follows within given time period) will also be needed, as will be explained further below.
• activeVideoCounter fields have been removed, as this data is now accessible through extended queries. See Custom type-graphql Resolvers section for more details;
• createdAt and updatedAt fields are no longer automatically added, so in some cases I included them in the input schema (for Event entity however I decided to name the field timestamp instead);
• Many-to-Many relationships are not supported in Subsquid so they were refactored to 2-side Many-to-One relationships with a specific "join entity".

Custom models

Subsquid comes with a nice directory structure alowing us to define our own TypeORM models separately from the autogenerated ones, however they will all become part of the same database.

Use cases:

The primary use-case for definig those custom models is when we don't want Subquid to autogenerate the public api endpoints for quering certain (private) data, but we still want to keep this data as part of the same database to take advantage of the relational model. Take User entity for example. We want to be able to connect users with channels through User>-ChannelFollow-<Channel relationship, but we don't necessarily want to expose any User data through the api, that's why we define custom models for User and ChannelFollow, but we don't include those entities in the input schema.

Custom GraphQL api extensions

Subsquid allows use to add some custom extensions to the autogenerated GraphQL api. Those are stored in the src/server-extension/ library and constitute a significant part of the project.

Custom type-graphql Resolvers

Custom type-graphql resolvers are classes where we can define our custom GraphQL queries, mutations and subsriptions that will then be included in the final API.

Normally we run a Subsquid graphql server using the @subsquid/graphql-server library/service, which generates and runs a GraphQL server based on the input schema. For the purpose of generating the final ("output") schema and resolves it uses another library called @subsquid/openreader. The schema generated by @subsquid/openreader is then merged with the schema generated from our custom resolvers that we are providing in src/server-extension/resolvers. For this merge, the mergeSchemas method from graphql-tools library is used.

The interesting property of mergeSchemas is that this method also merges all individual GraphQL types defined in both schemas, which makes us able to reuse the autogenerated types like Video, VideoWhereInput, VideoOrderByInput etc. All we have to do is define a graphql object with the same name in our resolvers space and at least one property which matches with the autogenerated object (for entities it can be, for example, id: string). Then when the types are merged, we will get a consistent Video object with all the expected properties in the final schema.

This can be probably better understood by looking at the implementation inside https://github.com/Lezek123/subsquid-orion/tree/main/src/server-extension/resolvers, especially https://github.com/Lezek123/subsquid-orion/blob/main/src/server-extension/resolvers/baseTypes.ts where the "placeholders" for the to-be-autogenerated types are defined.

There are also many other useful references in this directory:

• In https://github.com/Lezek123/subsquid-orion/blob/main/src/server-extension/resolvers/ChannelsResolver/index.ts#L31 I implemented extendedChannels query to ilustrate how we can take advantage of the subsquid libraries to define queries that build "on top of" the autogenerated queries like channels. In this example implementation we make use of the autogenerated ChannelWhereInput, ChannelOrderByInput types and translate the client's request to a corresponding SQL query, same way it would've been done by the channels query. However, on top of this, we're able to include some custom, additional fields (like, in this case, the count() of videos that meet certain criteria in each channel) and allow the client to filter and sort the results by those fields (they can also be included in the results set if needed). In this specific implementation I showed how we can get rid of Channel.activeVideosCount field, which would otherwise need to be constantly updated on many different occasions, by replacing it with a subquery that can be ran for each queried channel upon client's request instead;
• In https://github.com/Lezek123/subsquid-orion/blob/main/src/server-extension/resolvers/StateResolver/index.ts#L30 I've shown how we can implement a custom subscription to keep Atlas informed about the current processing state;
• In https://github.com/Lezek123/subsquid-orion/blob/main/src/server-extension/resolvers/ChannelsResolver/index.ts#L164 and a few other places you can see how we can easily add some custom mutations.

Use-cases summary for custom resolvers:

• Introducing custom queries or extending the autogenerated ones:
  • extendedChannels query (allows querying channel along with activeVideosCount aggregation)
  • extendedVideoCategories query (allows querying video categories along with activeVideosCount aggregation)
  • mostRecentChannels query (a query that allows filtering and ordering results among X most recent channels)
  • channelNftCollectors query (allows querying the list of members who collected the highest number of nfts issued by a given channel)
  • searchChannels query (allows implementing custom channel search logic)
  • searchVideos query (allows implementing custom video search logic)
  • mostViewedVideosConnection query (allows querying videos with the highest number of views in a given time period)
  • getKillSwitch query (allows retrieving the current Atlas "killSwitch" status)
  • videoHero query (allows retrieving information about content currently featured in the Atals Hero section)
• Introduing mutations:
  • followChannel
  • unfollowChannel
  • reportChannel
  • reportVideo
  • addVideoView
  • setKillSwitch (operator only)
  • setVideoHero (operator only)
• Introducing subscriptions:
  • processorState (allows Atlas to stay updated about the current processing state, similar to the Hydra's stateSubscription)

checkRequest plugin

The checkRequest plugin is a Subsquid feature that allows us act on the Apollo server's requestDidStart event. The handler function can be implemented inside src/server-extension/checkRequest.ts and recieves information like request headers, ip of the origin, all the data specific to the graphql request etc.

The current example implementation shows how this plugin can be used to introduce some authentication for all mutation requests.

Use cases:

• Authentication & access restriction: The main use-case I see here is restricting access to operator-only mutations like setKillSwitch, setVideoHero etc., however, as described in the Known issues section, I believe this is a suboptimal way of doing this.

Atlas queries: Refactored!

I refactored all existring Atlas queries (https://github.com/Joystream/atlas/tree/master/packages/atlas/src/api/queries) to match the new schema.

The results can be seen here. The directory structure matches the one in the Atlas repository, which makes it easy to do side-by-side comparison. I also added CHANGE: comment in all places where the changes where introduced.

The most notbale changes can be observed in the notifications / events queries, due to the refactorization of Event entities. It is now easier to query all the events of interest together and apply filtering, sorting and limit on the results of one query instead of making separate queries for each event type and then post-processing the results client-side.

Some other notbale changes include:

• Some unused (by Atlas) queries were removed;
• Wherever there was a reference to entityId, it had to be replaced with entity.id, as Subsquid doesn't support the former syntax anymore;
• Wherever there was a reference to ID graphql type id had to be replaced with String, as Subsquid doesn't support the former anymore;
• Wherever there was entityByUniqueInput query used it had to be replaced w/ entityById, as entityByUniqueInput is not supported in Subsquid. For members, which used to be queryable by handle, we can now either add a custom query or use the existing members query (providing handle in the where clause);
• Some redundant relations (like event.data.bidder, if it can be also accessed through, for example event.data.bid.bidder) were removed, so filtering now goes more deep in some cases;
• For NFTs, fields like nftOwnerMember, isNftOwnerChannel, nftOwnerCuratorGroup were replaced with NftOwner union;
• Some queries were renamed (like admin => getKillSwitch);
• Channel/videoCategory queries that included activeVideosCounter were changed to extendedChannels/extendedVideoCategories queries;
• Videos featured in a category are now simply queried via category.featuredVideos relation;
• Some very specific, REST-api-like Orion queries like top10Videos were replaced with a little bit more generic/customizable queries like mostVievedVideosConnection;
• New search queries (separate one for channels and videos);
• Changes related to Many-to-Many relationships no longer being supported.

Custom migrations: Setting up the database

Subsquid allows us to generate database migration files that we can then use to setup the processor database. Besides that, we can also specify some custom migrations that will be ran before or after the generated ones. In the draft implementation I introduced 2 custom migrations: Views and Indexes (since the filenames and class names need to include timestamp I've just choosen some arbitrarly high values to make sure those migrations are always ran after the autogenerated one)

Use cases for custom migrations:

• We can specify indexes on jsonb fields or expressions, which is not possible through the input schema. This is useful when dealing with unions where some of the variants include a reference to another entity, like the new EventData union.
• We can introduce views, which has a few benefits:
  • We can simplify complex queries
  • We can replace tables with views, for example, a channel can be replaced with channel view, which allows us to filter out certain channels from the results of any autogenerated query. In the draft implementaton I use channel view to exclude moderated channels, this way Atals doesn't need to worry about including this filter in each query, the censored channels are also hidden from anyone trying to query the server directly. If the channel gets "unmoderated" however, it will instantly re-appear in the results (which wouldn't be possible if we just deleted moderated channels permanently).

Performance

Using the mocked data I did some performance tests against the current implementation, here are some results:

1. GetExtendedBasicChannels query

Arguments:

  where: { activeVideosCount_gt: 2 },
  orderBy: createdAt_DESC,
  limit: 50

Number of channel entries: 12,921 Number of video entries: 257,400 Time to execute the query: 86ms

2. GetNotifications query

Arguments:

  channelId: "1",
  memberId: "1",
  limit: 50

Number of event entries: 2,574,000 Time to execute the query: 880ms

3. GetNftHistory query

Arguments:

  nftId: "1"

Number of event entries: 2,974,000 Time to execute the query: ~9 seconds (!)

Potential candidate for optimalization

Benchmarks to be continued...

Known issues and unresolved questions

• The Context provided by @subsquid/graphql-server is very limited, for example, we cannot access client's IP address or any request headers inside the graphql resolver. This is problematic, as it makes authentication more complex (we have to use the separate "checkRequest" plugin) and also makes it difficult to handle mutations like addVideoView, where such data would be necessary to prevent abuse.
• There are two ways I can think of in terms of introducing global category filtering, but both have their pros and cons:
  1. We can avoid storing unrelated videos/events completely, which seems like a natural approach to avoid unnecessary bloat of the database with unrelated content, however this has two obvious drawbacks:
     • if the video category changes there is no easy way to get all the requeired information in order to (re)store this video,
     • if the operator changes the supported category set we run into the same problem - we have no access to data from categories that were not supported before (unless we re-process the chain from scratch)
  2. We store all videos/events, which kind of defeats the idea of vertical scaling, however we can then use views to filter-out unrelated content as described in Custom migrations: Setting up the database section.

Alternatives to consider

"Manually" setting up graphql server instead of using @subsquid/graphql-server

To have more control of the setup we can run the graphql server from within our own codebase instead of using the @subsquid/graphql-server, we can still take advantage of @subsquid/openreader however to generate the initial schema and resolvers.

Pros of this approach:

• We solve the Context issue, so we can more easily implement our own authentication, authorization, rate limits and other restrictions.
• Generally more freedom when writing our own resolvers, we can also decide how we want to merge them with the autogenerated schema.
• We avoid potential issues if the @subsquid/graphql-server changes in the future in a way that no longer supports the current assumptions.

Cons:

• Requires more work
• We will be probably repeating a lot of work that Subsquid team already did/does, while we could try to push for more customizability instead (which I think Subsquid as a project would benefit from too)

[dmtrjsg]

I removed collaborators field from the Channel entity. This is mainly for simplification purposes and to reduce the initial scope of work, they can of course be added later if needed;

This is used for YPP project, and added to channels with single permission to add videos, used for the purpose of YPP auto sync feature.

searchChannels query (allows implementing custom channel search logic), searchVideos query (allows implementing custom video search logic)

What are the realms of possibility how search relevancy can be defined? Would it be possible to define it as mix of title/ description text match (fuzzy), date uploaded, nft minted, video views, channel subscribers defined as some sort of polynom?

[kdembler]

I'm amazed we managed to get a working draft that fast. All the improvements like deeper filtering and especially nested field queries will make our lives much easier. Looks very promising

@dmtrjsg

This is used for YPP project, and added to channels with single permission to add videos, used for the purpose of YPP auto sync feature.

We only add collaborators, we don't actually read them from QN, so not a problem

[dmtrjsg]

MM from 13 Dec:

Status

Most mappings are done, some mappings are still WIP, NFT mappings are to be done

Features Round Up

v1 vverall objective is to relaunch Orion using the immediate benefits, and build a solid foundation for future expansions in the areas of Orion2 accounts, Notifications, Advanced Featuring, Advanced Search Relevancy, Scalability and Performance improvements.

1. Filtering by operator must be supported - must be in v1 release
2. Search is just string match now - (enhanced search can be left out of v1 release)
3. Categories can be processed even the ones that are not interested - we do not need pre-processing for speed in V1
4. Featuring will need to be replicated from Orionv1
5. Some deep technical issues on speeding things up - can be left out of v1

Next Steps

1. Atlas team efforts led by @drillprop will focus on trying to Integrate Atlas with Orion2, listing all places that needs to be addressed in a separate dedicated issue to be created
2. We will meet once a week on Tuesdays @dmtrjsg to set up the recurring meeting same time as today to discuss the progress and sync up on next steps. Next meeting is Tuesday 20th.
3. Atlas team has Orion2 as new top priority. (Payouts/ YPP bugfixing and release assistance is priority 0, and Playlists is prio2)

[dmtrjsg]

MM Dec 20th:

Orion2 All mappings are now completed incl NFT Mutations are added to graph ql that were supported by Orion Adding support for custom categories No custom queries just yet.

Atlas Pr opened with new queries are re-written, some things are not working mentioned in the PR.

Next Milestone Prepare testable environment to test the functionality after all of the queries are re-written. Test environment would entail deployment of Orion2 pointing to Atlas dev.

Remaining effort on Orion2 side

50% spanning custom queries, testing and benchmarking.

[dmtrjsg]

MM: 10th January 2023

Orion side

Finished the views. All the videos and channels should not appear, censored, filtered out will not be displayed by default Added mutation to set the supported categories by operator and it also supports settings whether videos with no categories will be supported, and recently added categories that did not exist before. Most viewed videos connection query added. Extended channels query added (ones with most views). Resolved custom parser issues.

Full list of the queries to implement are tracked in this issue:

• https://github.com/Joystream/orion/issues/52

🟢 Action: @Lezek123 to add some indicative timelines for the groups of tasks indicated in the tracker issue for better coordination with Atlas teams pace.

Atlas side

Small changes added after follows and views were added on the Orion2 side. Updated PR (as prev mentioned) with playground for Orion but haven't started testing latest changes. Memberships creation, uploading videos etc in the new integrated environment will be attempted today if time allows after Gleev launches.

Follow ups will be shared after testing.

🟢 Action: @drillprop will create the tracker issue similar to #52 to track the progress of testing and remaining effort.

[dmtrjsg]

Comments on scope:

• Everything under Operator mutations and Queries can possibly be finished by the end of the week and this already provides a great testing ground,
• After that I think the main focus will be on testing + reviews + fixes, I suppose it's a process that will take a few weeks to complete,
• W.r.t. to benchmarks, I think a few of them should be done as a sanity-check, this would be mainly for some custom queries that were added to make sure they are not very badly optimized (like the activeVideoCounters queries etc.). I think another thing worth looking into is how severe stress on the processor can slow down mutations, as they are executed against the same database etc.
• I'll try to get in touch with Subsquid team to figure out how to progress with https://github.com/Joystream/orion/issues/54, but as the issue states, there are a few steps that can be taken even now,
• Probably the biggest unknown is how many of the things that are / have been worked on in parallel on the QN/Atlas side (like playlists, payouts and probably many other things that I'm not even aware of) will need to be supported in Orion v2. I think I'll try to do some research on my own to figure this out and then perhaps follow up in case I have any questions

[dmtrjsg]

Release Update

⚠️ All queries for Atlas functioning are now complete 🥳 ETA on the first release now strictly depends on the outcome of manual testing. Positive scenario of first release estimates: in 2 weeks time subject to outcomes of manual testing 🔜

Orion

First release scope will not include

• Automated Testing
• Elaborate benchmarking

• Operator queries of getting rich context on

  Video views Channel/video follows Channel/video reports

  Which is not blocking Atlas functionality and are designed for analytics purposes

Currently Lezek is working on the

• https://github.com/Joystream/orion/issues/54 But the solution chosen no longer depends on Subsquid team so we are not blocked in any way.

Atlas

On the Atlas side, no tests done yet in the light of other priorities last week, but will be the second priority after "Apps as Channels" work is done and unblocking Radek to progress with "Apps as first class citizens"

Manual testing tracker issue:

• https://github.com/Joystream/atlas/issues/3630

[dmtrjsg]

@Lezek123 we've concluded that Orionv2 scope needs to be increased to accommodate for the Apps feature:

• https://github.com/Joystream/joystream/issues/4158

There's an open question whether CLI interface needs to be built for Orion now to create channels, so that we don't duplicate this functionality in QN and Orion. Smth for you to advise us on. I'll share this in Discord and we can all discuss there.

[dmtrjsg]

Orion

• Entity caching is done

New issues added:

• Ephesus mapping for channel payouts + Direct channel payouts

• Censorship functionality for GW operators (hide channels/ videos/ objects if censored)

• Deployment options: Aquarium allows for simple deployments for operators, currently working on resolving a small issue with making that work

• Documentation added

• Some minimal test coverage ahead of launch, perhaps this week

Details in the tracker issue:

• https://github.com/Joystream/orion/issues/52

Atlas

• Fixes listed in the Atlas tracking issue -- specifically membership view, not being able to get correct count for NFT issued/ sold, resolved now; bugs wth filters and displaying NFTs in the membrship view) and other problems from the tracker issue
• cache seems hard to fix, perhaps ask Leszek to help with the cache part

ETA on resolving issues: few days, then about 1 week of testing.

Details in the tracker issue:

• https://github.com/Joystream/atlas/issues/3630

[Lezek123]

Documentation links:

• Orion v2 Developer guide: https://github.com/Lezek123/orion/blob/orion-v2/docs/developer-guide.md
• CHANGELOG: Orion v2 vs Orion v1: https://github.com/Lezek123/orion/blob/orion-v2/CHANGELOG.md

[dmtrjsg]

Release timeline

Release next week does not seem unrealistic.

Atlas

Everything is fixed now 👍 Testing is in active stage 👍 Notifications issue discovered before is now fixed (notiffs from all users coming to inbox) ✅ new Issue with NFT widget found, fixed ✅

❓ What to do with the sections, we have strange algos in Orionv1 like shuffling the order of the videos in the featured sections. Do we need to include this to v2? Yes, but it is unclear whether that can be replicated in v2. @drillprop, @Lezek123 can you pls add the details here as comments? If it is possible to take results + shuffle; plus unclear how to find out which entities to be shown on page 2,3. Solution to shuffle them on the client side.

Applies to

Homepage

• Popular videos Discovery:
• Popular videos
• Popular channels

⚠️ Remainder tests are due to be done in the next few days, possibly tomorrow by eod. (subject to Apps workstream)

Orion2

Pending: Apps as first class citizens mappings + one mutation ⚠️ <- top Prio. 1-2 days. Finish testing on

• https://github.com/Joystream/orion/issues/65 Operator guide (4h)

ℹ️ No operator queries yet, which allows to explore private entities and make channel and video reports. Not critical.

[dmtrjsg]

Release Approach Update

If we release Orionv2 after Apps and YPP the mappings have to be maintaned on both QN and Orionv2. So there's an argument to releases it together with YPP and Apps. Equally there is a risk of releasing wo Orion2 (like channel deletion etc..)

@attemka to partake in review of Bartosz' PRs and second pair of eyes would be beneficial.

After the data comparison the confidence in Orion2 is pretty high, so the consensus is leaning towards a joint release: YPP + Apps + Orion2.

Progress update

Atlas

Ground work is done. Testing every query was performed. Some small issues detected, can be dealt separately on Atlas side.

Specifically:

• custom queries for the sections requiring shuffling is done ✅ -no-cache ehancement ⚠️ Batching of queries, is it disabled on v2? @drillprop to share this with @Lezek123 to enable batching.. Not a big one, so if its not an easy setting, then it will be removed from the scope / postponed as queries are fast (max duration is 125 ms)

Other than above we are good to use Orion2 on ehpesus env for subsequent more testing.

Orion

Other priorities took the attention. Comparison script was almost finished to compare performance. Deleting the channels that have subscribers caused some problems; One issue with previous bid - for open auction wasn't set on Orion1. -> Decided to preserve the current behaviour in Orion2.

Tasks left before release: Orionv2 needs to include apps as members, due
Operator queries are out of scope for the first release. Update the changelog based on the findings of the benchmarking script. Also would be nice to compare the data from the mainnet (run orionv1 and orionv2 and compare the data)

• Small adjustments needed before the release. Spec version of the events need to be specified, so multiple versions of chain spec can be worked with.

All of the above should be possible this week, but taking time till next Tuesday in the light of Ephesus work/ reviews.

⚠️ it will be also possible to exclude the content from the platform (videos, nfts, channels). It is now done on the Atlas side. With Orion2 this feature can be removed completely. is_public_tracking should not be removed. is_censored will be automatically excluded;

⚠️ excluded videos are managed via operator mutations queries: (screenshot from changelog)

⚠️ ⚠️ Operator will not be seeing excluded content as a result of this managed on the Orion2 side. ☝️ To allow for the pessimistic approach where only approved videos will be displayed; this would require additional change on Orion and Atlas sides.

[dmtrjsg]

MM 14.02.23

 Atlas

Everything is done, awaiting Playground deployment for extensive testing Test scripts/ scenarios to be prepared @drillprop will take care of this ⭐

NB: Bathing is not supported on Orion2 no-cache is not implemented, but its a low prio enhancement:

• https://github.com/Joystream/atlas/issues/3786

 Orion

• [x] Orionv2 needs to include apps as members, due
• [x] Compare the data from the mainnet (run orionv1 and orionv2 and compare the data) + fixes introduced already
• [ ] ~Spec version of the events need to be specified, so multiple versions of chain spec can be worked with~ we didn't change any of hte events so not needed
• [ ] Update the changelog based on the findings of the benchmarking script.

Release

We need a playground which has: Apps branch Atlas+QN+ Master Runtime + Orionv2 + colossus/argus of master branch ⏭️ next week as we want App actions to be included to Orionv2 @attemka will try to do it next week with support from @Lezek123 and @mnaamani

Other notes

⚠️ Orionv2 improvements not added yet: @drillprop will raise the issues on Atlas side.

[dmtrjsg]

MM 28.02

Release

YPP release first with QN dependency. Orionv2 last tasks this week to be tested on the current PG with Orionv2 enabled. Releasable date circa next week, after YPP is released to Prod (after YPP is released).

Envs

@Lezek123 will deploy Orionv2 on same instance where current PG is, so we can test before release.

Atlas

• https://github.com/Joystream/atlas/issues/3821
• https://github.com/Joystream/atlas/issues/3877

Testing is almost complete, last two outstanding issues (work the same as in Orionv1).

We are currently using events query we, and recently these were updated to more performant ones. These need to be introduced to Atlas before release. @drillprop will raise issue and work on this in this sprint, the work is small, and notifications need to be retested (Half day).

Orion

Fix to be applied for outstanding issues in

• • https://github.com/Joystream/atlas/issues/3877 Operator queries now added Ephesus mappings -> removed but they are part of the history on the branch. It will be easy to branch out from this and continue working on the Ephesus. Apps: one change coming - remove apps deletion from Orionv2 and QN. Mutation on generatiing commitment now added to Orionv2 and being added to Orionv1. 🟡 Same signed message can be reused for channel and video creation. This can be updated in Joystream JS package, small change to protobuff package, and updated in Orionv1+v2.
• https://github.com/Joystream/joystream/issues/4672

YPP

YPP does not depend on Orion. It depends on QN only.

❓ we need to migrate queries from QN to Orionv2 for YPP?

• No, we do not have to rebuild YPP to work with Orionv2, so YPP can work with QN, while Atlas can work with Orionv2.

[dmtrjsg]

1️⃣ Objective: make Atlas concurrently testable for YPP+Apps; Orionv2.1; Ephesus branches

Tasks:

• Atlas-next env needs to be updated. @Lezek123 will create a PG with Orionv2 ephesus events. Since there were quite a few changes since last update, now it makes more sense to deploy totally new Playground. Orionv1 is not needed there.
• @drillprop's changes to Orionv2 ephesus version.
• @attemka will handle the Atlas side

__

2️⃣ Objective: Orionv2 release Resolved URLs is now implemented on both Orionv2 and Atlas.

• https://github.com/Joystream/atlas/pull/3562 -> needs to get merged and we are good to deploy to production. Review Orionv2 code -> @WRadoslaw light-reviewed it; @zeeshanakram3 to do a second review. Review Atlas changes -> @attemka reviewed it; @WRadoslaw to do a double run. Most viewed videos connection to be tested by @drillprop @dmtrjsg to test uploads and studio activities on Atlas here: ~https://atlas-44iurweas-joystream.vercel.app/~
• https://atlas-git-orion-v2-joystream.vercel.app/

[dmtrjsg]

Release

Aiming end of next week release of Orionv2; with YPP+Apps released early next week.

Atlas

Release branch is ready: https://atlas-git-orion-v2-joystream.vercel.app/ To test:

• [x] Channel reports; video reports. Reporting, checking status, and blocking content. @drillprop

Orion

Everything is ready for release.

• [x] State comparison of current state of the testnet in the QN (Orionv1) vs Orionv2 check; in case of any changes on mappings for QN and Orion @Lezek123 will add this as checkbox to the Orionv2 tracker. Will be covered this week.
• [x] Data migration: video views; channel follows. Complexity: Couple of hours. @Lezek123 to migrate prod stats to test env before release.

[dmtrjsg]

Release date: 23rd in the morning ⭐

Q: are we going to use the same node that we use for Orionv1? Or do we want to add more firepower for further boost in performance? Optimising for geo location?

A: decided to launch on the same machine, and boost the specs or migrate the instance to a diff node if required.

⚠️ @zeeshanakram3 to review this pr before the release:

• https://github.com/Joystream/orion/pull/49

[dmtrjsg]

Release date: was reviewed in the light of APPs + YPP release, decided to move to this week.

Atlas: Bartosz created PR on Assets mapping

• https://github.com/Joystream/atlas/pull/3987 (logging unavailable distributor to Sentry)

We need to merge the changes for Orionv2 on Atlas side to dev branch

• PR to be created and merged today or tomorrow. @attemka

Orion:

⚠️ @zeeshanakram3 to review this pr before the release:

• 49

to be done by end of today or tomorrow morning.

@Lezek123 run the script of data comparison between Orionv2 mainnet and QN

Data migration: there's an instance where the data was migrated and that was tested.

Release execution plan (steps):

On Thursday 11am CET we are starting to execute the release.

1. Atlas is set to maintenance mode @attemka
2. Create the backup of the data base and migration files @Lezek123
3. Deploy Orionv2 (merge Orioinv2 to master) @Lezek123
4. Atlas dev to master merge @attemka
5. Gleev update (code and env variables) @attemka
6. Excluded content (videos and channels) and add featured content @drillprop
7. Atlas is out of maintenance mode @attemka
8. Smoke test by @attemka @drillprop @dmtrjsg

Release comms will happen in Orion Channel.