Closed Lezek123 closed 1 year ago
I removed collaborators field from the Channel entity. This is mainly for simplification purposes and to reduce the initial scope of work, they can of course be added later if needed;
This is used for YPP project, and added to channels with single permission to add videos, used for the purpose of YPP auto sync feature.
searchChannels query (allows implementing custom channel search logic), searchVideos query (allows implementing custom video search logic)
What are the realms of possibility how search relevancy can be defined? Would it be possible to define it as mix of title/ description text match (fuzzy), date uploaded, nft minted, video views, channel subscribers defined as some sort of polynom?
I'm amazed we managed to get a working draft that fast. All the improvements like deeper filtering and especially nested field queries will make our lives much easier. Looks very promising
@dmtrjsg
This is used for YPP project, and added to channels with single permission to add videos, used for the purpose of YPP auto sync feature.
We only add collaborators, we don't actually read them from QN, so not a problem
MM from 13 Dec:
Status
Most mappings are done, some mappings are still WIP, NFT mappings are to be done
Features Round Up
v1 vverall objective is to relaunch Orion using the immediate benefits, and build a solid foundation for future expansions in the areas of Orion2 accounts, Notifications, Advanced Featuring, Advanced Search Relevancy, Scalability and Performance improvements.
Next Steps
MM Dec 20th:
Orion2 All mappings are now completed incl NFT Mutations are added to graph ql that were supported by Orion Adding support for custom categories No custom queries just yet.
Atlas Pr opened with new queries are re-written, some things are not working mentioned in the PR.
Next Milestone Prepare testable environment to test the functionality after all of the queries are re-written. Test environment would entail deployment of Orion2 pointing to Atlas dev.
Remaining effort on Orion2 side
50% spanning custom queries, testing and benchmarking.
MM: 10th January 2023
Finished the views. All the videos and channels should not appear, censored, filtered out will not be displayed by default Added mutation to set the supported categories by operator and it also supports settings whether videos with no categories will be supported, and recently added categories that did not exist before. Most viewed videos connection query added. Extended channels query added (ones with most views). Resolved custom parser issues.
Full list of the queries to implement are tracked in this issue:
🟢 Action: @Lezek123 to add some indicative timelines for the groups of tasks indicated in the tracker issue for better coordination with Atlas teams pace.
Small changes added after follows and views were added on the Orion2 side. Updated PR (as prev mentioned) with playground for Orion but haven't started testing latest changes. Memberships creation, uploading videos etc in the new integrated environment will be attempted today if time allows after Gleev launches.
Follow ups will be shared after testing.
🟢 Action: @drillprop will create the tracker issue similar to #52 to track the progress of testing and remaining effort.
Comments on scope:
⚠️ All queries for Atlas functioning are now complete 🥳 ETA on the first release now strictly depends on the outcome of manual testing. Positive scenario of first release estimates: in 2 weeks time subject to outcomes of manual testing 🔜
First release scope will not include
Operator queries of getting rich context on
Video views Channel/video follows Channel/video reports
Which is not blocking Atlas functionality and are designed for analytics purposes
Currently Lezek is working on the
On the Atlas side, no tests done yet in the light of other priorities last week, but will be the second priority after "Apps as Channels" work is done and unblocking Radek to progress with "Apps as first class citizens"
Manual testing tracker issue:
@Lezek123 we've concluded that Orionv2 scope needs to be increased to accommodate for the Apps feature:
There's an open question whether CLI interface needs to be built for Orion now to create channels, so that we don't duplicate this functionality in QN and Orion. Smth for you to advise us on. I'll share this in Discord and we can all discuss there.
Orion
New issues added:
Ephesus mapping for channel payouts + Direct channel payouts
Censorship functionality for GW operators (hide channels/ videos/ objects if censored)
Deployment options: Aquarium allows for simple deployments for operators, currently working on resolving a small issue with making that work
Documentation added
Some minimal test coverage ahead of launch, perhaps this week
Details in the tracker issue:
Atlas
ETA on resolving issues: few days, then about 1 week of testing.
Details in the tracker issue:
Documentation links:
Release next week does not seem unrealistic.
Everything is fixed now 👍 Testing is in active stage 👍 Notifications issue discovered before is now fixed (notiffs from all users coming to inbox) ✅ new Issue with NFT widget found, fixed ✅
❓ What to do with the sections, we have strange algos in Orionv1 like shuffling the order of the videos in the featured sections. Do we need to include this to v2? Yes, but it is unclear whether that can be replicated in v2. @drillprop, @Lezek123 can you pls add the details here as comments? If it is possible to take results + shuffle; plus unclear how to find out which entities to be shown on page 2,3. Solution to shuffle them on the client side.
Applies to
Homepage
⚠️ Remainder tests are due to be done in the next few days, possibly tomorrow by eod. (subject to Apps workstream)
Pending: Apps as first class citizens mappings + one mutation ⚠️ <- top Prio. 1-2 days. Finish testing on
ℹ️ No operator queries yet, which allows to explore private entities and make channel and video reports. Not critical.
If we release Orionv2 after Apps and YPP the mappings have to be maintaned on both QN and Orionv2. So there's an argument to releases it together with YPP and Apps. Equally there is a risk of releasing wo Orion2 (like channel deletion etc..)
@attemka to partake in review of Bartosz' PRs and second pair of eyes would be beneficial.
After the data comparison the confidence in Orion2 is pretty high, so the consensus is leaning towards a joint release: YPP + Apps + Orion2.
Ground work is done. Testing every query was performed. Some small issues detected, can be dealt separately on Atlas side.
Specifically:
no-cache
ehancement
⚠️ Batching of queries, is it disabled on v2? @drillprop to share this with @Lezek123 to enable batching.. Not a big one, so if its not an easy setting, then it will be removed from the scope / postponed as queries are fast (max duration is 125 ms) Other than above we are good to use Orion2 on ehpesus env for subsequent more testing.
Other priorities took the attention. Comparison script was almost finished to compare performance. Deleting the channels that have subscribers caused some problems; One issue with previous bid - for open auction wasn't set on Orion1. -> Decided to preserve the current behaviour in Orion2.
Tasks left before release:
Orionv2 needs to include apps as members, due
Operator queries are out of scope for the first release.
Update the changelog based on the findings of the benchmarking script.
Also would be nice to compare the data from the mainnet (run orionv1 and orionv2 and compare the data)
All of the above should be possible this week, but taking time till next Tuesday in the light of Ephesus work/ reviews.
⚠️ it will be also possible to exclude the content from the platform (videos, nfts, channels). It is now done on the Atlas side. With Orion2 this feature can be removed completely. is_public_tracking
should not be removed. is_censored
will be automatically excluded;
⚠️ excluded videos are managed via operator mutations queries: (screenshot from changelog)
⚠️ ⚠️ Operator will not be seeing excluded content as a result of this managed on the Orion2 side. ☝️ To allow for the pessimistic approach where only approved videos will be displayed; this would require additional change on Orion and Atlas sides.
MM 14.02.23
Everything is done, awaiting Playground deployment for extensive testing Test scripts/ scenarios to be prepared @drillprop will take care of this ⭐
NB:
Bathing is not supported on Orion2
no-cache
is not implemented, but its a low prio enhancement:
We need a playground which has: Apps branch Atlas+QN+ Master Runtime + Orionv2 + colossus/argus of master branch ⏭️ next week as we want App actions to be included to Orionv2 @attemka will try to do it next week with support from @Lezek123 and @mnaamani
⚠️ Orionv2 improvements not added yet: @drillprop will raise the issues on Atlas side.
MM 28.02
YPP release first with QN dependency. Orionv2 last tasks this week to be tested on the current PG with Orionv2 enabled. Releasable date circa next week, after YPP is released to Prod (after YPP is released).
@Lezek123 will deploy Orionv2 on same instance where current PG is, so we can test before release.
Testing is almost complete, last two outstanding issues (work the same as in Orionv1).
We are currently using events query we, and recently these were updated to more performant ones. These need to be introduced to Atlas before release. @drillprop will raise issue and work on this in this sprint, the work is small, and notifications need to be retested (Half day).
Fix to be applied for outstanding issues in
YPP does not depend on Orion. It depends on QN only.
❓ we need to migrate queries from QN to Orionv2 for YPP?
1️⃣ Objective: make Atlas concurrently testable for YPP+Apps; Orionv2.1; Ephesus branches
Tasks:
Atlas-next
env needs to be updated. @Lezek123 will create a PG with Orionv2 ephesus events. Since there were quite a few changes since last update, now it makes more sense to deploy totally new Playground. Orionv1 is not needed there.__
2️⃣ Objective: Orionv2 release Resolved URLs is now implemented on both Orionv2 and Atlas.
Aiming end of next week release of Orionv2; with YPP+Apps released early next week.
Release branch is ready: https://atlas-git-orion-v2-joystream.vercel.app/ To test:
Everything is ready for release.
Release date: 23rd in the morning ⭐
Q: are we going to use the same node that we use for Orionv1? Or do we want to add more firepower for further boost in performance? Optimising for geo location?
A: decided to launch on the same machine, and boost the specs or migrate the instance to a diff node if required.
⚠️ @zeeshanakram3 to review this pr before the release:
Release date: was reviewed in the light of APPs + YPP release, decided to move to this week.
Atlas: Bartosz created PR on Assets mapping
We need to merge the changes for Orionv2 on Atlas side to dev branch
Orion:
⚠️ @zeeshanakram3 to review this pr before the release:
to be done by end of today or tomorrow morning.
@Lezek123 run the script of data comparison between Orionv2 mainnet and QN
Data migration: there's an instance where the data was migrated and that was tested.
Release execution plan (steps):
On Thursday 11am CET we are starting to execute the release.
Release comms will happen in Orion Channel.
Orion V2 initial implementation draft
The draft implementation I'm describing in this document can be found HERE.
This repository is based on the current squid-substrate-template and contains all the input schemas, refactored Atlas queries, drafts of the custom graphql resolvers etc., as well as some basic setup and reference code to ilustrate some of the ideas of how certain issues can be addressed.
The official Subsquid documentation, which is often referenced throught this document can be found here: https://docs.subsquid.io/
How to run the local setup
I'll be explaining each part of the setup more deeply later in this document, but here is just a quick-start reference:
joystream-node
service in the Joystream repositoryRun and migrate the processor datbase
Run the processor
Run the GraphQL server
After performing those steps you should be able to go to![orion-graphql](https://user-images.githubusercontent.com/19978260/203102071-574097f7-6470-44c4-9ba3-53c904074e09.png)
http://localhost:4350/graphql
and see something like this:Currently the processor will produce some mock data on each block so you can also test some of the existing queries:![orion-query](https://user-images.githubusercontent.com/19978260/203102088-a23cb357-2e26-479c-928c-fb5c2b0125ae.png)
On-chain data indexing and processing
Squid Archive
Squid Arachive is analogous concept to the Hydra Indexer, it uses the Joystream node websocket rpc endpoint to fetch data about on-chain events and extrinsics and store it in a relational database (PostgreSQL).
We can configure the archive via a docker-compose file located in
archive/docker-compose.yml
.The current squid archive configuration is using a local Joystream docker node (
ws://joystream-node:9944
) that runs onjoystream_default
network as a source.SubstrateBatchProcessor
SubstrateBatchProcessor
is a class we use to instantialize the events processor. As opposed to Hydra, where we would only implement the "mapping" functions (or "mappings"), Subsquid let's us instantialize and programatically configure the processor ourselves (manifest.yml
file no longer required), which gives us more controll over its behavior.SubstrateBatchProcessor
is just one of the many processor implementations available in Subsquid, but it's the one currently recommended for processing substrate events and extrinsics. This specific processor implementation queries all blocks along with the events of interest from the Squid Arachive (using the@subsquid/substrate-gateway
service). The maximum number of blocks in a single batch currently depends on the@subsquid/substrate-gateway
implementation, it's still a little unclear how this will work in the future, but currently there are two main components that affect the batch size:Current processor implementation:
In the current draft implementation:
new TypeormDatabase({ isolationLevel: 'READ COMMITTED' })
instance (which is then used to insert data into the database). As you can see we can easily provide config forTypeormDatabase
here, which we take advantage of by specifingisolationLevel: 'READ COMMITTED'
. This isolation level reduces the possibility of conflicts, since the database state will be modified by both the processor and through the external api (ie. counting video views, featuring, reporting video/channel etc., as explained further down below);System.ExstrinsicSuccess
event;Content.VideoCreated
andContent.ChannelCreated
among events of interest for ilustration purposes.This impelementation provides a decent general overview of how the "mappings" are written in Subsquid and how one can extract the events&data of interest from a batch and then perform bulk inserts/updates at the end of processing a batch, which considerably increases the performance.
The API
Input schema
The current input schema files can be found here: https://github.com/Lezek123/subsquid-orion/tree/main/schema I tried to preserve a similar schema to the one we currently use in Hydra, however there are some notable differences:
collaborators
field from theChannel
entity. This is mainly for simplification purposes and to reduce the initial scope of work, they can of course be added later if needed;EventData
union instead. The result can be seen here. I will explain the differences that come from this change further down below;nested field queries
, I removed some redundant entity relationships etc., which I assumed were mostly serving as a workaround for the lack of those features before;nftOwnerMember
,isNftOwnerChannel
,nftOwnerCuratorGroup
in the NFT entity with aNftOwner
union for better clarityChannelReport
etc. Some of those entities will probably be moved away from the input schema, unless we want to take advantage of the autogenerated queries that Subsquid will provide for them. If not - we can just use custom models instead (as described further below).Video
andChannel
now include newfollowsNum
andvideoViewsNum
counters, as Atlas relies on being able to execute queries that include sorting based on those values, as well as recieving those values as part of the result set. Having that in mind, I decided that the cost of introducing those additional fields is relatively low compared to the efficiency&simplicity benefits, however some custom aggregation queries (like sorting based on number of follows within given time period) will also be needed, as will be explained further below.activeVideoCounter
fields have been removed, as this data is now accessible through extended queries. See Customtype-graphql
Resolvers section for more details;createdAt
andupdatedAt
fields are no longer automatically added, so in some cases I included them in the input schema (forEvent
entity however I decided to name the fieldtimestamp
instead);Many-to-Many
relationships are not supported in Subsquid so they were refactored to 2-sideMany-to-One
relationships with a specific "join entity".Custom models
Subsquid comes with a nice directory structure alowing us to define our own TypeORM models separately from the autogenerated ones, however they will all become part of the same database.
Use cases:
The primary use-case for definig those custom models is when we don't want Subquid to autogenerate the public api endpoints for quering certain (private) data, but we still want to keep this data as part of the same database to take advantage of the relational model. Take
User
entity for example. We want to be able to connect users with channels throughUser
>-ChannelFollow
-<Channel
relationship, but we don't necessarily want to expose anyUser
data through the api, that's why we define custom models forUser
andChannelFollow
, but we don't include those entities in the input schema.Custom GraphQL api extensions
Subsquid allows use to add some custom extensions to the autogenerated GraphQL api. Those are stored in the
src/server-extension/
library and constitute a significant part of the project.Custom
type-graphql
ResolversCustom
type-graphql
resolvers are classes where we can define our custom GraphQL queries, mutations and subsriptions that will then be included in the final API.Normally we run a Subsquid graphql server using the
@subsquid/graphql-server
library/service, which generates and runs a GraphQL server based on the input schema. For the purpose of generating the final ("output") schema and resolves it uses another library called@subsquid/openreader
. The schema generated by@subsquid/openreader
is then merged with the schema generated from our custom resolvers that we are providing insrc/server-extension/resolvers
. For this merge, themergeSchemas
method fromgraphql-tools
library is used.The interesting property of
mergeSchemas
is that this method also merges all individual GraphQL types defined in both schemas, which makes us able to reuse the autogenerated types likeVideo
,VideoWhereInput
,VideoOrderByInput
etc. All we have to do is define a graphql object with the same name in our resolvers space and at least one property which matches with the autogenerated object (for entities it can be, for example,id: string
). Then when the types are merged, we will get a consistentVideo
object with all the expected properties in the final schema.This can be probably better understood by looking at the implementation inside https://github.com/Lezek123/subsquid-orion/tree/main/src/server-extension/resolvers, especially https://github.com/Lezek123/subsquid-orion/blob/main/src/server-extension/resolvers/baseTypes.ts where the "placeholders" for the to-be-autogenerated types are defined.
There are also many other useful references in this directory:
extendedChannels
query to ilustrate how we can take advantage of the subsquid libraries to define queries that build "on top of" the autogenerated queries likechannels
. In this example implementation we make use of the autogeneratedChannelWhereInput
,ChannelOrderByInput
types and translate the client's request to a corresponding SQL query, same way it would've been done by thechannels
query. However, on top of this, we're able to include some custom, additional fields (like, in this case, thecount()
of videos that meet certain criteria in each channel) and allow the client to filter and sort the results by those fields (they can also be included in the results set if needed). In this specific implementation I showed how we can get rid ofChannel.activeVideosCount
field, which would otherwise need to be constantly updated on many different occasions, by replacing it with a subquery that can be ran for each queried channel upon client's request instead;Use-cases summary for custom resolvers:
extendedChannels
query (allows querying channel along withactiveVideosCount
aggregation)extendedVideoCategories
query (allows querying video categories along withactiveVideosCount
aggregation)mostRecentChannels
query (a query that allows filtering and ordering results among X most recent channels)channelNftCollectors
query (allows querying the list of members who collected the highest number of nfts issued by a given channel)searchChannels
query (allows implementing custom channel search logic)searchVideos
query (allows implementing custom video search logic)mostViewedVideosConnection
query (allows querying videos with the highest number of views in a given time period)getKillSwitch
query (allows retrieving the current Atlas "killSwitch" status)videoHero
query (allows retrieving information about content currently featured in the Atals Hero section)followChannel
unfollowChannel
reportChannel
reportVideo
addVideoView
setKillSwitch
(operator only)setVideoHero
(operator only)processorState
(allows Atlas to stay updated about the current processing state, similar to the Hydra'sstateSubscription
)checkRequest
pluginThe
checkRequest
plugin is a Subsquid feature that allows us act on the Apollo server's requestDidStart event. The handler function can be implemented insidesrc/server-extension/checkRequest.ts
and recieves information like request headers, ip of the origin, all the data specific to the graphql request etc.The current example implementation shows how this plugin can be used to introduce some authentication for all
mutation
requests.Use cases:
setKillSwitch
,setVideoHero
etc., however, as described in the Known issues section, I believe this is a suboptimal way of doing this.Atlas queries: Refactored!
I refactored all existring Atlas queries (https://github.com/Joystream/atlas/tree/master/packages/atlas/src/api/queries) to match the new schema.
The results can be seen here. The directory structure matches the one in the Atlas repository, which makes it easy to do side-by-side comparison. I also added
CHANGE:
comment in all places where the changes where introduced.The most notbale changes can be observed in the notifications / events queries, due to the refactorization of Event entities. It is now easier to query all the events of interest together and apply filtering, sorting and limit on the results of one query instead of making separate queries for each event type and then post-processing the results client-side.
Some other notbale changes include:
entityId
, it had to be replaced withentity.id
, as Subsquid doesn't support the former syntax anymore;ID
graphql type id had to be replaced withString
, as Subsquid doesn't support the former anymore;entityByUniqueInput
query used it had to be replaced w/entityById
, asentityByUniqueInput
is not supported in Subsquid. For members, which used to be queryable by handle, we can now either add a custom query or use the existingmembers
query (providing handle in thewhere
clause);event.data.bidder
, if it can be also accessed through, for exampleevent.data.bid.bidder
) were removed, so filtering now goes more deep in some cases;nftOwnerMember
,isNftOwnerChannel
,nftOwnerCuratorGroup
were replaced withNftOwner
union;admin
=>getKillSwitch
);activeVideosCounter
were changed toextendedChannels
/extendedVideoCategories
queries;category.featuredVideos
relation;top10Videos
were replaced with a little bit more generic/customizable queries likemostVievedVideosConnection
;Many-to-Many
relationships no longer being supported.Custom migrations: Setting up the database
Subsquid allows us to generate database migration files that we can then use to setup the processor database. Besides that, we can also specify some custom migrations that will be ran before or after the generated ones. In the draft implementation I introduced 2 custom migrations:
Views
andIndexes
(since the filenames and class names need to include timestamp I've just choosen some arbitrarly high values to make sure those migrations are always ran after the autogenerated one)Use cases for custom migrations:
jsonb
fields or expressions, which is not possible through the input schema. This is useful when dealing with unions where some of the variants include a reference to another entity, like the newEventData
union.channel
can be replaced withchannel
view, which allows us to filter out certain channels from the results of any autogenerated query. In the draft implementaton I usechannel
view to exclude moderated channels, this way Atals doesn't need to worry about including this filter in each query, the censored channels are also hidden from anyone trying to query the server directly. If thechannel
gets "unmoderated" however, it will instantly re-appear in the results (which wouldn't be possible if we just deleted moderated channels permanently).Performance
Using the mocked data I did some performance tests against the current implementation, here are some results:
GetExtendedBasicChannels
queryArguments:
Number of channel entries: 12,921 Number of video entries: 257,400 Time to execute the query: 86ms
GetNotifications
queryArguments:
Number of event entries: 2,574,000 Time to execute the query: 880ms
GetNftHistory
queryArguments:
Number of event entries: 2,974,000 Time to execute the query: ~9 seconds (!)
Potential candidate for optimalization
Benchmarks to be continued...
Known issues and unresolved questions
Context
provided by@subsquid/graphql-server
is very limited, for example, we cannot access client's IP address or any request headers inside the graphql resolver. This is problematic, as it makes authentication more complex (we have to use the separate "checkRequest" plugin) and also makes it difficult to handle mutations likeaddVideoView
, where such data would be necessary to prevent abuse.Alternatives to consider
"Manually" setting up graphql server instead of using
@subsquid/graphql-server
To have more control of the setup we can run the graphql server from within our own codebase instead of using the
@subsquid/graphql-server
, we can still take advantage of@subsquid/openreader
however to generate the initial schema and resolvers.Pros of this approach:
Context
issue, so we can more easily implement our own authentication, authorization, rate limits and other restrictions.@subsquid/graphql-server
changes in the future in a way that no longer supports the current assumptions.Cons: