bedeho
commented
4 years ago This Hydra team meeting had conclusions with bearing on the plan, will need to be updated later.
https://github.com/Joystream/hydra/issues/10#issuecomment-688324522
Closed bedeho closed 2 years ago
bedeho
commented
4 years ago This Hydra team meeting had conclusions with bearing on the plan, will need to be updated later.
https://github.com/Joystream/hydra/issues/10#issuecomment-688324522
bedeho
commented
4 years ago The following two meetings have an additional bearing on the final plan
Name
BabylonGoals
1. Launch Atlas 0.1
2. Deploy a new content directory and corresponding working group
3. Deploy first self-hosted Joystream Hydra node only supporting content directory.
Overview
This is a complex release with interdependencies between three major areas of technical improvement. These more or less have to come together like this for the release to be coherent. In what follows below we will cover each part of the release in terms of scope and current status of work.
Runtime
Team
Work Scope
Current Status
The organisation and function of the new content directory is described here:
https://joystream.gitbook.io/joystream-handbook/subsystems/content-directory
The content directory has been fully implemented & unit tested in up to date Substrate version and is currently being implemented into a full runtime.
Initial Division of labour
Arsen does everything.
Content Directory
Team
Work Scope
Schemas
We must define new schemas that will capture the new data model. One important change for example is that channels can now live directly inside the content directory, not externally in the working group as is the case today. Another important one is how much of the special display choices in Pioneer, like the landing screen content, can come from the chain, however that would require having schemas facilitating this. The actuals schemas should be constructed informed by a new schema standard (see below in tools section) and the final API the Atlas team is expecting
https://github.com/Joystream/joystream/issues/824#issuecomment-653150085
Seed Content
Given that we are clearing the directory, we need to seed the initial directory with some content so that the experience people will have from the get-go is somewhat representative of how the system will work eventually. This is of particular importance because publishing will be relegating to only be CLI based, which will require a native install step, and be much more complicated than any publisher will be used to. An additional point here is that we have to select some specific content to occupy distinguished salient places in the Pioneer front-end, and these may need non-standard high-quality media artefacts.
We have actually procured a list of a few thousand public domain movies that could be published, the only issue with this is that the image preview on these has a different aspect ratio to the one used in the current design. The two alternative approaches would be to
a. Have manual creation of new covers, for example by using stills from the videos. b. Adapt Atlas video previews to be aspect ratio of public domain videos.
Current Status
Nothing is done here.
Initial Division of labour
Hydra
Team
Clarification
We will refer to the framework as Hydra, and the usage of that framework for Joystream purposes as the query node.
Current Source & Issue Management
Hydra is being migrated out of a long-running branch in the monorepo and into this standalone repo in order to facilitate third parties using and discovering it.
https://github.com/Joystream/hydra/
This process is not complete, and there are also some ports on Dmitrii's personal NPM profile. This will all be consolidated.
All issues pertaining purely to Hydra should live in that repo, while issues pertaining to the query node should live in the monorepo with label
query-node. The query node itself, in the form of input schemas, mappings and anything else, will live in the monorepo as well.Work Scope
Dependencies
There will be no working chain with a runtime for at least a few weeks, and even then, there will be nothing to power the mappings, because there has been no usage. This means that the schema modeller and mapping author has to work in the blind. Once there is a functioning runtime, the work can be continuously integrated into the network integration tests, which will have some relevant transaction scenarios. Before this is ready, the only other option is unit tests on individual mappings.
Current Status
We have some progress on separation of databases, and also some form of deployment works, but everything else is basically at zero.
There is some very valuable pre-existing schema references that would be of great value to writing the query node:
a) Here is a early draft of input schemas for the content directory: https://github.com/Joystream/joystream/issues/644 They are not complete, and maybe slightly out of date with the underlying model, but a great starting point.
b) Here is a sketch of the API the Atlas team is expecting, roughly: https://github.com/Joystream/joystream/issues/824#issuecomment-653150085
Initial Division of labour
Atlas
Team
Work Scope
Current Status
Risks
Successful, and bug free, development, deployment and operation of Hydra is the biggest risk factor of this entire release. For example, there is an out of memory bug that was never fully accounted for, described here. For this reason, we should keep building on top of the mocked API as late as possible, at least it has become fully certain that, not only will the query node be ready and working well for us to rely on, but also that we will have time to do any inevitable API reconciliation that will be needed as a result of the currently mocked API not being identical.
Initial Division of labour
Klaudiusz, Francesco
Pioneer
Team
Work Scope
The general idea here is to not even attempt to reconcile Pioneer with the new content directory, it's not worth the effort given the complexity and pace of progress on Pioneer 2. This means a lot of stuff will simply be disabled, specifically
The media page can be re placed with a stats page that just shows a few key figures of importance, such as number of channels, videos etc, also perhaps lower level of abstraction, down to classes, entity counts, permissions, etc.
It probably is worth replacing the working group related logic in Pioneer, as we can pretty much 100% reuse what we have for storage now, as the system is identical.
Current Status
Nothing has been done.
Initial Division of labour
Leszek does everything.
CLI & Tooling
Team
Work Scope
Low level interactions
Low level interactions means allowing initiation of individual extrinsics in the content directory. These operate at the data and permission model of the directory, which is much lower than concepts such as "video" or "episodic content" or "playlists". One way to think about it is as having direct access to SQL database, rather than through a high level API over that database. These interactions are still useful for doing low level tasks, such as creating classes, new schemas, updating permissions, and other operational and working group related activities. Given the current very simple model we have of curation, that is basically just marking content as curated, this can also be done with as single direct property value updating. The CLI should have some set of
We had an old standard for describing in JSON files the prospective changes to be made to the content directory, it is now stale as the data model and permission system has been altered. The very nice part about it was that it allowed people to share representations of complex changes, such as adding new schemas and permissions, have them validated by the tooling. It's also a practical way to provide the very rich input to the CLI itself, rather than having to type complex input in an interactive sequence. Some subset of these standards would be of great use to retain.
High level interactions
If you want to make it at all easy to publish content to a channel, then we may want to have an explicit command for this that integrates the interactions with both the storage and content directory systems, and which perhaps done some simple input validation on things to make sure image links and storage size is acceptable. An alternative would be to just make sure that there is a specific command for uploading to storage, and then requiring publishers to read a tutorial to understand how to put it all together. We should delay deciding until we know more about the pace of progress in the release.
Current Status
There has been some work by Arsen on a new standard and updating the tooling, but largely nothing has been done.
https://github.com/iorveth/joystream/tree/cont_dir_json_schemas/content-directory-schemas
Initial Division of labour
Leszek does everything.
Network Testing
Team
Work Scope
We need to land an up front testing plan which explicitly covers what we attempt to get out of our testing. The main goal here is to extend the integration testing to encompass the query node. This is not only to make sure it works well in production, but it will also be critical for the development experience as well. Specifically, it will need to be adjusted in the following ways
Dependencies
There will be no working chain with a runtime for at least a few weeks, and even then, there will be nothing to power the mappings, because there has been no usage. This means that the schema modeller and mapping author has to work in the blind. Once there is a functioning runtime, the work can be continuously integrated into the network integration tests, which will have some relevant transaction scenarios. Before this is ready, the only other option is unit tests on individual mappings.
Current Status
Nothing has been done.
Initial Division of labour
Mokhtar does everything.
Deployment
Team
Work Scope
Current Status
Nothing has been done.
Initial Division of labour
Leszek
Mokhtar
Manual QA
Team
Work Scope
The testing here should be targeting Pioneer, CLI and Atlas. Pioneer should be quite easy to test, as we are mostly removing. CLI should be a bit more involved, as there will be some new non-trivial features. Atlas will also be very new experience to test, and here we may need to pay some attention to different browsers and screen dimensions during testing. Currently Atlas has next to no automated testing.
Current Status
Nothing has been done.
Initial Division of labour
Not determined.
Community and Communications
Team
Work Scope
I think we should announce this network much sooner than the last one we did, I (Bedeho) will personally make sure that any branding production delay will not prevent this. Some guides or tutorials will probably need to be updated.
Beyond the normal tasks associated with communications, it seems like something on the community side should complement the new stuff here. However, that system is still in flux as of this moment. Given the lag time involved before we can start executing on this part of the release, that uncertainty is probably not a big problem at this time. But setting aside time to address and execute this should be accounted for now.
Current Status
Nothing has been done.
Initial Division of labour
Not determined.
Project Management
Release Manager
BedehoThe release manager is responsible for tracking overall progress, detecting delays and course-correcting. Tracking will be done weekly in this issue
https://github.com/Joystream/joystream/issues/1242
as described by the following methodology
https://github.com/Joystream/joystream/issues/474
Release Branch
<missing>Github Project(s)
We are doing everything in one project.
https://github.com/orgs/Joystream/projects/28
The existing Hydra and Atlas projects have been closed, and we can open new ones once we establish new major project specific milestones independent of this release.
Team Meetings
Existing Meetings Continue
Work on Hydra and Atlas has commenced outside of the normal release pace, as a result the work has been coordinated primarily through focused weekly team meetings that are tracked here
We will continue with these, as they have been very successful at keeping us on track, unless we detect
Biweekly(ish) Release Meetings
A release wide meeting will be organised semi-regularly, we could start at a pace of once ever 2 weeks to begin with. Experience has shown that, even when there is no obvious issues that need tackling, discussions often reveal issues and corner cases that have not been accounted for. We can aim for one team rep. being present, so as not to distract everyone. Log will be kept here:
https://github.com/Joystream/joystream/issues/1241