search

Joystream / joystream

Joystream Monorepo
http://www.joystream.org
GNU General Public License v3.0
1.42k stars 115 forks source link

Babylon: Create initial, consumable joystream-js library #1396

Open Lezek123 opened 4 years ago

Lezek123 commented 4 years ago

During the last few weeks/months I was aiming to extract some of the potentially reusable functionality from Pioneer's packages to pioneer/packages/src/joy-utils. This process began with https://github.com/Joystream/apps/pull/465 and the main idea is also further described in that PR.

I think Babylon release may be a good time to extract most of the code in joy-utils into the first version of long-awaited joystream-js library.

Benefits

Benefit 1: Transports

Currently the main benefit of doing this is that it will make the so-called "transports" (see: https://github.com/Joystream/joystream/tree/iznik/pioneer/packages/joy-utils/src/transport) accessible for all our JavaScript projects (Pioneer, CLI, storage node, status server etc.).

Basically all of those projects currently have some custom wrapper class/classes for the api to simplify fetching, combining and parsing commonly used data. Having this normalized across projects could greatly simplify adapting them after runtime changes that affect the api/types (which currently generates a need to update the same / similar query scenarios in all projects separately).

Some examples of data querying scenarios and how they are simplified via transports:

There are lots of other similar common scenarios that are already covered by the transport classes inside joy-utils (some of them very simple, like querying timestamp based on block number)

Other benefits that shared transports provide:

Benefit 2: Common constants

There is still some data that isn't exposed by the chain, but can be very useful (or even required) by some apps. Some of it is already hardcoded inside: https://github.com/Joystream/joystream/blob/iznik/pioneer/packages/joy-utils/src/consts

This includes data like:

Benefit 3: Types and typescript helpers

All the types used by the library (ie. interfaces for objects returned by the transport etc.) are available inside: https://github.com/Joystream/joystream/tree/iznik/pioneer/packages/joy-utils/src/types.

In the future I see the potential of also creating generic utility types that can be used by other projects, ie. SimplifiedTypeInterface from https://github.com/Joystream/joystream/blob/iznik/pioneer/packages/joy-utils/src/types/common.ts can be extended to allow converting complex structs like OpeningPolicyCommitment into simple, but still type-safe objects that will be accepted and correctly converted into types expected by the runtime via @polkadot/api (it already does that, but doesn't provide the TS interface)

Benefit 4: Other utility functions

joy-utils currently also contain a few other, generic utility functions in https://github.com/Joystream/joystream/tree/iznik/pioneer/packages/joy-utils/src/functions, ie.:

Benefit 5: A place where we can keep shared tooling, validation, schemas, scripts etc.

There is currently no obvious place to keep reusable scripts, json schemas etc. that can be useful for all of our JavaScript projects.

Once we have joystream-js library we can expand it to contain as much of the reusable code as possible (or later split into a few more specific libraries) in order to keep it in one place and reduce the amout of work needed to update our JavaScript projects one-by-one.

We can also for example:

To sum it up, I think there is a lot of future potential beyond what's currently available in joy-utils and some of it can be leveraged even during the current release (ie. the introduction of new content directory schemas)

Implementation

For this release my main goal is to factor joystream-js out of joy-utils and make is publishable and consumable as a standalone library. joy-utils package is currently still dependent on Pioneer and the monorepo setup and it's hard for other projects to take the benefit of it.

Babylon release also seems like a good time to normalize workingGroups transport. The main blocker to do this before was that joy-roles package in Pioneer needed to handle both contentWorkingGroup and storageWorkingGroup (which currently have quite different logic), while CLI and proposals system in Pioneer only needed to deal with storageWorkingGroup (which is part of the updated workingGroup runtime module). Now that this will be normalized, it makes sense to also have one "transport-layer" shared between Pioneer (joy-roles, joy-proposals), CLI and possibly status server.

The last benefit of having joystream-js during Babylon is that we can use it as a place to keep new content directory schemas and possibly some functionality related to those (ie. to make tasks like creating a channel, uploading a video etc. easier for other projects in the future)

Other projects

Having access to a consumable joysream-js library may be useful for projects like storage-node etc. too during this release. Some of the reusable code can also be moved to joysream-js from cli and storage-node (ie. transactions batching) to share the benefits of worked out solutions and move faster in the development, but as mnaamani once pointed out, this can be step-by-step process and I don't think it's feasible to do everything at once during this release.

Documentation

In the future we can use tools like http://typedoc.org/ to create a more clear documentation with examples etc. and provide an easy way for the developers to start building tools for Joystream. During this release I will try to create some very basic documentation to explain what's possible (as I already partially did in this issue)

Scope of this issue (Babylon)

┆Issue is synchronized with this Asana task by Unito

bedeho commented 4 years ago

A few reactions

  1. I suspect almost all the transport code related to reading will be displaced by the query node, and as such has no long term future unless we want to maintain a separate codebase for apps that try to be powered directly be a full node? This is not entirely without merit, in particular if light client validation can be added, as it means the query node trust problem is sidestepped. However, I don't think we can justify maintaining and adding additional such reading transport code without very specific goals for this, so I am sceptical.

  2. I think the transport code does not do any writing/extrinsics?

  3. The most important thing we have to settle is: what is the role of joystream.js?

Is joystream.js the canonical developer library when they want to write an application for the Joystream Network, or is it just an internal library? The name certainly seems to suggest the former, and if not, then I think we at least need to perhaps change the name in order to make space for a future application developer facing library.

Now, what do I mean by applications for the Joystream Network? By that I mean that it gives a very high level of abstraction for talking ot the network as a whole, for example in Atlas/Pioneer/CLI, so it will allow you to talk to query nodes, storage nodes, full nodes, add serving nodes and other future service infrastructure, through one unified library. It would allow you to generate keys for roles in a way that follows the Joystream wallet standard, so that you can recover all your role keys later with one seed, etc.

If we take this as the role of joystream.js, could it then at the same time be the library used by the internals of the system, like the storage or query node?

Lezek123 commented 3 years ago

Issue covered partially in https://github.com/Joystream/joystream/pull/1572 due to complexity of updating Pioneer's joy-roles app and the need for additional testing it would generate: