Giza Tracking Issue

[bedeho]

Purpose

The purpose of this issue to track overall progress on the storage system in the Giza release. This work is complex, in particular in the in that it introduce two new nodes, coordinated by a new large runtime module, and which have to interact in large numbers with each other. It will be critical to have the lead developers on this system to stay tightly synchronized, and keep other people depending on their work (e.g. the release manager, devops, etc.) updated on their progress.

[bedeho]

Meeting Date

Monday, July 26th, 2021.

Participants

• Leszek
• Shamil
• Bedeho
• Mokhtar

Weekly Summaries

• Runtime: no summary first week
• Storage Node: no summary first week
• Distributor Node: no summary first week

Agenda

There was no explicit agenda for the first meeting, the goal was to try to figure out what was needed in order to nail down the scope for the Giza release plan.

Conclusions

• Here is a brief summary of where we were
  • Shamil has largely completed the runtime portion of the storage system, and has given Leszek sufficient clarity about how some remaining parts pertaining to distribution will work. The major outstanding task is to introduce the synchronisation logic which brings the locally stored data in the node in line with its on-chain obligations
  • Leszek has made good initial progress on a distributor node which implements a simple least recently used caching policy, and the query node in order to read current obligations.
• Both nodes will use a filesystem folder as the abstraction for where the data objects live, and then node operators can choose to virtualise whatever they want into this, be it an actual local folder, a network drive or S3.
• We discussed not having the storage nodes actually hold any data directly, and instead use S3, but opted to not go down this path because it couples the storage node very tightly with a particular cloud storage vendor, and also makes it difficult to later introduce the kind of access policy and monitoring required for introducing incentives for sharing data to other storage providers and distributors.
• For logging we decided to use the Elastic Stack, and it's accompanying solutions for dashboards and visibility. Logging must serve the dual purpose of being used to for ex-post forensics in failure scenarios and also for public dashboards with up to date stats that are useful for the operator.
• Caching in the distributor node should not only depend on what was least recently used, but also bias towards smaller data objects. The rationale for this conclusion was that there are a large number of small assets who's latency would significantly impact the UX, thus making sure they are highly available would be very valuable, and also not costly storage wise. The canonical example here would be video preview images.
• There will be no incentives, or even authentication, in the communication layer between storage nodes and distributors for Giza. It's easy to roll out later.
• The metadata for distribution buckets should be able to capture something that would make it possible for apps to decide what bucket is likely to have the lowest latency for a given bag.

Todo

• Bedeho should prepare a summary for everyone to review.
• Leszek will come up with a metadata standard for distribution families.
• We need to have a second meeting, at least, to scope out an actual release plan for Giza, still too many moving parts.

[bedeho]

Meeting Date

Monday, Aug 2nd, 2021.

Participants

• Leszek
• Shamil
• Bedeho
• Mokhtar

Weekly Summaries

Filled in just before meeting

• Runtime: no summary first week
• Storage Node: no summary first week
• Distributor Node: no summary first week

Pre-meeting deliverables

In order to get synchronised on two important topics, it would be good if Shamil and Leszek could prepare two deliverables no later than Monday midday, then we can use that for the meeting the same day

• State APIs: It's important, both for the operator, the lead, other providers and end-products, to have rich APIs that expose the current state of a node. By this we mean insight into all of the ongoing processes in the node, in terms of synching, downloading, cleanup, etc. For example, a big problem currently is that Atlas has a hard time understanding what the stat of an upload is in the interim between completion and confirmation on chain. This has caused a lot of problems. In order to make sure our APIs are robust enough, we should have detailed proposals for the relevant APIs on each node. If there are parts of the design that are explicitly known to only be for the short term, and not ideal for mainnet, then make this clear.
• Chain Synchronisation: A major source of problems in our current system is the lack of sophistication in how, exactly, the storage node manages to keep its internal data storage in synch with what the chain claims it is obligated to do. There is both issues in terms of speed when bootstrapping a new node, but also just keeping up on an ongoing basis. In the interest of making sure we are doing this well in v2, let's have some reasonably precise description (e.g. state machine) of how each node is currently being designed to achieve this. It should be clearly explained what data source (query node, filesystem, local state, full nodes reads, full node events) is driving a given part of the process. If there are parts of the design that are explicitly known to only be for the short term, and not ideal for mainnet, then make this clear.
• Weekly Summaries: In order to facilitate everyone understanding where our progress is at any given time, lets have Leszek & Shamil prepare a weekly progress summary by end of day Friday, which can be used to drive the agenda for any meeting next week. It should cover what was achieved since the last summary, and what are currently known outstanding tasks. We should have these for every meeting, unless otherwise explicitly agreed.

Agenda

We discussed lots of topics the first meeting, but we have still not settled enough to have a proper release plan. The primary purpose of this meeting is to try to nail down any outstanding issues for us to be able to prepare such a plan.

• Discuss whether we can start to use only data object id, or object id for short, as the way to identify data objects, rather than hashes, or content id. The latter may change for example when commitment schemes change, or if we somehow want to allow changing content under an object in the future, and it is in general harder to practically use with a running chain, because it the chain uses object ids.
• Does our current uploader authentication step involve providing a proof to the liason that the upload was completed? otherwise the operator of any bucket for a given bag can claim to confirm, and claim the possible reward.
• Review State APIs: Come to shared understanding and agreement.
• Review Synchronisation Designs: Come to shared understanding and agreement.
• Giza:
  • Documentation: how will Atlas, testing devs or anyone else understand how the system works?
  • How will migration work?
  • Integration testing scenarios: ?
  • Benchmarking: ?
  • CLI: ?
  • Hydra: ?
  • Atlas: ?
  • Logging: ?

Conclusions

• Deliverables
  • State Api
  • Sync algorithm description
  • Distribution Node
• We agreed to use object ids consistently as identifiers.
• We agreed to not use authentication upon upload initiation or completion, hence storage providers can steal
• Storage nodes will only move objects into filesystem once they have been fully authenticated, including downloads from other storage providers.
• We agreed some sort of lookup API is needed on storage nodes, both for synching with other storage nodes and distributors, in a reliable way. We need to settled on exactly how this will look, as it is a dependency for the distributor.
• We need to ensure that there is some practically workflow for easily increasing the capacity of a storage node without interruption.
• Some kind of chunking screen can help optimise synch times, and also relates to the ability to do streaming authentication of synchronisations & uploads, and also to the proof-of-storage scheme thay may be introduced.
• We should have meetings very frequently in the coming days to narrow down the many moving and interacting parts, as we also did not nail down most of the open questions about Giza either.

Todo

• We need another meeting quickly.
• Bedeho will review Distribution Node, and in particular current design on how to manage cache.

[bedeho]

Meeting Date

Wednesday, Aug 4th, 2021.

Participants

• Leszek
• Shamil
• Bedeho
• Mokhtar

Agenda

Nail down the scope for the Giza release plan.

Conclusions

• Mokhtar will be reviewer for both nodes, and each maintainer will also review the other node.
• Leszek is taking off for two weeks starting Mon 16th.
• We we will keep doing weekly meeting to stay synched.
• We will not do any sort of chunking to make synchronisation and upload verification more efficient.
• Storage nodes will have API which provides visibility about what objects and bags the node has. Shamil will use it for internal synching, as well as the storage lead and other analytics tools, but Leszek does not need it. It will have the following endpoints
  • Get all data objects ids you have
  • Get all data objects you have from my list
  • Get all data objects you have in this bag
  • Get all data objects you have from my list of bags
• All CLI work will remain in the individual node CLIs being built, which means that each of the leads will have to use this node CLI, not the joystream-CLI.
• We will not do on-chain migrations, as its fundamentally not feasible. Hence, we will need to rebuild the content directory and data object index using setup scripts that run right after the upgrade goes live.
• New integration code is needed between the content directory and the new storage system.
• We will introduce an endpoint in both nodes for event log streams that we will use with the Elastic search stack. Node operators will have to look directly at these logs for forensics.

Todo

• Mokhtar will define the precise set of integration testing and benchmarking automation we will introduce, base don upcoming developments in our DevOps infrastructure.
• Bedeho will be responsible for authoring public overarching technical docs as part of GitBook.
• Bedeho will plan changes needed to the content directory and the runtime to facilitate the rebuilding scripts.
• Bedeho will prepare release plan draft.

[bedeho]

Meeting Date

Friday, Aug 13th, 2021.

Participants

• Leszek
• Shamil
• Bedeho
• Mokhtar

Agenda

• Review TODO from last meeting
  • Mokhtar will define the precise set of integration testing and benchmarking automation we will introduce, base don upcoming developments in our DevOps infrastructure.
  • Bedeho will plan changes needed to the content directory and the runtime to facilitate the rebuilding scripts.
  • Bedeho will prepare release plan draft.
• Review Giza release plan, is it complete? Do we have a clear target? With particular emphasis
• How to make storage nodes quickly understand newly introduced storage obligations, so that upload initiation latency is minimal.
• Any findings Leszek has from his preliminary CDN background research.
• What is the MVP setup for someone to start actually testing this on a staging network.
• How to work around Leszek being away next 2 weeks.

Conclusions

• Mokhtar will define the precise set of integration testing and benchmarking automation we will introduce, base don upcoming developments in our DevOps infrastructure.
• Decisions
  • Both nodes should accept assets even when mime types cannot be determined, and the mime type check should only inspect the first, and possibly also last, 4kb chunks.
  • For Giza we do not introduce any proposals to control storage system, its done by lead actions.
  • Storage working group already exits, and we will keep it across the upgrade.
  • We will make it possible to start a colossus or argus instance in a Kubernetes cluster.
  • Shamil & Leszek will review the plan for testing that Mokhtar prepares.
  • We postpone automated benchmarking, and instead use manual benchmarking during development and manual QA.
  • We should update query node to Hydra v3.
  • We will be introducing integration test for the content directory for the first time, they were not added in Sumer.
  • Arsen or Ignacio to work on the runtime instead of Mokhtar, alternatively Arsen can work on the tests, Leszek can assists with tests, runtime & tooling.
• Its not clear whether we can get an MVP setup working while Leszek is away, but Mokhtar will try, it requires some Hydra relationship bug workarounds which are not trivial.

Todo

• We will tackle these agenda items in a future meeting, we had to prioritise Leszek related issue as he is leaving for a break
  • How to make storage nodes quickly understand newly introduced storage obligations, so that upload initiation latency is minimal.
  • What is the MVP setup for someone to start actually testing this on a staging network.
  • How to work around Leszek being away next 2 weeks.
• We identified a set of questions that must be reviewed and tackled at a later meeting.
  • Should it be possible to upload to unserviced bag, or to create a bag which does not have storage and/or distribution? This has some bearing on the bootstrapping process for Giza network. A related question is: should we block removal of storage buckets storing bags if that dips below some critical value, such as 1 or current replication factor.
  • Should we make the runtime block block third party publishing during bootstrapping?
  • Should Colossus accept uploads for objects which are known to be confirmed on-chain, and in those cases not issue an on-chain confirmation. this is thought of as a recover feature, for example during a failed bootstrap?
  • Should the tests start to use the CLIs, or keep working as libraries?
  • Are we risking speed problems associated with resolving data objects, since we have to resolve a few layers of relationships on the query node side?

[bedeho]

Meeting Date

Tuesday, Aug 24th, 2021.

Participants

• Shamil
• Bedeho
• Mokhtar

Agenda

• Review Todo from last meeting
• What to merge into Giza release on runtime?
  • comments & subreddits
  • channel payouts
  • NFTs
  • Core Giza fixes
• Review Division of labour proposition for Giza.
  • Ignazio does the content directory stuff.
  • Mokhtar & Shamil & Leszek split the integration testing.
• What is required to get a working network going?
• Review test plan: https://github.com/Joystream/joystream/issues/2590
• Some subset of these technical questions
  • How to make storage nodes quickly understand newly introduced storage obligations, so that upload initiation latency is minimal.
  • Should it be possible to upload to unserviced bag, or to create a bag which does not have storage and/or distribution? This has some bearing on the bootstrapping process for Giza network. A related question is: should we block removal of storage buckets storing bags if that dips below some critical value, such as 1 or current replication factor.
  • Should we make the runtime block block third party publishing during bootstrapping?
  • Should Colossus accept uploads for objects which are known to be confirmed on-chain, and in those cases not issue an on-chain confirmation. this is thought of as a recover feature, for example during a failed bootstrap?
  • Should the tests start to use the CLIs, or keep working as libraries?
  • Are we risking speed problems associated with resolving data objects, since we have to resolve a few layers of relationships on the query node side?

Conclusions

• Anything that will be part of Giza needs to have master merged into it, in particular if it's older.
• We will not include any non-essential work in Giza, we will instead create a new unification branch Suzdal, which includes all of the new content directory work, and which has within it the combination of Giza branch and master, so creating this combination will be an important first step. This gives
• Mokhtar & Shamil & Leszek split the integration testing.
• Testing correctness of CLI itself will be done in standalone tests later.
• The upload tool will try to leverage the existing publishing tool for Sumer.
• This should be big focus area when Leszek is back: What is required to get a working network going?
• We did not cover any of the tech questions, no time!

Todo

• Cover tech questions in future meeting(s).
• Mokhtar will prepare all testing issues, with initial estimates and assignments, can be adjusted later.
• Shamil will prepare comment in the chat about how he believes a cluster based storage node is feasible with little effort because storage nodes have little state.

[bedeho]

Meeting Date

Tuesday, Aug 30th, 2021.

Participants

• Shamil
• Bedeho
• Leszek

Agenda

• Review Todo from last meeting
• When can we have master merged into Giza done?
• What is required to get a working network going?
• Some subset of these technical questions
  • How to make storage nodes quickly understand newly introduced storage obligations, so that upload initiation latency is minimal.
  • Should it be possible to upload to unserviced bag, or to create a bag which does not have storage and/or distribution? This has some bearing on the bootstrapping process for Giza network. A related question is: should we block removal of storage buckets storing bags if that dips below some critical value, such as 1 or current replication factor.
  • Should we make the runtime block third party publishing during bootstrapping?
  • Should Colossus accept uploads for objects which are known to be confirmed on-chain, and in those cases not issue an on-chain confirmation. this is thought of as a recover feature, for example during a failed bootstrap?
  • Should the tests start to use the CLIs, or keep working as libraries?
  • Are we risking speed problems associated with resolving data objects, since we have to resolve a few layers of relationships on the query node side?

Conclusions

• We will merge master into Giza after Shamil has incorporated review comments from Leszek, and Mokhar will provide his review at a later time when he is available.
• Leszek will do query node workaround for broken m-2-m relationships querying in order to unblock us.
• What is needed in order to launch test network, will be added to release plan:
  • merge distributor & query node into Giza
  • storage node synch part ... depends on updating query node
  • docker setup for each node ... currently dont have it for either.
  • update CLI to allow managing groups, so people can enter roles.
  • update CLI to enable uploading
  • update Pioneer v1 to join groups
  • ideally also a working integration between content directory and storage&distribution.
  • update content directory mappings to work with new features and new storage system.
  • update to use Polkadotjs 5.2+ (or just newest 5.7)
  • update mappings to use hydra v3
  • update mappings to use protobufjs
• Postpone the technical questions for another meeting.

Todo

• Bedeho will add isseus to release plan & we should make issus for everything in Giza.

[bedeho]

Meeting Date

Wednesday, Sep 15th, 2021.

Participants

• Shamil
• Bedeho
• Mokhtar
• Leszek
• Ignazio
• Anuj (UTC +5.5)

Agenda

• What is status in order to launch test network?
• What deployment support % infra is needed?
• How can we split up work on testing plan, so that it can proceed even if Mokhtar is fully unavailable going forward?
  • How can we more directly include Anuj in the release workflow & collaboration?
  • Where can Ignazio contribute?

Conclusions

Overview

The primary goal was to find a way to quickly get to the milestones of

a. running an internal staging network with all relevant nodes b. starting to write integration test.

in a way that does not depend on the participation of Mokhtar, as his availability going forward is unpredictable. We decided the following overarching points

. Try to unblock the ability of Anuj to start working on the local test setup with all nodes as soon as possible. This setup is required to actually start work on the integration tests. This will be done by racing to quickly merge all outstanding work, with only light review, that is required for him to get started. Deeper reviews can be done in parallel. This is expect to take at most a few days of work, with the outstanding

. Reconvene in a later meeting, perhaps Thursday this week or Tuesday next week, to deep dive on how to parcel out the writing of the testing scenarios in the testing plan.

We also settled some minor points

• we will update Polkadotjs, for btreeset benefits.
• creation of local testing setup with all nodes will take 3-7 days.
• we will delay work on automated network deployment until we are atleast done with the local testing tooling, as it would take at least 2 weeks of Anuj's time.

Release

We can break the release down into the following structure of task collections and dependencies. One major risk factor here is that Leszek has a very large number of responsibilities, if possible, some should be assigned to Ignazio while he waits for integration testing environment to get done.

Todo

• Bedeho will invite Anuj to RocketChat.
• Bedeho will create two milestones
  • Staging Network: https://github.com/Joystream/joystream/milestone/1
  • Integration Testing Unblocked: https://github.com/Joystream/joystream/milestone/2
• Mokhtar will create his testing plan issues, with estimates, to both the Giza project and to the relevant milestone.
• Leszek will create issues for any missing issues, such as Pioneer1, mappings, Polkadotjs, etc. where he has, at least temporariyl been assigned.
• Bedeho will create final estimates.
• Shamil will create docker support for Colossus, with input from Mokhtar.
• Shamil will create new branch anuj_base, which forks off master, which will serve as base for Anuj to work on.
• Leszek will merge his ongoing Argus work into anuj_base.
• Shamil will merge his ongoing Colossus work into anuj_base.
• Ignazio will complete his reviewed work on the runtime side this week.

[shamil-gadelshin]

Devops branch: https://github.com/Joystream/joystream/tree/storage_v2_devops

[bedeho]

Meeting Date

Wednesday, Sep 24th, 2021.

Participants

• Shamil
• Bedeho
• ~Mokhtar~
• Leszek
• Ignazio
• Anuj

Agenda

• What is everyone working on?
• When can we have staging network running?
• What should everyone work on?

Conclusions

• We will reincorporate the two new operations working groups, which implies some changes in Pioneer v1 and CLI.

Where we are

• Leszek
  • updated mappings already.
  • started on CLI, finishes today, or Monday.
  • Leszek expects Polkadotjs update to be easy, has already done it for Olympia.
  • Leszek will add all existing issues to the two milestones.
• Anuj is done with integrated testing environment with all nodes, which is required for running local integration tests. Only blocker is small remaining bug fixes Leszek will help with.
• Shamil is finalising synchronization between storage nodes.
• Ignazio is done with all of the runtime integrations and is just now out of assigned.

When can we have staging network running?

1. There a few small minor bugs, identified by Anuj, which need to be resolved.
2. We need to provide initial values for runtime constants.
3. Some minor tweaks and finalisation.
4. We need to figure out how to setup such a network, as Mokhtar may not be available, and he has had this responsibility before.

Todo: What should everyone work on?

• Shamil will
  • merge the operations runtime stuff back into Giza.
  • complete work on runtime constants.
  • work on integration tests until he can begin on review of Argus.
  • start on Argus review
• Leszek will
  • finish off the final work on distributor next week after Monday
  • Do work on P1 & CLI for new operations working group
  • start on Colossus review
• Anuj will prepare new staging network for community testing once everything is done, likely some time in the middle of next week. He will consult Leszek, Martin and Mokhtar to learn how it is done, and possible write some brief documentation on this, even though we are going for an automation of this soon.
• Ignazio will start working on integration tests as soon as possible, using Mokhtar test plan as a guide. He will look at existing test as guide, as well as Olympia tests which already have query node integrated, and can also ask Andy, Leszek and possibly Mokhtar for advice. There is also some partial documentation there that can be reviewed, and improvements to it are very welcome, to lower onboarding frictions for the next person.
• Mokhtar will (when back)
  • making final integration testing issues with estimates
  • help with staging network deployment
  • contribute to node reviews

Gantt

Staging network ETA: Monday October 4th

Tried to summarise focus areas across contributors in a Gantt chart, as there are many inter-dependencies. Had to make a good number of assumptions, so next time we should try to update this together.

[ignazio-bovo]

Meeting Date

Mon 4th October 2021

Participants

• Ignazio
• Mokhtar

  Summary

• Ignazio is still looking at the tests and writing a onboarding doc as I mentioned during the standup
• Shamil will merge the content directory fixes into giza as soon as he's back
• Mokhtar will have to merge the giza branch into devops branch as soon as Anuj PR: https://github.com/Joystream/joystream/pull/2683 is merged into the devops branch. This PR will allow to execute the query node and start a chain with no initial state on AWS.
• When Anuj code will be merge we will be able to automate the part 1 of Mokhtar's testing #2590 on aws and locally.
• once Ignazio's work is merged on giza together with Anuj's we can complete the whole plan

[bedeho]

Meeting Date

Wednesday, October 6th, 2021.

Participants

• Shamil
• Bedeho
• Mokhtar
• Leszek
• Ignazio
• Anuj

Agenda

• What is everyone working on?
• When can we have staging network running?
• What should everyone work on?

Conclusions

Where we are

• Runtime constants are done, but not merged.
• Leszek has X days left on distributor,
• Leszek has Y days P1 & CLI is is almost done.
• Leszek has already started Colossus review.
• Ignazio is wrapping up deletion fixes PR and has started integration testing.
• Anuj is just wrapping up local testing network deployment today.

Staging network ETA

Monday 11th of October

Launch ETA

Monday 1st November

Todo: What should everyone work on?

• Mokhtar will prepare issue about initial state of integration network.

See Gantt chart.

[Lezek123]

Staging network ETA: Monday 11th of October

From my point at least those need to be completed before we can launch a network:

• Updating CLI & mappings PRs with new assets removal logic (optimistically ~3h)
• @joystream/types update reviewed & merged (optimistically ~1-2h)
• Pioneer update PR reviewed & merged (optimistically ~1-2h)
• Mappings PR reviewed & merged (depends on the review time)
• Polkadot api updated + reviewed + merged (~1-2 days)

Having that in mind, even late monday seems to me quite optimistic, assuming at this point we could merge unreviewed colossus, argus and cli into some new staging network branch, which implies:

1. Creating a new giza_staging branch.
2. Probably solving some merge conflicts, as all 3 PRs are huge
3. Setting up the actual network (with at least 2 storage/distributor nodes etc.)
4. The utility of the stating network tests at this point in time will be quite limited, as there may be many more (possibly non-trivial) changes coming due to pending colossus/argus/cli reviews.

[bedeho]

Thanks for that sobering follow up, I guess we will get a more modest estimate when we do this next time around.

[bedeho]

Meeting Date

Wednesday, October 13th, 2021.

Participants

• Shamil
• Bedeho
• Mokhtar
• Leszek
• Ignazio
• Anuj

Agenda

• What is everyone working on?
• When can we have staging network running?
• What should everyone work on?

Conclusions

• As the staging network will include an upgrade from Sumer, it requires certain upgrade related work to be done which was previously overlooked.
  • Bootstrapping code is require to generate the initial snapshot state of the query node that will start processing only after the upgrade, and hence will miss all state-transitions before this that are still relevant, like memberships.
  • We will, on the upgrade, fire all existing workers in the operations working group, as the names of these groups are changing, and if we change the name without unstaking existing workers, their funds will effectively be lost forever.
  • We will do an unsafe, but simple, on-chain upgrade migration where we explicitly delete the video and channels maps. This is sufficient for testing, but not safe enough for production, where we will do unrolled deletion with a low bounded number of deletions per block, while the content directory state is locked. This logic will be written when the staging network is up and running.
  • Also, next video & channel ids will be reset to a very large value, so as to not end up producing same ids from pre-upgraded chain, since avoiding this would allow Atlas to redirect key videos which may be externally referenced, if we wanted to.

Staging network ETA

Monday 18th of October

Launch ETA

Monday 12th November

Plan

[bedeho]

Meeting Date

Wednesday, October 13th, 2021.

Participants

• Shamil
• Bedeho
• Mokhtar
• Leszek
• Ignazio
• Anuj

Agenda

• What is everyone working on?
• When can we have staging network running?
• What should everyone work on?

Conclusions

• We will not automatically fire any of the workers in the working groups that are getting renamed, instead, the leads will have to manually let them go, or they have to leave, to not loose their stake. @bwhm @blrhc will have to add this to communications plans.
• We will nail down exactly who does what part of the integration tests in next meeting.
• We decided that next major investment on the Devops side is to facilitate automatic deployment of staging networks.

Staging network ETA

Monday 19th of October

Launch ETA

Monday 12th November

Plan

[bedeho]

Meeting Date

Wednesday, October 28th, 2021.

Participants

• Shamil
• Bedeho
• Mokhtar
• Leszek
• Ignazio
• Anuj
• Martin

Agenda

• What is status of
  • problems already identified through early testing
  • staging network
  • integration tests
  • reviews.
• What should everyone work on next.
• Update Gantt chart to reflect this.

Conclusions

• The problems already identified had two candidate fixes, one in runtime related how we assumed instance modules are mapped to and one relating to how our types library handle encoding of variant types. The fix around the latter issue came up as needing a second meeting to fully hash out as a candidate fix, but we are very close to a fix.
• Staging network is up and running, and creating it takes about an hour and some manual work to run some
• We identified the need for deeper coordination on writing the integration test across Mokhtar, Leszek, Ignacio and Shamil in a way which both does not cause redundancy, gaps and maintains a style and approach, hence there will be a integration testing planning meeting next Monday (November 1st) to hash this out in further detail.
• Leszek took on task of writing bootstrapping script for Giza content directory, which will be used used to migrate content over. So this will include both export script, which grabs a subset of videos, channels and their assets, as well as uploading script.
• Mokhtar will do script for bootstrapping "useful" state for Sumer to be executed both before staging network upgrades and integration test upgrades.
• First reviews are done for Argus and Colossus, fixes are in the process of being finalised, upon which a second review cycle will be initiated.
• There have been some delays on infrastructure side, automated query node deployment is still WIP.

Staging network ETA

Done

Launch ETA

This is unclear, as we have not included Atlas work, but it does include lots of integration testing work which in principle is not a blocker, but we are likely going to end up at a later time than estimated last time.

Monday 23th November

Plan

`

[bedeho]

Meeting Date

Wednesday, November 3rd, 2021.

Participants

• Shamil
• Bedeho
• Mokhtar
• Leszek
• Ignazio
• Anuj

Agenda

• What is status of
  • staging network
  • integration tests
  • reviews.
• What should everyone work on next.
• Update Gantt chart to reflect this.

Conclusions

• An integration testing planning meeting was conducted with the goal of making sure everyone involved is coordinating their efforts and style, and using reusable fixtures.
  • Shamil: https://github.com/Joystream/joystream/issues/2590#issuecomment-957358000
  • Leszek: https://github.com/Joystream/joystream/issues/2590#issuecomment-958019333
  • Ignazio: https://github.com/Joystream/joystream/issues/2590#issuecomment-960551342
  • Mokhtar is expecting to not write integration tests.
• Integration testing timelines are still very uncertain.
• There was significant confusion and lack of agreement on the question of uploader authentication, which came up in the context of how the uploader script should do uploading. Conclusion was that Colossus currently only does authentication that the uploader is a member, and that the issue will be revisited post Giza, as described in this issue: https://github.com/Joystream/joystream/issues/2815
• Leszek will write uploader script using new collaborator feature built by Ignazio, but channel creation will be done using sudoAs. For end-to-end use of collaborators we would need channel ownership transfer, added issue about this here: https://github.com/Joystream/joystream/issues/2816
• There is no ETA as of yet for Atlas, as Klaudiusz is still in planning stage, so this will def. be major bottleneck for release date.
• Shamil will not be available the next two weeks starting on Monday 8th, and his goal is to complete addressing first round of review comments from Leszek on Colossus, and also to provide second review of Argus.
• The team agreed we should consider switching integration testing infra to Pulumi/Minibox, as it unlocks the benefit of abstracting away the testing infra from the internals of the test code, but it may also be quite slow. Mokhtar will review this alternative.
• Colossus network automatic cloud deployment work has been somewhat delayed.

Staging network ETA

Done

Launch ETA

This is unclear, as we have not included Atlas work, but it does include lots of integration testing work which in principle is not a blocker, but we are likely going to end up at a later time than estimated last time.

Monday 24th November

Plan

[bedeho]

Meeting Date

Wednesday, November 11rd, 2021.

Participants

• ~Shamil~
• Bedeho
• Mokhtar
• Leszek
• Ignazio
• Anuj
• Dmitry

Agenda

• What is status of
  • staging network
  • integration tests
  • reviews.
• What should everyone work on next.
• Update Gantt chart to reflect this.

Conclusions

• We do currently have staging network, it's still on Sumer runtime, Mokhtar needs to fix this for Martin. It is not fully automated, it require three manaul steps.
• Mokhtar will document this process for launching and upgrading a new staging network in an issue for the benefit of Martin and others.
• Mokhtar will not write test.
• Ignazio ran into an issue with getting test to work with types that change across upgrade, current best fix candidate is to not do pre-upgrade setup through transactions, but instead by directly importing a full chain state using a tool that exists as a fork of something found as part of Polkadotjs toolkit.
• We need to add some new API fixes to Argus and Colossus to faciltiate testing.

Staging network ETA

Done

Launch ETA

This is unclear, as we have not included Atlas work, but it does include lots of integration testing work which in principle is not a blocker, but we are likely going to end up at a later time than estimated last time.

Friday 10th December

Plan

[mnaamani]

Meeting Date

Tuesday, November 23rd, 2021.

Participants

• Shamil
• ~Bedeho~
• Mokhtar
• Leszek
• ~Ignazio~
• Anuj
• ~Dmitry~
• ~Klaudiusz~

Agenda

• Brief Shamil on recent changes in testing framework
• What is status of
  • giza playground for atlas team
  • integration tests
  • reviews.
• What should everyone work on next.

Conclusions

• Mokhtar gave a quick intro to changes in testing framework and service startup mechanism as well as assumptions to be made by the integration tests for benefit of Shamil.

• Mokhtar and Shamil will review Query-Node PR

• Shamil will look into new build issue for Joystream-node

• We need to look into best solution to deal with colossus role_account being used outside of node (cli, pioneer) and how transaction nonce handling can or should be done. At minimum operator guide should advise to avoid running the node while using the cli or pioneer directly to submit transactions.

• Mokhtar and Leszek will prioritize reviews of Argus and Colossus this week.

• Giza playground launching script completed by Anuj, being tested by Mokhtar but is still missing small piece, initialization of storage and distributor bucket config and metadata.

• We discussed requirements for basic buckets and metadata configuration required for playground. Leszek is preparing a “test scenario” as part of his integration testing work which we can try to use to configure the playground.

• Collaboration PR - Shamil reviewed and waiting for change requests to be implemented by Ignazio

Launch ETA

Not addressed