search

Joystream / joystream

Joystream Monorepo
http://www.joystream.org
GNU General Public License v3.0
1.42k stars 115 forks source link

Ephesus Meeting Minutes #4565

Closed dmtrjsg closed 1 year ago

dmtrjsg commented 1 year ago

Context

Issue to encapsulate the meeting minutes from the JSG Ephesus calls

dmtrjsg commented 1 year ago

RUNTIME

Referendum

⚠️ We need to establish new rules on how we merge to monorepo. Two reviewers is a must! And only these reviews are tracked in the REVIEW pipe.

Channel payout proposal should not include uploader_account Assigned to @Ignazio for this week. Integration tests to be added

Integration tests - 4540 - Extended with 4520 + all runtime issues

⚠️ New issue added -4563 running list of runtime changes for Ephesus: @Martin to review Add Channel payout documentation to Handbook

Remove call filter content.set_channel_paused_features_as_moderator

Blocked Payouts parameter - does atlas check it in payouts?

Re-anable and fix runtime upgrade CI check

Proposal to upgrade runtime

Mainnet liberated integration test fix

CLI

Direct ChannelPayment

COLOSSUS + ARGUS:

Expose QN status in Argus and Colossus

⚠️ !!This requires to notify the right people for change management on the infra level.

OPERATOR MGT TOOL

Added for next week @Zeeshan. Not strictly a blocker for the release but makes sense to track as part of it.

ORION:

We need versioning and changelog for v1.

YT-synch

Zeeshan and Dmitry to update the timelines and status of YT group of tasks before next meeting.

ATLAS

Most work to be completed this week. One issue to be estimated - Payouts Table update.

PIONEER

Payout proposals is main prio. Estimates to be updated next meeting on Wednesday.

dmtrjsg commented 1 year ago

Meeting 25th February

Streams

Runtime

Ephesus CI checks - added to next week

Prevent providing arbitrary ‘uploader_account” as part of updateChannelPayoutParameters. @Mokhtar + @Ignazio assigned as reviewers, to prioritise this.

Ignazio is working on RE-enable and fix int tests. Merge Mainnet liberated to Ephesus depends on PR for fixing integration tests

Running list of changes to runtime in Ephesus. Added task: integration tests for government blacklisting, assigned to Ignazio.

Atlas

Added issue to handle blocked claims on Atlas side, if proposal limits prevent from claiming or claims by channels are disabled.

Pioneer

Remove uploader_account to be added to the PR that Theo works on already.

Excluding blacklisted accounts after proposal.

Move all tasks to Tracker issue @DM

YT Sync

Only subtitles left for this week Reward mgt tool
tolerance testing planning and execution

MILESTONES

Scope frozen runtime added - 2nd Feb

Everything merged) Runtime upgrade sanity checks that council needs to conduct Check that UpdateChannelPayouts proposal is included to runtime

Pioneer completed QA - Feb 3rd

All tasks finished -> next week

Atlas - Completed QA 8th Feb

YPP Payout Member Remarks @DM to add links to YPP test script from this issue

Complete Runtime QA - 13th Feb

Propose to Council - Feb 17th

dmtrjsg commented 1 year ago

MM as of 30.01.2023

Streams

YPP auto-sync

Exclude unlisted videos from syncing added to next week.

ℹ️ Fully tested YPP milestone is pushed to next week.

Runtime

Expose Min/max cashout limits for Pioneer Proposal. Assigned to @ignazio-bovo , scheduled for this week, ideally to be merged before Thursday.

⚠️ Added new task: Regenerate weights for content module @mnaamani. Added with higher priority to hit Friday milestone for Runtime

Add integration test for governance blacklisting - likely to be postponed to next week. Subject to other int tests getting merged. Timeline decided to be reviewed on Wednesday.

QN

Make payloadDataObject Nullable.. was a change request. It also contains bug fix that allows to create Proposal without the payload without breaking QN. @Leszek is reviewing it.

@dmtrjsg to add this case for the test of Pioneer (to test proposal without payload)

4579 assigned to @Lezek123 - claim and withdraw in a single extrinsic- needs fixing as introduces error to QN when triggered.

Reviews

QN: 4530 needs to be re-reviewed as payload can be optional. This is to show the hash on preview or proposal. CI upgrade: its a draft now, @ignazio-bovo will finish it in 2 days.
4542 Mainnet liberated integration test fix - not reviewed just yet. Assigned to @mnaamani 4409 - reviewed by @ignazio-bovo , not @mnaamani r just yet.

Pioneer

For excluding blacklisted accounts, loading spinner on voting account selection to fetch blacklisted acc. We decided not to add this info to QN for the time of release, but maybe do it later.

Other

Joystream JS PR is not added to the board, one brought to the attention of the group by @thesan

Milestones

Launch date - unchanged ✅

Pioneer completed QA pushed to wc 6th Feb. Runtime completed QA has risk of being pushed to next week from Friday 3rd, subject to review on Wednesday based on pace of int tests merges. Atlas QA completed, intact, as payments history rework to include member remark payments is not strictly required for hte launch. Member rewards mgt tool (ytsync -128) is not required for the launch, and estimates are still pending. If it takes 2 weeks, we need to take a view whether that is altogether required.

dmtrjsg commented 1 year ago

Ephesus MM 01.02.23:

Key updates

YPP removed from Ephesus release. Payouts dashboard will be hidden for the first release. Focus on Runtime, Pioneer working with the Runtime is the new focus of Ephesus calls.

Milestones

Actual release is scheduled to be after 4 consecutive councils, following the proposal to upgrade runtime created 15th Feb.

Streams

Runtime

Remove uploader account from payout proposal -> Merged

Int tests wo sudeo calls -> pretty much done.

Expose cashout limits -> needs chain metadata update already addressed by @Ignazio, ready to be approved. Assigned to @mokhtar and @leszek for review, eta - eod today

Add integration tests for govt blacklisting - to be started today by @ignazio. @mokhtar will review.

Re-enable and fix runtime upgrade CI check. Done, working locally. Depends on “liberated” merged.

Re-genrate weights -> awaits one runtime PR to get merged to get unblocked. Early next week.

:! Code-freeze for runtime expected early next week. :! Complete runtime QA @Martin, to start on Feb 9th.

Reviews

Mainnet liberated int tests fixed - very minor change needed and should be complete. One reviewer is enough, assigned to @Mokhtar

Referendum: Add voting opt out functionality -> small change request left. To be reviewed again by @leszek. Eta tomorrow.

Make payloadDataObject nullable -> can be merged right now.

Atlas

Payouts dashboard and member remarks need to be tested. But after runtime release is ok.

Pioneer

Mappings in the QN required for exposing blacklist to council. Design is required, but pre-condition for launch. Design will be undertaken by Community dev design.

QN

ExposeUpdateChennelPayoutsProposalDetails -> merged Event handlers for channel.cumulative.RewardClaimed-> PR opened yesterday, can be fixed and merged today.

Unrelated to release

What are we doing with the parallel testnet

Final test to be done on the parallel network (which is identical production). Chat created on discord: https://discord.com/channels/811216481340751934/1070288124757938186. to be discussed with @bwhm

New issue raised 4592 - security issue - would be good to get fixed as part of Ephesus. Assigned to @leszek. Not part of Ephesus release.

Updated Rust compiler New runtime, using updated rust compiler needs to be tested during QA on the test branch. No guarantee the the runtime binary will stay exactly the same using new compiler. @mokhtar to take care of setting up the experiments..

dmtrjsg commented 1 year ago

Mm Ephesus 6th Feb:

⚠️ Top prio: Atlas-dev env needs hydra fixed + restart processor without resetting the chain. @Lezek123

Runtime

Channel payout mappings and missing checks - merged

Do we need to add nft_max_buy_now_price as runtime const as part of Ephesus? @bedeho to clarify

Re-enable and fix runtime upgrade CI check - PR is in draft

Add integration tests on governance blacklisting ends this week, assigned to @ignazio

Runtime Reviews

+new issues added Ephesus governance opt out int tests @leszek and @mokhtar

Ephesus merge main net liberated @mokhtar is reviewing it rn; some things ar failing, review will be finished today.

Output Blake3 hash of channel payload file for CLI @leszek assigned as reviewer.

Pioneer

@dmtr to test Payout Proposal PR *(create and preview proposal)

Orion2

~@dmtr to raise issue for NFT royalties to be added to be shown in payments. Will be assigned to @leszek. Not urgent against other prios desribed later~ - already added to Pioneer2.

Details are in this Discord msg>

Atlas

@dmtr to create Ephesus tracker issue to encapsulate payments table update work

QN

Ephesus

Event handlers QN update - completed

Update qn from master with Hydra fixes on big int

Milestones

Scope frozen runtime - moved to Friday, 10th.

Fully working pioneer - feb 15th.

Non-ephesus

⚠️ For QN two tasks of higher priority than Ephesus; 2 issues for the master branch.

Channel deletion Deleting channels causing QN processor to crash if collaborator was set up. If no collaborator - no problem deletion. @zeeshan will be working on that issue. @lezsek will be assigned as reviewer.

Big int (for Hydra) Ready for review, assigned for @lezsek PR in the mono repo required to be using that. @zeeshan will raise

@mokhtar to write a guide and @dmtrmltsv to inform builders channel that new mappings are added. You can build QN and restart processor docker; building new container.

YPP

New separate env is needed for testing YPP. @Zeeshan will create

Apps

Leszek will review metaprotocol changes (pr from Bartosz) ahead of proceeding working with QN.

dmtrjsg commented 1 year ago

Ephesus call MM 8.02

NEW TASKS

Atlas

Runtime

CLI

Pioneer Types for ephesus need to be published. Mokhtar to weigh in on it, he will be intermittently available.

Blacklisting - will start end of this week.

MILESTONES

Feature freezing of runtime -> completed :check:

  1. Merge mainnet-liberated to ephesus - done
  2. Add int tests for govt bl, @Leszek to review
  3. Re-enable and fix runtime upgrade CI Check - to be done later today. @Mokhtar was assigned, keeping him as assignee
  4. Failing Playground Deploy CD action - pre-condition for launching new PG. There isn’t now the Ephesus PG. So far we were testing atlas-next pg and it has sudo enabled. For the new PG we need this fixed.
  5. Re-generate weights for runtime. PR opened yesterday. @Ignazio added review, and it is now merged. @Leszek did not review just yet but agreed to have a look and okay it.
  6. Integration tests for Blacklisting -> PR waiting to be reviewed.

Next milestone: Complete QA by Tuesday - on track, will start on Saturday Failing Playground Deploy CD action < prerequisite for this milestone (playground needed), but not blocking @Martin till Saturday.

Prepare Ephesus testing plan - in flight Conduct Testing - Payout tutorial - can be done after testing Council Notes: Creator Payout Handbook @Ignazio

Frozen Runtime - stays on the 16th Feb

Fully working Pioneer - next Wednesday

Release ready - requires Atlas to be ready, milestone to be actualised after YPP / Orion release is out.

dmtrjsg commented 1 year ago

Ephesus call MM 13.02.23

NEW / UNSCHEDULED

Atlas Make sure Ephesus version of Atlas works w main net runtime and QN. This is to release Atlas ahead of Runtime release. To ensure QN works with old runtime and Atlas. Not a problem for Atlas, as payouts is commented out and we have a new version for release ahead of Ephesus.

QN QN backward compatibility is not maintained. Member remarks diff versions may not be able to get processed. Indexer is not a problem and does not have knowledge about specific meta data.

New milestone added (next week or sooner):

Assigned @Mokhtar eta 27th

Runtime Tweak remarks - wip @ Leszek

Re-enable storage in CI & Fix setupNewChain test - this allows deploy of ephesus gh branch, it is working now, should be merged soon (today-tomorrow). This is a blocker for the deploying new pgs, assigned @Ignazio

Make sure that apss don’t exceed GraphQL request size limit - this issue is to ensure the problem discovered on the storage does not occur in other places. [the request cannot be >100kb, limit can be increased but the proper solution needs to encapsulate refactoring long queries with lots of parameters like ids from array that we are passing to a query]

We need to implement: Interception of request, check size, add logging for cases when this limit is approached // Nginx reverse proxy to put infant of qn so we detect size of request // Apps actually need to deal with it so we need to check app queries and make sure its dealt with on the originating app level.

Ephesus label removed; colossus + argus label.

Pioneer Proposal description update - done

STREAMS

Runtime Re-enable storage in CI and fix setupNew chain testFailring pg deploy cd actions - addressed in PR above.

Pioneer Everything is done, but need to be reviewed by community and merged after.

Orion Changelog - to be updated after testing

QN

There’s a PR: bumping all packages for Cli types/ qn and all.. Update QN from master with Bigint hydra Fix when ready - not ready yet. Assigned to @mokhtar Changelog is not ready to be done yet

MILESTONES

Focus QA of runtime milestone - eta tomorrow eod

New milestone added (next week or sooner):

Other

Rest of milestones unchanged

dmtrjsg commented 1 year ago

MM 20.03.23

NEW ISSUES

Runtime Tweak remarks (Add initial invitation balance to..) - moved to Wednesday

PR: Updates to the apps branch (mappings, metadata protobuff lib, int tests)

Failing playground deploy CD action - should be merged soon

REVIEWS

Blocking qa:

  1. Fix:re-enable storage in CI & Fix setupNewChain test
  2. Update @joystream Readme CLI

MILESTONES

Propose runtime for council -> Wednesday

Depends on JS improvement proposal Mokhtars contribution needs to be reviewed on how to review runtime upgrades Proposal in pioneer cannot be created with the runtime upgrade (Since substrate v3 upgrade was not tested) we can disable WASM compression to bypass it.

Complete Runtime QA Postponed by new changes added to runtime and wasm blob fix till later in the week.

OTHER ISSUE

JS library NPM package has dependency on the Ephesus branch, there’s some complications with deployment but no impact on the current milestones.

⚠️ Making Ephesus work post runtime upgrade with QN is a challenge, that is still unresolved. To be revisited after runtime qa is completed.