🪣 Fix Chain/Community Semantics

[CowMuon]

We need to clean up the semantics around how we handle (define) chains & communities. We've been discussing this for a bit, it's about time to get going on fixing it.

• [x] #3765
• [x] #3779
• [ ] #5290

[ianrowan]

*Throwing research into comments here as I go

Database root table + branches

• Chains are really communities by other nomenclature in our DB
• ChainNodes is the most relevant table that actually relates to a chain At the core Chains could be renamed to communities, fallout of effected foreign keys is pretty large:

  SELECT
  tc.table_name, 
  kcu.column_name, 
  ccu.table_name AS foreign_table_name,
  ccu.column_name AS foreign_column_name 
  FROM 
  information_schema.table_constraints AS tc 
  JOIN information_schema.key_column_usage AS kcu
  ON tc.constraint_name = kcu.constraint_name
  JOIN information_schema.constraint_column_usage AS ccu
  ON ccu.constraint_name = tc.constraint_name
  WHERE tc.constraint_type = 'FOREIGN KEY' AND ccu.table_name = 'Chains';

The following tables would need a definite refactor per column names for chain id. And on further research there are many more tables referencing chain w/o a foreign key relationship(will documents these in next comments)

[ianrowan]

All tables referencing Chain

• I think we should standardize all of these to either community or community_id but not both as part of the refactor/cleanup
• We should potentially create Foreign key relationships for the 11 tables that do not have a foreign key to 'Chains' currently

[ianrowan]

Current Scope of DB refactor look like it would include

1. Renaming of the following tables
   • ChainCategory -> CommunityCategory
   • ChainEntityMeta -> CommunityEventsMeta ??
   • Chains -> Communities
2. Update all foreign key relationships of Chain.id
3. Update all usage of chain/chain_id to community_id and add a fkey constraint where not currently

App Scope

• Update models accordingly + add new constraints
• Refactor all usage of fkeys across app
• Rename/discover existing min-named files/routes

[timolegros]

After talking with Ian we agreed it makes sense to split this ticket into 2 PRs.

1. Purely semantic changes PR for renaming tables, foreign keys, etc.
2. Functional PR that introduces origin for events/entities which defines where the event/entity came from without referencing a community (Chains). This will affect proposal links, event generation/storage, and ChainEntityMeta.

As mentioned in-call, I'll follow up with another comment describing some of the changes that are required chain-events side (mostly relating to PR 2).

[timolegros]

The Problem

Currently ALL events and entities are directly linked to a chain from the Chains model (being renamed to Communities). This is restrictive since it means an event cannot be consumed by more than 1 community. If we want a system that is truly a general indexer we need to be able to determine the origin of an event or entity without referencing any CW-specific data.

The Solution

Thus we should introduce a new data point called origin.

Directory structure refactor

• Rename src/chains to Bases or something agnostic to chain
• Create a directory EVM and move all EVM chain-event listener modules to that directory
  • aave, commonwealth, compound, erc20, erc721, moloch
  • Resulting high-level directories under src/bases/:
    • EVM
    • Cosmos
    • Substrate

Chain-Events Library Refactor

• Replace all instances of chain in processor.ts, storageFetcher.ts, subscriberFunc.ts, subscriber.ts, and /filters with origin
  • Instances of chain in these files are almost exclusively used for enhanced logging prefixes like [Substrate:Stafi] some log info here the appearance of the log will remain the same for event-feeds that don’t contain a contract address but since origin contains contract addresses the new log prefixes for say Aave will be [Aave:Ethereum:Aave:0x1234] some log info here. While it might look strange to have Aave precede Ethereum in the prefix, this is intentional. Aave in this case is meant to describe which network or base chain-events is using (info that would be unavailable to us while debugging unless we go to the contract address on Ethereum and see that it is an Aave contract).
  • New log prefixes: [SupportedNetwork:chain-name:protocol:module]
• Replace chain with origin in Listener.ts

Chain-Events Services Refactor

ChainSubscriber

• Refactor queries to get the origin value and create event-feeds
  • For Substrate chains, the origin is the name in ChainNodes table for the ChainNode associated with each Substrate chain/community from the Chains table
    • has-chain-events-listener = true
    • type = 'chain'
    • base = 'substrate'
  • For EVM contracts, the origin is a combination of name in ChainNodes table with the address from the Contracts table
    • has-chain-events-listener = true
    • base = 'ethereum'
  • For Cosmos, the origin is the name in the ChainNodes table for the ChainNode associated with each Cosmos chain/community from the Chains table
    • has-chain-events-listener = true
    • type = 'chain'
    • base = 'cosmos'
• Organize the queried event-feeds first by ChainNode (name/url) and then by SupportedNetwork
  • This enables us to eventually refactor listeners to be ChainNode centric i.e. only 1 listener per endpoint with that listener handling a variety of networks i.e. 1 listener to Ethereum processing Aave + compound + erc20

CE Database

• Replace chain column with origin in ChainEntities and ChainEvents tables (requires cross-database migration).

[jnaviask]

@timolegros really great breakdown -- please break it out into a separate, linked ticket for chain event specific discussion.

@ianrowan couple comments:

• We should probably remove the ChainCategory system (incl ChainCategory and ChainCategoryType tables) if possible. It's so clunky for literally adding a name or set of names to a Chain. Like a tiny version of the linking PR. That can go in immediately, as a separate, preparatory PR: #3765
• ChainEntityMeta likely does not need renaming, because "ChainEvents" will remain the name of the system for the time being. They really are events from the Chain, not for a Community.
• Definitely agree that we should not make functional changes alongside semantic changes. I think the breakdown of changes makes sense, but be warned that there will be A LOT of required changes across the app. Zak and I spent a full day trying to do it all by hand, and we kept finding more and more. So...
• Are there automated tools we can use to ease this renaming? Maybe jscodeshift? Or, how can we de-risk and expedite the work as much as possible? I don't want this to become a month-long whack-a-mole grind.

[ianrowan]

@jnaviask

1. Got it will take a look at that and check what implications on the app side will be included, but looks like it should be minimal to do that as a quick separate task
2. This makes sense, especially with the new way we'll be thinking about CE given Tim's comment and changes to polling that are already in progress. Only the chain column will need to be update in CEMeta
3. /4. I think the immediate de-risk here would be to find a way to initially support the DB migration with minimal app change, ie aliasing columns to old chain names very temporarily. In this case CE related work could be unblocked by the migration and changes to app could be made fresh off of the migration as an immediate separate task.

As far as automation, I'll keep looking into this. I've done some initial research here, and havent found an all encompassing solution(strategies I've seen so far just update models and really direct usage of them, but not nested etc) The de-risk on this side would be to continue to support aliases || actual column names until we can remove those guardrails

[jnaviask]

How exactly would aliasing work? Would it be at the database or the server level? If we can figure out a clean scope of work here, then we should create a ticket and get moving.

[ianrowan]

@jnaviask aliasing would happen at the app level in the models the way I'm thinking about it. Basically the exported attributes and the table definitions keys remain the same(as used across the app), but the options allow us to specify what column a given key points to.

So for example, somewhere using Addresses.chain_id could still use that. but the model(in the app) would resolve that to community_id

And from there we could chip away at removing aliases/replacing with correct naming.

The Chains table itself would be the only table name to change, so I think at a minimum we would refactor usage of Chains but alias all fkeys in other models where they are used

[jnaviask]

I really like this solution @ianrowan. Please put up a ticket for it, linked to this ticket (which will be converted to bucket), and then we can start executing.

[ianrowan]

@jnaviask created issue #3779

[timolegros]

@ForestMars @jnaviask I'd like to transfer this ticket to someone on the PENG team as the only remaining variable semantic changes are on the client i.e. app.chain and someone from PENG should lead that change given they will have a better understanding of when/how to do this. That part of the refactor was delayed because PENG had anticipated potentially removing the app state objecting but months have passed and we have not progressed much in that direction.

I previously noted that this should be transferred to PENG on May 2: https://github.com/hicommonwealth/commonwealth/issues/5335#issuecomment-2089316120.