Extracting Archive Information

[dashsuperman]

Hello, I running into a number of type issues when attempting to extract archive data. The issue revolves around block 3139200, when a system upgrade occurred.

When attempting to decode blocks and their events the following warning / errors are printed:

Unable to resolve type VoteType, it will fail on construction
Unable to resolve type VoteStage, it will fail on construction
Unable to resolve type Balance2, it will fail on construction
Unable to decode storage system.events: createType(Vec<EventRecord>):: Struct: failed on 'data':: Unable to create Enum via index 20, in Normal, Operational, Mandatory
RPC-CORE: getStorage(key: StorageKey, at?: BlockHash): StorageData:: createType(Vec<EventRecord>):: Struct: failed on 'data':: Unable to create Enum via index 20, in Normal, Operational, Mandatory
Error: createType(Vec<EventRecord>):: Struct: failed on 'data':: Unable to create Enum via index 20, in Normal, Operational, Mandatory
    at createTypeUnsafe (/root/node_modules/@polkadot/types/create/createType.js:69:11)
    at Rpc._formatStorageData (/root/node_modules/@polkadot/rpc-core/index.js:421:40)
    at Rpc._formatOutput (/root/node_modules/@polkadot/rpc-core/index.js:387:21)
    at callWithRegistry (/root/node_modules/@polkadot/rpc-core/index.js:249:62)

I am using @edgeware/node-types version 3.0.3, and @polkadot/api version 1.33.1.

I am setting up the API as follows:

import { ApiOptions } from '@polkadot/api/types';
import { Mainnet } from '@edgeware/node-types';
...
const apiOptions: ApiOptions = {
        provider: new WsProvider(config.provider),
        ...Mainnet
    }
const api = await ApiPromise.create(apiOptions);

[drewstone]

Hmm this does look right against the readme, @jnaviask any ideas?

I'd say try bumping to 3.0.4 but that's mostly new for the Beresheet testnet.

[jnaviask]

The problem is accessing the old archival data requires an old version of the API overrides. Note that:

The issue revolves around block 3139200, when a system upgrade occurred.

Let me put together something to use instead, because effectively you need the pre v3.0 mainnet spec plus a few additional type overrides...

[jnaviask]

It will be harder to fix these errors without downgrading your version:

Unable to resolve type VoteType, it will fail on construction
Unable to resolve type VoteStage, it will fail on construction
Unable to resolve type Balance2, it will fail on construction

But they should only affect old edgeware-specific types. So if you're not working with those, you can try initializing the ApiPromise using the following config object, and removing the new additions once the upgrade goes through:

const apiOptions: ApiOptions = {
  provider: new WsProvider(config.provider),
  types: {
    Address: 'GenericAddress',
    Keys: 'SessionKeys4',
    StakingLedger: 'StakingLedgerTo223',
    Votes: 'VotesTo230',
    ReferendumInfo: 'ReferendumInfoTo239',
    Weight: 'u32',
    RewardDestination: 'RewardDestinationTo257',
    ...Mainnet.types,
  },
  typesAlias: Mainnet.typesAlias,
}

This should at least fix the fatal Unable to create Enum issue. If not, please let me know and we can try something else.

[dashsuperman]

It will be harder to fix these errors without downgrading your version:

Unable to resolve type VoteType, it will fail on construction
Unable to resolve type VoteStage, it will fail on construction
Unable to resolve type Balance2, it will fail on construction

But they should only affect old edgeware-specific types. So if you're not working with those, you can try initializing the ApiPromise using the following config object, and removing the new additions once the upgrade goes through:

const apiOptions: ApiOptions = {
  provider: new WsProvider(config.provider),
  types: {
    Address: 'GenericAddress',
    Keys: 'SessionKeys4',
    StakingLedger: 'StakingLedgerTo223',
    Votes: 'VotesTo230',
    ReferendumInfo: 'ReferendumInfoTo239',
    Weight: 'u32',
    RewardDestination: 'RewardDestinationTo257',
    ...Mainnet.types,
  },
  typesAlias: Mainnet.typesAlias,
}

This should at least fix the fatal Unable to create Enum issue. If not, please let me know and we can try something else.

First, thanks for help everyone! But unfortunately, downgrading to @edgeware/node-types version 2.4.5 and adding the above recommended types gives the same error.

[jnaviask]

I'll have to look more deeply into this and test some things out myself -- would like to have a solution for you.

[dashsuperman]

I'll have to look more deeply into this and test some things out myself -- would like to have a solution for you.

Okay, thanks for the help

[jnaviask]

Heads up that this looks like an API bug. I've created a ticket here to address the issue: https://github.com/polkadot-js/api/issues/2723

[jnaviask]

Never mind! I've figured out an immediate fix for you:

import { ApiOptions } from '@polkadot/api/types';
import { Mainnet, dev } from '@edgeware/node-types';
...
const apiOptions = {
  provider: new WsProvider(),
  typesBundle: {
    spec: {
      'edgeware': {
        types: [
          {
            minmax: [0, 32],
            types: {
              Address: 'GenericAddress',
              Keys: 'SessionKeys4',
              StakingLedger: 'StakingLedgerTo223',
              Votes: 'VotesTo230',
              ReferendumInfo: 'ReferendumInfoTo239',
              Weight: 'u32',
              DispatchInfo: 'DispatchInfoTo244',
              OpenTip: 'OpenTipTo225',
              ContractExecResult: 'ContractExecResultTo255',
              CompactAssignments: 'CompactAssignmentsTo257',
              RewardDestination: 'RewardDestinationTo257',
              RefCount: 'RefCountTo259',
              ...dev.types,
            },
          },
        ]
      }
    }
  },
  ...Mainnet
};
const api = await ApiPromise.create(apiOptions);

This will add the correct types for all "specVersions" older than 32 (31 is the version prior to the upgrade).

I will also figure out how to integrate this directly into the library shortly.

[dashsuperman]

Thank you so much @jnaviask ! All the types are working now.

I am hoping to use api.query.staking.ledger to get the staking balance, but this only returns null before block 3139200...is there a different call I should be using before the upgrade? And is there a specific place I should be looking to figure this out?

Sorry for all the questions, and thanks again.

[jnaviask]

Try replacing StakingLedger: 'StakingLedgerTo223' with StakingLedger: 'StakingLedgerTo240' in the above initialization code and let me know if that makes a difference.

[dashsuperman]

Try replacing StakingLedger: 'StakingLedgerTo223' with `StakingLedger: 'StakingLedgerTo240' in the above initialization code and let me know if that makes a difference.

Unfortunately that doesn't make a difference - returns null at blocks <= 3139200

[jnaviask]

Can you share with me the account whose ledger you're trying to query? It shouldn't return null unless there's no data found for it in the map -- you would get a construction error if the type override were wrong.

[dashsuperman]

Can you share with me the account whose ledger you're trying to query? It shouldn't return null unless there's no data found for it in the map -- you would get a construction error if the type override were wrong.

Seems to be happening on all addresses I try. Here is a random one that is a good example - i4HH3qfEAMJ9a5ujG3taQCQTRYmxHCL4QxWSDnw9Bz1neYF

[jnaviask]

That address returns null for me even at blocks > 3139200 ... have any others available? I was able to query historical staking.ledger info for my personal address even prior to that block.

[jnaviask]

You can verify this using our edgeware-cli utility: https://www.npmjs.com/package/@edgeware/cli

Install it and run edge -b <blockNumber> -r <nodeUrl> staking ledger i4HH3qfEAMJ9a5ujG3taQCQTRYmxHCL4QxWSDnw9Bz1neYF to query the ledger info.

[dashsuperman]

I may have created confusion by only giving the stash previously.

Here is an account that bonded at block 395422:

stash - kBpQx4PgvQYYSfwr1yn1dNPYMEveGbQBd4GX5wmxoUF2Nck controller - kzjj4BdhS198RcfPUpSaiqx3UBYCbSrVoFsofEqifeR2LoJ

here is the relevant code in my script:

import { Option } from '@polkadot/types/codec';
import { StakingLedger } from '@polkadot/types/interfaces';

public async getLedger(of: string, at: number): Promise<Option<StakingLedger>> {
    let blockHash = await this.api.rpc.chain.getBlockHash(at);
    let stakerController = await this.api.query.staking.bonded.at(blockHash, of);
    return await this.api.query.staking.ledger.at(blockHash, stakerController.toString());
}

[jnaviask]

Subscan lists the controller as kzjj4BdhS198RcfPUpSaiqx3UBYCbSrVoFsofEqifeR2LoJ, which is also what i get when i make the call api.query.staking.bonded.at(blockHash, of). When I run staking.ledger for that controller, I get seemingly accurate historical output. Not sure what's wrong because that code sample looks fine.

[dashsuperman]

Ah yes I incorrectly copied the controller address above. Thanks for noticing that.

But that is strange we are getting different outputs. Every time I query blocks <= 3139200 I get null, then immediately above it I get what looks like a correct number.

[jnaviask]

I had a bug in my code -- now I'm replicating your results fine. Investigating but not sure -- it may be related to the migration performed on staking ledgers at block 3139200, where the storage keys were modified, but I'm uncertain right now.

[dashsuperman]

I had a bug in my code -- now I'm replicating your results fine. Investigating but not sure -- it may be related to the migration performed on staking ledgers at block 3139200, where the storage keys were modified, but I'm uncertain right now.

@jnaviask what do you think the best path forward on this is? not quite sure how to solve for this

[jnaviask]

@dashsuperman my best guess at this point is that the data you're looking for was moved by the storage migration we did on upgrade v3.0.8, so it may not be accessible via an archival node, but I'm not 100% certain about the dynamics of that. I'll ask in the Substrate chat and try and get some clarity.

[jnaviask]

@dashsuperman I spoke with Jaco and the values should be available in their old form via a direct storage call, but not through the api.query functionality -- this is because the storage location of the data changed as a result of the migration, so the API doesn't know where to fetch it from.

Unfortunately this is a larger project than I can undertake right now, but if you wanted to give it a shot, you will need to construct a storage key based on the old location and hashing function of the data you want to retrieve, and then use something like api.rpc.state.getKeys or api.rpc.state.queryStorageAt to fetch the data at the correct block.

Some information on constructing storage keys can be found here: https://www.shawntabrizi.com/substrate/querying-substrate-storage-via-rpc/

And you can look at the v3.0.0 and v3.0.8 tags of the "edgeware-node" repo to see the prior storage locations and the migration code respectively.

[dashsuperman]

Okay thanks @jnaviask. I've made progress and have figured out the storage keys I need to extract information from before the upgrade using the methods detailed at the article. For future reference, these are the storage prefixes I was looking for:

Pre / Post Upgrade	Module.Storage	Storage Key Prefix
Pre	Staking.Bonded	0x5f3e4907f716ac89b6347d15ececedca3ed14b45ed20d054f05e37e2542cfe70
Pre	Staking.Ledger	0x5f3e4907f716ac89b6347d15ececedca422adb579f1dbf4f3886c5cfa3bb8cc4

Now, I have had a fine time extracting the raw storage for both staking.bonded and staking.ledger for the addresses I care about. However, I have had a hard time figuring out how to transform the raw storage into a StakingLedger type. The article you posted details the method to get an address (staking.bonded) from the SCALE encoded AccountID, but not a more complicated type like StakingLedger. Any idea how to go about this?

[jnaviask]

You should be able to do api.createType('StakingLedger', data) and that will create the correct type for you automatically.

[dashsuperman]

That yields what looks like incorrect information (i.e. stash != expected stash). I believe it could be due to the change of staking ledger from the original staking ledger struct to the latest staking ledger struct

[dashsuperman]

Never mind, I needed to pass in the data as a string. I also downgraded to node-types 2.4.1.

Looks like it is working now! Thank you for your consistent help here @jnaviask

[jnaviask]

Glad you got it working! Closing this ticket now as the issue has been resolved.