TIP-382: Optimize the data structure of account assets

[xxo1shine]

tip: 382
title: Optimize the data structure of account assets
author: wb_bupt@163.com
discussions to: https://github.com/tronprotocol/TIPs/issues/382
status: Final
type: Standards Track
category: Core
created: 2022-02-28

Simple Summary

This TIP is to optimize the data structure of account assets.

Abstract

In the process of executing the transaction, most of the time is spent in the deserialization operation of the account. Later analysis, it is caused by the account asset list being too large. Some accounts have thousands of assets. During the deserialization process, it takes a lot of time.

The asset data structure is a map. When performing asset operations, I found that the map has no size limit, and there are certain loopholes in the design.

message Account {
  ...
  map<string, int64> asset = 6;
  map<string, int64> assetV2 = 56;
  ...
}

This article details the optimization of the data structure of account assets.

Motivation

According to the latest online statistics, there are currently 79,185,490 accounts and 6,595 assets. The statistics of the number of account assets are as follows..

range of assets	number of accounts
> 1000	2
500 ~ 1000	51
200 ~ 500	1910
100 ~ 200	13219
50 ~ 100	161214

Through the performance analysis of the flame graph, it can be seen that the deserialization of the account takes a lot of time. Randomly select a batch of accounts, and perform 10,000 deserialization of complete data and clear asset data respectively. The analysis results are as follows:

	complete data	clear asset data
total cost	340,123ms	29,575ms
average cost	69ms	6ms

It can be seen that the performance of the deserialization operation of the account data is improved by more than ten times after the assets are cleared.

In order to improve the performance of the blockchain, increase the TPS, it is necessary to reduce the transaction execution time. In order to reduce the deserialization time of the account, it is necessary to redesign the asset data structure.

Rationale

In order to prevent the infinite expansion of asset data, I have come up with the following two solutions, and you are welcome to propose better solutions.

1. Limit the asset map size. Set the number of assets per account to no more than max-asset-count(1000), when account A transfers asset(1) to account B, The following two situations need to be considered. a. If the number of assets in account B is less than max-asset-count, the transfer can be done normally. b. If the number of assets in account B is greater than or equal to max-asset-count, and if account B contains asset(1), the transfer can be performed normally, otherwise the transfer is refused.

2. Use k-v data to store assets, key is a string that connects the account ID and the asset number, and the value is the asset balance. The method of getting and setting asset(1) of account A (ID:12...) is as follows. a. get asset interface: assetStore.get("12...1") b. set asset interface: assetStore.put("12...1", 100)

The advantage of scheme 1 is that the implementation logic is relatively simple, but when the assets are full, new asset transfers cannot be accepted, and the user experience is not very good. Scheme 2 does not affect the asset transfer, but the implementation logic is more complicated.

Implementation

Choose to use the k-v storage scheme to achieve, mainly involves the import and splitting of assets.

Import assets only when asset operations are involved.

public long getAssetV2(String key) {
    importAsset(key.getBytes());
    Long balance = this.account.getAssetV2Map().get(key);
    return balance == null ? 0 : balance;
  }

  public Map<String, Long> getAssetMapV2() {
    importAllAsset();
    Map<String, Long> assetMap = this.account.getAssetV2Map();
    if (assetMap.isEmpty()) {
      assetMap = Maps.newHashMap();
    }
    return assetMap;
  }

When the account is written to the DB, asset separation is performed.

private void processAccount(Map<WrappedByteArray, WrappedByteArray> batch) {
    AccountAssetStore assetStore = ChainBaseManager.getInstance().getAccountAssetStore();
    Map<WrappedByteArray, WrappedByteArray> accounts = new HashMap<>();
    Map<WrappedByteArray, WrappedByteArray> assets = new HashMap<>();
    batch.forEach((k, v) -> {
      if (ByteArray.isEmpty(v.getBytes())) {
        accounts.put(k, v);
        assets.putAll(assetStore.getDeletedAssets(k.getBytes()));
      } else {
        AccountCapsule item = new AccountCapsule(v.getBytes());
        if (!item.getAssetOptimized()) {
          assets.putAll(assetStore.getDeletedAssets(k.getBytes()));
          item.setAssetOptimized(true);
        }
        assets.putAll(assetStore.getAssets(item.getInstance()));
        item.clearAsset();
        accounts.put(k, WrappedByteArray.of(item.getData()));
      }
    });
    ((Flusher) db).flush(accounts);
    if (assets.size() > 0) {
      assetStore.updateByBatch(AccountAssetStore.convert(assets));
    }
  }

[joker-gdb]

How many TRC10 tokens on TRON now, it looks like no one is using TRC10 token，I know there is BTT, others don't know

[xxo1shine]

@joker-gdb I'm doing statistics, I'll update it later, the number is not important, the key is to prevent the risk of further growth in the number of assets, this is a potential risk.

[joker-gdb]

@joker-gdb I'm doing statistics, I'll update it later, the number is not important, the key is to prevent the risk of further growth in the number of assets, this is a potential risk.

Yes, understand, we should pay attention to this risk, but my suggestion is to gradually remove the TRC10, because there are really too few usage scenarios, can not participate in DEFI, very few exchange support, of course, this is another topic.

[xxo1shine]

@joker-gdb We will consider your suggestion, thank you very much for your suggestion.

[xxo1shine]

It's resolved in PRs: https://github.com/tronprotocol/java-tron/pull/4392, https://github.com/tronprotocol/java-tron/pull/4401

[ethan1844]

Close this issue as it is implemented by GreatVoyage-v4.5.0. Check TIP detail at TIP-382 Check implementation PR at https://github.com/tronprotocol/java-tron/pull/4392