feat: add ft, nft aggregated tables

[telezhnaya]

Please have a look at the columns in both tables. FT NFT

FT

https://github.com/near/NEPs/issues/141

We can also add nullable memo field. Though, I thought it's not so important and we can store it in args. I also have a plan to put there all the other data that we have.

NFT

https://github.com/near/NEPs/discussions/171

Same things about memo and args.

We also can add nullable approval_id field, read more about it here: https://github.com/near/NEPs/blob/master/specs/Standards/NonFungibleToken/ApprovalManagement.md

Any feedback is highly appreciated!

I guess I can implement the first draft and check what we put into args. After that, we can finalize the columns.

[telezhnaya]

I've pushed another version of ft/nft tables. It's even more important to discuss not the interface, but the process of collecting the data.

On the call, we discussed how should we split the data by line: transaction-based or receipt-based. For now, I've chosen a transaction-based way: I guess noone will need more fine-grained splitting.

We should trust each contract implementation. It does not sound reliable. Instead of that, I suggest the following algorithm for collecting the data:

• We find all transactions with the methods that could change ft/nft balance of the user.
• For each transaction, we collect all receipts. (It means, we wait for all the receipt chains to be finalized)
• For each receipt, we collect predecessor_account_id and receiver_account_id
• For each account_id, we call view method ft_balance_of or nft_token to get the balance.
• For FT, for each account_id (both predecessor and receiver), we store the info in the DB.
• For NFT, we store the info about the current NFT owner.

---

A minute of self-criticism:

1. The PKs are awful, it repeats the story with account_changes table, I suggest repeating the solution from there and to return id columns.
2. While re-reading my own report, I hane an idea that maybe it's better to store data receipt-based. It will be easier to understand what's going on with more data.
3. Imagine the scenario: A gives its NFT to B. We have 2 records about it: A has NFT, then B has NFT. I want to have third record: A does not have NFT anymore. I guess I need to add the boolean column, something like is_the_owner (I should think about naming more).

[frol]

I feel that receipt-based records will be much easier to reason about and won't break apart if there is a contract that once being called initiates two independent FT transfers (being a single transaction, it will be hard to store this kind of scenario in transaction-based records); also, waiting for all the receipts sounds too much of a hassle on the indexer level since we process data block-by-block.

From the usage perspective, I would love to be able to:

SELECT
  ft_sender_account_id,
  ft_receiver_account_id,
  ft_name, -- ambiguous name, so we should do better
  ft_amount, -- UNSURE if we should have a transfer amount or should we store the updated balance or both (I feel that the transfer amount is more useful and the balance can be fetched on-demand via view call given we know the block reference; though we have a minor issue if a few transfers land on the same block, then we need to fetch the balance on a previous block, and all the transfers on the current block in the correct order and apply the changes correctly)
  originated_from_transaction_hash,
  updated_at_receipt_id,
FROM aggregated__fugnible_tokens_operations
WHERE ft_sender_account_id = 'frol.near' OR ft_receiver_account_id = 'frol.near'
ORDER BY updated_at_block_timestamp DESC, updated_at_index_in_chunk DESC
LIMIT 10

In order to see:

frol.near history:

frol.near sent 10 wNEAR to olga.near | frol.near's balance is 90 wNEAR:
  [deadbeef01] frol.near has been deducted 10 wNEAR to send to olga.near
  [deadbeef02] olga.near has received 10 wNEAR from frol.near
frol.near received 7 wNEAR from khorolets.near | frol.near's balance is 97 wNEAR
  ...

[telezhnaya]

We merged NFT support recently #169; it's easier for me to start from the beginning with FT support rather than using this discussion. Closing the PR