near / near-indexer-for-explorer

Watch NEAR network and store all the data from NEAR blockchain to PostgreSQL database
https://near-indexers.io/docs/projects/near-indexer-for-explorer
GNU General Public License v3.0
124 stars 56 forks source link

Decode values from args_base64 and store it in DB #138

Open telezhnaya opened 3 years ago

telezhnaya commented 3 years ago

We have some tables with JSONB fields Some of them contain encoded values. Some lines duplicate values (they have both encoded and not encoded versions), while some lines have only encoded views.

Example:

select * from transaction_actions where args -> 'method_name' = '"nft_transfer"';

85XsaAAYdPik4KZy8QXrHxBGr59qakpKH7g9vbingkY6 | 0 | FUNCTION_CALL | {"gas": 30000000000000, "deposit": "1", "args_json":

{"memo": "wiadomoscccccccc", "amount": "1000000000000000000", "receiver_id": "mbodnar.near"}

, "args_base64": "eyJyZWNlaXZlcl9pZCI6Im1ib2RuYXIubmVhciIsImFtb3VudCI6IjEwMDAwMDAwMDAwMDAwMDAwMDAiLCJtZW1vIjoid2lhZG9tb3NjY2NjY2NjYyJ9", "method_name": "ft_transfer"}

BDGkLZ53CKEbygRYfkZZeVyjrDnEJQsNNC3Nt7QRw5jJ | 0 | FUNCTION_CALL |

{"gas": 100000000000000, "deposit": "1", "args_base64": "eyJyZWNlaXZlcl9pZCI6ImFzMi5uZWFyIiwiYW1vdW50IjoxMjAwMDB9", "method_name": "ft_transfer"}

First line has both encoded and not encoded versions, while we have to decode second line. We will get the following:

{"receiver_id":"as2.near","amount":120000}

It would be great to decode base64 and add the info to the existing JSONB field

frol commented 3 years ago

This should have been done in #87. I believe we forgot to migrate the old data. We can implement a simple script that will go over the data and update it. Alternatively, we may update the Indexer for Explorer, so it updates the records instead of inserting them, and then re-index from genesis.

Any solution that does not require too much of our engineering time is fine with me.

cc @khorolets

khorolets commented 3 years ago

@telezhnaya I believe the reason for that might be in the python migration script from #99

We can try to check specific records. The script had some sort of fallback logic if it could not decode the encoded data. You can play around if you have time or I can check later.