search

shapeshift / web

ShapeShift Web
https://app.shapeshift.com
MIT License
165 stars 175 forks source link

Solana Data Spike #7565

Open 0xean opened 3 weeks ago

0xean commented 3 weeks ago

From initial research it appears the a raw solana node does not offer indexed token data in any queryable fashion from the front end. This spike is to answer questions around how we will provide tx history, asset balances, and the like if we are to provided support for Solana.

Blockbook may be a solution, but due to its scaling issues (see gnosis) we should be very skeptical that it will have the ability to handle the transaction volume on Solana. A third party API may be a better solution or maybe some other OS solution is available that we could look at self hosting.

https://www.helius.dev/solana-apis might be worth a look.

  1. Confirm raw solana nodes do not provided needed data
  2. Confirm that trezor doesn't seem to be running blockbook for their solana implementation (https://status.trezor.io/ - the solana endpoints don't appear to be blockbook)
  3. Explore potential for blockbook support
  4. Look at other APIs that may be viable alternatives ( @0xean can help with some of this)

updates from eng call:

This spike should just cover what data / functionality we could source from a raw node only (which can be hosted by nownodes). From there product can make a decision on if this would be an acceptable UX to start with vs a more full support option with a 3rd party API. Also should confirm with NN docs that they support the needed endpoints (docs linked below)

Ideally we could support

NN urls:

sol.nownodes.io sol.nownodes.io/wss

https://nownodes.gitbook.io/documentation/nodeapis/sol-solana/http-methods https://nownodes.gitbook.io/documentation/nodeapis/sol-solana/wss

0xean commented 3 weeks ago

not @kaladinlight to pick up - spread the love for data

NeOMakinG commented 1 week ago

Solana node

Are Trezor using blockbook as Solana endpoints?

Trezor are using a node as their solana endpoints, it appears that the history is pretty big on their node, they might be opting for an archive node or full node from a third-party service or self-hosted

I tested their endpoint using a RPC method:

curl --location 'https://solana4.trezor.io' \
--header 'Content-Type: application/json' \
--data '
  {
    "jsonrpc": "2.0",
    "id": 1,
    "method": "getTransaction",
    "params": [
      "3EKVxNaGsuZBm24Kb33exdbB3uy3XTGv5GbuLYhmmZifrQpEka3PEhNZs2m8gbLWBHo3TLMLsxM8QjabW3tuDkXs",
      "json"
    ]
  }
'
{"jsonrpc":"2.0","result":{"blockTime":1657998630,"meta":{"err":null,"fee":5000,"innerInstructions":[],"loadedAddresses":{"readonly":[],"writable":[]},"logMessages":["Program 11111111111111111111111111111111 invoke [1]","Program 11111111111111111111111111111111 success"],"postBalances":[3098329330,499120720,1],"postTokenBalances":[],"preBalances":[3099334330,498120720,1],"preTokenBalances":[],"rewards":[],"status":{"Ok":null}},"slot":141868622,"transaction":{"message":{"accountKeys":["BRWrkVaTTyq3eRJw4t8YjkJuH9EtnoVeyeQ4A3eDqU86","AFEkH2vF1CYGJnPncDw6PzaitjQgdQipL2hxSWLh9iDs","11111111111111111111111111111111"],"header":{"numReadonlySignedAccounts":0,"numReadonlyUnsignedAccounts":1,"numRequiredSignatures":1},"instructions":[{"accounts":[0,1],"data":"3Bxs4Bc3VYuGVB19","programIdIndex":2,"stackHeight":null}],"recentBlockhash":"DNPkznoW42X8FTW5NNSVEcqiVNproH9QGMKCNnNt5XXn"},"signatures":["3EKVxNaGsuZBm24Kb33exdbB3uy3XTGv5GbuLYhmmZifrQpEka3PEhNZs2m8gbLWBHo3TLMLsxM8QjabW3tuDkXs"]}},"id":1}%

How to retrieve SPL Tokens by Account

Here is an example of token list I retrieved for one of my wallets:

image

TX History

Get balance

How to get blocks informations

How to get fees estimations

There are 5k lamport base fee by TX + compute units, can be simulated by consuming getFeeForMessage (we need to pre-build the TX), can also be directly done by using web3.js which will consume the same RPC endpoint under the hood

Real time informations

Market Data

To retrieve assets data, coingecko can be our best bet like we do for other chains: https://tokens.coingecko.com/solana/all.json

To retrieve market data, it seems like coingecko's endpoint returns every Solana's tokens market data's we need, doesn't seems so complexe to consume, confirmed by crawling our API

Conclusion

Everything seems feasible expect retrieving a old TX history, we might need to rely on an external service to get a older tx history

Useful links

Official Solana RPC methods: https://solana.com/fr/docs/rpc/http/gettokenaccountbalance A lot of solana tutorials: https://solana.com/docs/advanced/versions How solana are using Google BigTable as a full archive node: https://github.com/solana-labs/solana-bigtable/blob/master/README.md Solana's big table access: https://console.cloud.google.com/marketplace/product/google/bigquery.googleapis.com?p=bigquery-public-data&d=crypto_solana_mainnet_us&page=dataset&project=arconnect-299920&returnUrl=/bigquery?p%3Dbigquery-public-data%26d%3Dcrypto_solana_mainnet_us%26page%3Ddataset%26project%3Darconnect-299920%26folder%3D%26organizationId%3D NOWNodes Postman: https://documenter.getpostman.com/view/13630829/TVmFkLwy#82fcc8ab-ebcc-4010-b9f1-375e2f0365f5 Solana official programs docs: https://spl.solana.com/token-2022

0xean commented 1 week ago

please add:

NeOMakinG commented 1 week ago

More informations

Returned values contains everything we need:

Example of error status:

      "status": {
        "Err": {
          "InstructionError": [
            3,
            {
              "Custom": 6001
            }
          ]
        }
      }

Finalized is Ok: null

Example of transactions retrieved by getSignaturesForAddress followed with getParsedTransaction:

[
  {
    "blockTime": 1724265389,
    "meta": {
      "computeUnitsConsumed": 38554,
      "err": null,
      "fee": 11000,
      "innerInstructions": [
        {
          "index": 1,
          "instructions": [
            {
              "accounts": [],
              "data": "2GJh7oUmkZKoaW9Xsb3VtGc882db3mK7Fppkyk946NiVysLhQ3rcz6H7MaQ5f6YaMfLoP7KoEka3Ux2A8HR1tZG4WXKnLEhQnavaSRuEMQs5E2xhYz6YTEKXHHGYgeJsSypKpKtguEF8NmUqa5ppXn3BxTDdUmT48o2vs9nXD47JsT8STjRw47kpJFkymq9sc4PNFVpeRbqsPxoQ5qYXxXoTU8aydhMcpqAqveo8CRMfEQ2SURftaEyp5PLzNw5ctrbdwtF41mH4t1tVzPVn5QsGWL2ji",
              "programId": "noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV",
              "stackHeight": 2
            },
            {
              "accounts": [
                "6kouVRPvskaw8Kmq8pPtrbfgTMjoCcwjAB6RmLFHdtBU",
                "DZmKUFR7zvCztXqBymu1mbyDfXMZV2cDtZ2b2nZgC7KJ",
                "noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV"
              ],
              "data": "8RkZ9BWdS73EUj8ZMPAkFHrKo9ZqGbgsXHT9E6Jxn3JFpwuyT12AyKE",
              "programId": "cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK",
              "stackHeight": 2
            },
            {
              "accounts": [],
              "data": "113XSDricWBGY82sU7rRieDZfZPenYd2hbpGrVnx5fQKYDhF4PV1ojf9tX1d1mBgcAmCs5eN9yfuJa1gMDPhDt1V83ictPD2U7uesXVaCjmwSdGa4ucNxz5nsefJv4LutHsvzUxqMzLkLGchLYKR5UxaVHeQp2KNYs9CGMmzZPp29PmsvR4cLY7UViZDV17gkKoj4cyBDS4mMArniox169pEsr6edM47tHz4TKHPstFonFtV46xpu3ShwRVXQ6yWxwb8Qk4w2VfPkdpicxDxSNB4Z6VapxMA4i2j9sfvtM8u1rexweiQGdQtUT9UmbBhTTwKkaTv3HjE464TUzBNpje31VakeUmZUkmtErcynJ6QmDjRighxKvnvD125Xsq7JXWBcxWuMCs2hkG8caMUYqNJJss5ahREH62utPEE8eJkxFBP7L65Jkg2c8vBgjcLriVb4QgCwmPBpwEToW5XsB1rbQz3xXpA1N3VJeCE2mLaaEKCbrL8KA7VR29QiPVAM7QHYZ5MRDK2TZA3ZTVE69zhNjWYSzxGFQ8k75Z7qWBTWxrKFY3hhZFePHdLt1FzEqxhxVYaomN6NkhnbBiFGCx47QvhD1ZArBnMkiJeCkairj1rAHYT4dWs4bpMDeApSgU8We6RQVEgQ4iJiTuvVTcCpbvz2JHAibq4WtQ9rtPvDSJKdBv2VYoLrySC4uHiHyNfDsFnT4iH9GBwNYSzPhaXJ8FpWW9vDXcbLt7fR43kRK8HWRMdKNncxGGnwYxG7cS7YcASCBTbadUUfjbRW4yptkB4JorpVmP2tQ5Bc4tLfosBxz5Mr7h3H1waicsVBxguStHdW9t8JH2QZQdgQXT5MLgiQEhB9NZcPCRTT12nL4ARxTxvfic4bzXciZ3WMZUbaAkCCX8LhX98vxpbrZEoqN8gmRUjTY3Q5qGcrJFAPvGjY4nxyvukB",
              "programId": "noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV",
              "stackHeight": 3
            }
          ]
        }
      ],
      "logMessages": [
        "Program ComputeBudget111111111111111111111111111111 invoke [1]",
        "Program ComputeBudget111111111111111111111111111111 success",
        "Program BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY invoke [1]",
        "Program log: Instruction: MintV1",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV invoke [2]",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV consumed 39 of 175700 compute units",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV success",
        "Program cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK invoke [2]",
        "Program log: Instruction: Append",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV invoke [3]",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV consumed 39 of 163368 compute units",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV success",
        "Program cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK consumed 8537 of 171739 compute units",
        "Program cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK success",
        "Program BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY consumed 38404 of 199850 compute units",
        "Program return: BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY AL3sDdHJ+pUjrtM9ae0zoBof3FtywHBzaZ3uOvLKjfsFTodwtlFa28G/qLux9n1X23Gcf1MBKSA8i2SyZGwPgCKAynZTGDdw9vkEfAVCEJXVOBKf88E411gGE/lCk6wzAQsYAAAAAAAAdEATPF4ponkYdFRY1iq7pnpxxkt4s24bXZ+G+mlJ39dUgTCPSBX/Exw+tWxcQsX9Yvb6334OT/y3K3f9axXGpg==",
        "Program BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY success"
      ],
      "postBalances": [
        2631207408,
        265649280,
        1559040,
        1,
        1,
        1141440,
        1141440,
        42460720,
        1141440
      ],
      "postTokenBalances": [],
      "preBalances": [
        2631218408,
        265649280,
        1559040,
        1,
        1,
        1141440,
        1141440,
        42460720,
        1141440
      ],
      "preTokenBalances": [],
      "returnData": {
        "data": [
          "AL3sDdHJ+pUjrtM9ae0zoBof3FtywHBzaZ3uOvLKjfsFTodwtlFa28G/qLux9n1X23Gcf1MBKSA8i2SyZGwPgCKAynZTGDdw9vkEfAVCEJXVOBKf88E411gGE/lCk6wzAQsYAAAAAAAAdEATPF4ponkYdFRY1iq7pnpxxkt4s24bXZ+G+mlJ39dUgTCPSBX/Exw+tWxcQsX9Yvb6334OT/y3K3f9axXGpg==",
          "base64"
        ],
        "programId": "BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY"
      },
      "rewards": [],
      "status": {
        "Ok": null
      }
    },
    "slot": 284995220,
    "transaction": {
      "message": {
        "accountKeys": [
          {
            "pubkey": "9fkF9XbuJohcpUagYYdHumdMLj8rNsBW4njTVvLQxDKN",
            "signer": true,
            "source": "transaction",
            "writable": true
          },
          {
            "pubkey": "6kouVRPvskaw8Kmq8pPtrbfgTMjoCcwjAB6RmLFHdtBU",
            "signer": false,
            "source": "transaction",
            "writable": true
          },
          {
            "pubkey": "DZmKUFR7zvCztXqBymu1mbyDfXMZV2cDtZ2b2nZgC7KJ",
            "signer": false,
            "source": "transaction",
            "writable": true
          },
          {
            "pubkey": "11111111111111111111111111111111",
            "signer": false,
            "source": "transaction",
            "writable": false
          },
          {
            "pubkey": "ComputeBudget111111111111111111111111111111",
            "signer": false,
            "source": "transaction",
            "writable": false
          },
          {
            "pubkey": "cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK",
            "signer": false,
            "source": "transaction",
            "writable": false
          },
          {
            "pubkey": "noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV",
            "signer": false,
            "source": "transaction",
            "writable": false
          },
          {
            "pubkey": "6HYauFSi8D3m3ZvquJhQQdTDvRPRu8n9pRRNu5U4du1b",
            "signer": false,
            "source": "transaction",
            "writable": false
          },
          {
            "pubkey": "BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY",
            "signer": false,
            "source": "transaction",
            "writable": false
          }
        ],
        "instructions": [
          {
            "accounts": [],
            "data": "3MZf3aXK9QFy",
            "programId": "ComputeBudget111111111111111111111111111111",
            "stackHeight": null
          },
          {
            "accounts": [
              "DZmKUFR7zvCztXqBymu1mbyDfXMZV2cDtZ2b2nZgC7KJ",
              "6HYauFSi8D3m3ZvquJhQQdTDvRPRu8n9pRRNu5U4du1b",
              "9fkF9XbuJohcpUagYYdHumdMLj8rNsBW4njTVvLQxDKN",
              "6kouVRPvskaw8Kmq8pPtrbfgTMjoCcwjAB6RmLFHdtBU",
              "9fkF9XbuJohcpUagYYdHumdMLj8rNsBW4njTVvLQxDKN",
              "9fkF9XbuJohcpUagYYdHumdMLj8rNsBW4njTVvLQxDKN",
              "noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV",
              "cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK",
              "11111111111111111111111111111111"
            ],
            "data": "6UEAo7epG9XPHeqR9shnNLruyP3DEaswugqj2suYA7nFyfu2cYrxVAXtCRkWLit6NtdPj2fccZw1W17qEBxvAHyLdEjUXy16FgR9NRPTzbGzmBiNW6zUKZoXRZcxvYzykpknp9qXQkoJeQi1kK9ZMmQd9z3PXFcyEn8rrBZTEQqh8TcR9BYvXJYWJR8HAeshu",
            "programId": "BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY",
            "stackHeight": null
          }
        ],
        "recentBlockhash": "E2moUnEaxTikrPnR7pbwWZkyi5jvH2ikv3CHDcQHw2QQ"
      },
      "signatures": [
        "4v7dXK1ENbctH2db9cHJDVC3whrFmAPrEuVhYRkNLr2WsGi73niBhhMsYHJzugGkQffqv4cur4hYAvqT3ScQ8wyT"
      ]
    },
    "version": "legacy"
  },
  {
    "blockTime": 1722817162,
    "meta": {
      "computeUnitsConsumed": 42596,
      "err": null,
      "fee": 11000,
      "innerInstructions": [
        {
          "index": 1,
          "instructions": [
            {
              "accounts": [],
              "data": "2GJh7oUmkZKnSJLskfto12zrSDZPhZc8a8q9r8SxnPABWN9ECxLf8GEXTQkvkuCbVJDU1oZCcYH2BrkyFQwuAzsTayrwEYDHb6zurzb3aD1vxSc7cymMtDB39ZwAYKJykhXSsogCgKQySJyuCvinVvaXQ2m7AUst2zCKg9FxSjP3H5BohwiTrsg3Dexjr1vob9ixj1owBxFkBPcDzrLW9kPoTZtocmUvL9DY5qC5YW1gNKuzxZxTReSe5WdBLhHfcVqYTenBVsx6NHYB2Zh2TChFXH8jj",
              "programId": "noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV",
              "stackHeight": 2
            },
            {
              "accounts": [
                "AXcRrMH4oCgNEEZ7gAyPmYEqJiYzbWKw37DTgg5e7Udr",
                "7SPYyJJbTBbmh6TUuewMbukSniHNMGpZgu4qsxde1gZ4",
                "noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV"
              ],
              "data": "8RkZ9BWdS73KGiAaMZRteH4CegGZAnjTWuazoxsCNoKwSABUjZzrgaV",
              "programId": "cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK",
              "stackHeight": 2
            },
            {
              "accounts": [],
              "data": "115BRHfXRaP8Q2R2yvfRqsYvhuBRyCA9s3af3LK15sG7bB9NyQ1ZU3NQvUAQ1bSJXXEbKSC2AeMfzVbb29ZD9EkThi6HDKAPqtMkKgo7Sxp4sN1hANzPpyiMrT6HkFeuwha7M14GEhV2sMejCzf5YKfVnTEXnMHvo6DZyBGFWus47XxhPmhjQTSzSJ88o57PVfvdBVgywtrF6SkAtfzxvHrUvbG5ArB96wrZbPiAzdR4GLMTaFzufQmA6YYyRZTH6cyxHieG9Aez3W9XeYdU2wZ948Z96AEuwkRyJUuGG2TXGT95rB4VYExPUxLjUu6B4PPsvEDu3wCAdRanH9pRayS4q2qaECbjo1gWz7NDZTKVv1yjHx8oGbKam7BqAbp7tCNMQVDutBjZeBmpo7qF6jzXSSVAs4E5EqKUnfUErLjuMUJ1waAWDosZN5TBKnDdXX8RrTwSNz7Wj7vhoUg8uQYpoXBX8Fa5871qxrHwwwZf4WF4ScMp5Rr2EZppjJayEtuQjnY9bo8FCVws1k8RbhSebuN6fBJdjPLN6DrUzt455Bp2aHYD7CPKtfut181oL1qQJwhKTkJHwUTUXjRPSjrxczdEv6QsMxRi9kzebwCZhidZAztKeXuHQKjCaAZgZBs7887ue3TN1D7MQ5DaQX1LDCRULvkK6LPWtirdUqajgYPvgLpLEgsmQWRZ1T2kGoEeK3RYyRvvcW3x3FB8YpqQLSdMytGDwt1Cudq2rjs6bGBpUPv8SvW44KjApvisTaV28xrsaBFu7XYEDoxenoiSQDc4hnxHcXgR1aUZ7RhvEC25QhFr1gcN5Sk2za5ZsggmJ5Fotmud6B3VWVXRFK2P2U4bkMshcsbxVjZ9N9834pSp2yoMgtJWDN48vCVCPriMbb84X3cbgqnLowiSnJEz5SJKrPJGwWyupKgEECZxdagPtHTeX4Zef",
              "programId": "noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV",
              "stackHeight": 3
            }
          ]
        }
      ],
      "logMessages": [
        "Program ComputeBudget111111111111111111111111111111 invoke [1]",
        "Program ComputeBudget111111111111111111111111111111 success",
        "Program BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY invoke [1]",
        "Program log: Instruction: MintV1",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV invoke [2]",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV consumed 39 of 172531 compute units",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV success",
        "Program cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK invoke [2]",
        "Program log: Instruction: Append",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV invoke [3]",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV consumed 39 of 159459 compute units",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV success",
        "Program cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK consumed 9410 of 168570 compute units",
        "Program cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK success",
        "Program BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY consumed 42446 of 199850 compute units",
        "Program return: BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY ABK099gj25DYY8Xwrwz6ydFNW74kxbbpMxME4PTJFnhRTodwtlFa28G/qLux9n1X23Gcf1MBKSA8i2SyZGwPgCKAynZTGDdw9vkEfAVCEJXVOBKf88E411gGE/lCk6wzAby1AQAAAAAASR3plpAU9zSSM0EkzJIRM6x96+AIiLBFpHzIRM7DDuxUgTCPSBX/Exw+tWxcQsX9Yvb6334OT/y3K3f9axXGpg==",
        "Program BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY success"
      ],
      "postBalances": [
        133581508,
        1559040,
        265649280,
        1,
        1,
        1141440,
        1141440,
        42460720,
        1141440
      ],
      "postTokenBalances": [],
      "preBalances": [
        133592508,
        1559040,
        265649280,
        1,
        1,
        1141440,
        1141440,
        42460720,
        1141440
      ],
      "preTokenBalances": [],
      "returnData": {
        "data": [
          "ABK099gj25DYY8Xwrwz6ydFNW74kxbbpMxME4PTJFnhRTodwtlFa28G/qLux9n1X23Gcf1MBKSA8i2SyZGwPgCKAynZTGDdw9vkEfAVCEJXVOBKf88E411gGE/lCk6wzAby1AQAAAAAASR3plpAU9zSSM0EkzJIRM6x96+AIiLBFpHzIRM7DDuxUgTCPSBX/Exw+tWxcQsX9Yvb6334OT/y3K3f9axXGpg==",
          "base64"
        ],
        "programId": "BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY"
      },
      "rewards": [],
      "status": {
        "Ok": null
      }
    },
    "slot": 281606168,
    "transaction": {
      "message": {
        "accountKeys": [
          {
            "pubkey": "9fkF9XbuJohcpUagYYdHumdMLj8rNsBW4njTVvLQxDKN",
            "signer": true,
            "source": "transaction",
            "writable": true
          },
          {
            "pubkey": "7SPYyJJbTBbmh6TUuewMbukSniHNMGpZgu4qsxde1gZ4",
            "signer": false,
            "source": "transaction",
            "writable": true
          },
          {
            "pubkey": "AXcRrMH4oCgNEEZ7gAyPmYEqJiYzbWKw37DTgg5e7Udr",
            "signer": false,
            "source": "transaction",
            "writable": true
          },
          {
            "pubkey": "11111111111111111111111111111111",
            "signer": false,
            "source": "transaction",
            "writable": false
          },
          {
            "pubkey": "ComputeBudget111111111111111111111111111111",
            "signer": false,
            "source": "transaction",
            "writable": false
          },
          {
            "pubkey": "cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK",
            "signer": false,
            "source": "transaction",
            "writable": false
          },
          {
            "pubkey": "noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV",
            "signer": false,
            "source": "transaction",
            "writable": false
          },
          {
            "pubkey": "6HYauFSi8D3m3ZvquJhQQdTDvRPRu8n9pRRNu5U4du1b",
            "signer": false,
            "source": "transaction",
            "writable": false
          },
          {
            "pubkey": "BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY",
            "signer": false,
            "source": "transaction",
            "writable": false
          }
        ],
        "instructions": [
          {
            "accounts": [],
            "data": "3MZf3aXK9QFy",
            "programId": "ComputeBudget111111111111111111111111111111",
            "stackHeight": null
          },
          {
            "accounts": [
              "7SPYyJJbTBbmh6TUuewMbukSniHNMGpZgu4qsxde1gZ4",
              "6HYauFSi8D3m3ZvquJhQQdTDvRPRu8n9pRRNu5U4du1b",
              "9fkF9XbuJohcpUagYYdHumdMLj8rNsBW4njTVvLQxDKN",
              "AXcRrMH4oCgNEEZ7gAyPmYEqJiYzbWKw37DTgg5e7Udr",
              "9fkF9XbuJohcpUagYYdHumdMLj8rNsBW4njTVvLQxDKN",
              "9fkF9XbuJohcpUagYYdHumdMLj8rNsBW4njTVvLQxDKN",
              "noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV",
              "cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK",
              "11111111111111111111111111111111"
            ],
            "data": "JphXV3Wps8amyvZcMNUzuRdvr1ZqMbJmuaXWe5dWRFv3BHCvCX8as2RQjNuF9J6E42eWfWE69RgoMsTrTFqNDvbMiDuuy4A8vSjgig2fvDVZDPdBHfZ4bpmvS9Z8UX494M6Ba7apU1DNmGfNBvCk67xY4d51HMETn2HyTwfH933NR2D242aDoRQ5xrbs4k4qg6ryxm3kPzuG7",
            "programId": "BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY",
            "stackHeight": null
          }
        ],
        "recentBlockhash": "ETDrQrqidK912cFgeuSVvQ2DJpCKxteC12iMDmc8EBUD"
      },
      "signatures": [
        "5123rZidDyxA8JgUt7982qfp6CQJFAiBSeTxpLPFTbDXL3zfrdKEdt8JawiNr53ve6v4LvCyYA1FhD3qWwywAj72"
      ]
    },
    "version": "legacy"
  },
  {
    "blockTime": 1722750908,
    "meta": {
      "computeUnitsConsumed": 32903,
      "err": null,
      "fee": 11000,
      "innerInstructions": [
        {
          "index": 1,
          "instructions": [
            {
              "accounts": [],
              "data": "2GJh7oUmkZKnVTT9hLfzKaUnHY4wxrU1h47U5bAi5xv5SdMsu359FVGS4RRpV8qzP1ka9LG3GiJrMykF4j1nf6nsMdKBHpScydP7i7uhbvdS29EAfcxYh3GAdYiiqWsDjrRXSGbkRo19upu78pCt6L4rwV4PfGzPdSDx6juGaT3gdXdbiXvgSLzc5zLijWJERCqnrcGKu7Vvu7zG7Wvq8jQ1k7f54q8P8VKDqtwhvbB5oXPv4QAKYanixJbCT5jLmeiXXsj6xj1LLsQUjrBtYim6HFNrJ",
              "programId": "noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV",
              "stackHeight": 2
            },
            {
              "accounts": [
                "FLWC5bqXVKp6h1anDD8GdUub8M3Lfk4QiNkmZgAuGFyv",
                "He37C7Nieg3YT29LCPToWrbiWM4LLPKfayTmSKdArLQX",
                "noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV"
              ],
              "data": "8RkZ9BWdS73FPvmdtjQ6sfREvuoMuvNWxqWnJPWBdbPErh3sQ1KPJKg",
              "programId": "cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK",
              "stackHeight": 2
            },
            {
              "accounts": [],
              "data": "117HmFjWX88JjLP9wecjT9GK9ocJNrGFjXF9do1Zna2FC4A2Em6J4UTw1ShJrSZp22Zo52zuDB4HyF8rWyhjZHSmX6iMQupjA2bXags6HewRrqoH7vKmqtsHFEXiL7YncQodQCCtEar6gnqSTSfSQ9Di6yCLmLkenrNb92oT2F5kU3pPrm5e1EVCp6o1nKYVEaRPVnQQVPsou1Ufmnu3Q87Ep1KCNh2ScH7kDVxr1nPRGs2s3LKatphbkFCznmoUVEKfaMbhi1UBrxSZK5c3fFRrFLfqsX73bhgWoVhuei5H3pPVZFRmRxmQxA948DvN1dYADxMx7MNHUEtiezNh7q5C5DTvcuqfrzqVkqSyRQ7cDoJosM7hbfYgSfZo1V329m4xpamJppS854Rjy7sy4rYeRPUhSbuNRcq4kEJ1H6guRykrQFtT9tjA7yFfhKcyqiKsJvtsB7HojcwZeNYXZErdF3pPkD23aBmDVxiwLdVGn5k1U6rKxdTmeo4NZUPMh9MWcDky199VpbGjk2dr8QmkueDMUvGiTfLHFnniLbcwe69YovJN7bC6Y5GpVsqVaA25rYSBWfCjUCiGCfYhrx2nUMuoqdhq9DF5NGZiPezSyhZSzbCKyenK53m3JGZuKT2sFvHHhfnJNPhLoN4Q2pTzQ6a6SoJmueQfNDHvk5xTU6KSFhzb2pUmG3muXsx9JFNgkL68ypj4GaYCwzJs2rzYdNzUxeoeBtzqM9pWCsaRutc76CUXddV5HTd8KvwwFQ383BbeFvc8mkaDUYbh7fc9UKMugrj5GNKG2syuFcSAipwyhWSgrSqspyfutfXDCcgG8SWALdsmhUVFwXN2phXvvo5NdkYCzZDrj3xQMkHnEQBQCjJpQT6SQAwbYYbogcjxniK5qsa9zhTQHTrpTHTtCsRf5X8BmuJV2wjfJenRwyKxjnXRZL6dm",
              "programId": "noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV",
              "stackHeight": 3
            }
          ]
        }
      ],
      "logMessages": [
        "Program ComputeBudget111111111111111111111111111111 invoke [1]",
        "Program ComputeBudget111111111111111111111111111111 success",
        "Program BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY invoke [1]",
        "Program log: Instruction: MintV1",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV invoke [2]",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV consumed 39 of 181579 compute units",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV success",
        "Program cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK invoke [2]",
        "Program log: Instruction: Append",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV invoke [3]",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV consumed 39 of 169152 compute units",
        "Program noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV success",
        "Program cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK consumed 8765 of 177618 compute units",
        "Program cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK success",
        "Program BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY consumed 32753 of 199850 compute units",
        "Program return: BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY ABrfInlXrHhJ3E88pKtFf3/YFvBiB56G+35p/DvIAtNSTodwtlFa28G/qLux9n1X23Gcf1MBKSA8i2SyZGwPgCKAynZTGDdw9vkEfAVCEJXVOBKf88E411gGE/lCk6wzAVeIAAAAAAAAZ2jn31NhD13zL7I/2CghSClDQ0UueWv3uHIyYfqZDHtUgTCPSBX/Exw+tWxcQsX9Yvb6334OT/y3K3f9axXGpg==",
        "Program BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY success"
      ],
      "postBalances": [
        951906788,
        265649280,
        1559040,
        1,
        1,
        1141440,
        1141440,
        42460720,
        1141440
      ],
      "postTokenBalances": [],
      "preBalances": [
        951917788,
        265649280,
        1559040,
        1,
        1,
        1141440,
        1141440,
        42460720,
        1141440
      ],
      "preTokenBalances": [],
      "returnData": {
        "data": [
          "ABrfInlXrHhJ3E88pKtFf3/YFvBiB56G+35p/DvIAtNSTodwtlFa28G/qLux9n1X23Gcf1MBKSA8i2SyZGwPgCKAynZTGDdw9vkEfAVCEJXVOBKf88E411gGE/lCk6wzAVeIAAAAAAAAZ2jn31NhD13zL7I/2CghSClDQ0UueWv3uHIyYfqZDHtUgTCPSBX/Exw+tWxcQsX9Yvb6334OT/y3K3f9axXGpg==",
          "base64"
        ],
        "programId": "BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY"
      },
      "rewards": [],
      "status": {
        "Ok": null
      }
    },
    "slot": 281458092,
    "transaction": {
      "message": {
        "accountKeys": [
          {
            "pubkey": "9fkF9XbuJohcpUagYYdHumdMLj8rNsBW4njTVvLQxDKN",
            "signer": true,
            "source": "transaction",
            "writable": true
          },
          {
            "pubkey": "FLWC5bqXVKp6h1anDD8GdUub8M3Lfk4QiNkmZgAuGFyv",
            "signer": false,
            "source": "transaction",
            "writable": true
          },
          {
            "pubkey": "He37C7Nieg3YT29LCPToWrbiWM4LLPKfayTmSKdArLQX",
            "signer": false,
            "source": "transaction",
            "writable": true
          },
          {
            "pubkey": "11111111111111111111111111111111",
            "signer": false,
            "source": "transaction",
            "writable": false
          },
          {
            "pubkey": "ComputeBudget111111111111111111111111111111",
            "signer": false,
            "source": "transaction",
            "writable": false
          },
          {
            "pubkey": "cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK",
            "signer": false,
            "source": "transaction",
            "writable": false
          },
          {
            "pubkey": "noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV",
            "signer": false,
            "source": "transaction",
            "writable": false
          },
          {
            "pubkey": "6HYauFSi8D3m3ZvquJhQQdTDvRPRu8n9pRRNu5U4du1b",
            "signer": false,
            "source": "transaction",
            "writable": false
          },
          {
            "pubkey": "BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY",
            "signer": false,
            "source": "transaction",
            "writable": false
          }
        ],
        "instructions": [
          {
            "accounts": [],
            "data": "3MZf3aXK9QFy",
            "programId": "ComputeBudget111111111111111111111111111111",
            "stackHeight": null
          },
          {
            "accounts": [
              "He37C7Nieg3YT29LCPToWrbiWM4LLPKfayTmSKdArLQX",
              "6HYauFSi8D3m3ZvquJhQQdTDvRPRu8n9pRRNu5U4du1b",
              "9fkF9XbuJohcpUagYYdHumdMLj8rNsBW4njTVvLQxDKN",
              "FLWC5bqXVKp6h1anDD8GdUub8M3Lfk4QiNkmZgAuGFyv",
              "9fkF9XbuJohcpUagYYdHumdMLj8rNsBW4njTVvLQxDKN",
              "9fkF9XbuJohcpUagYYdHumdMLj8rNsBW4njTVvLQxDKN",
              "noopb9bkMVfRPU8AsbpTUg8AQkHtKwMYZiFUjNRtMmV",
              "cmtDvXumGCrqC1Age74AVPhSRVXJMd8PJS91L8KbNCK",
              "11111111111111111111111111111111"
            ],
            "data": "53CSwPKYEdA1dzmaQRvkUwe9WGyrZS5ZwSgYXytMPKRqffVpDZBWGpsoFJyYt4SmB4QzLTUKsfAuRLcTUTZ7NpoNkYypHuU9MrivXtVGmgkoGJx9XSsXQq8nM7aB1aT6VayJ1B6sG9XDY696HSwdJtjwYiTPqxaGFJJ5mMboyj8yZGrZ1BD4bhvWUiPPLHnTQiEQLXA8qs8w",
            "programId": "BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY",
            "stackHeight": null
          }
        ],
        "recentBlockhash": "8JnJbqdSy6fF8ZrD1kZdXUcBKanBnwDicx6Dv7wJzz62"
      },
      "signatures": [
        "5VHJqTHEP7JWqBewEA5hhRMat4yybiHnsdvMrD5piDSHsWtJqj4saqWJUa1Umq3iPce9CbZk1BNohqbE7qmjU9xX"
      ]
    },
    "version": "legacy"
  },
]

Script:

async function getSignaturesForAddress(
 connection,
 address,
 maxLength,
) {
 const signatures = [];
 let _signatures = await connection.getSignaturesForAddress(address);
 const signaturesData = await Promise.all(_signatures.map(sign => {
  return connection.getParsedTransaction(sign.signature, {maxSupportedTransactionVersion: 0})
 }))
 signatures.push(...signaturesData);

 while (true) {
   if (_signatures.length >= 1000 && signatures.length < (maxLength | 100000)) {
     const lastSignature = _signatures[_signatures.length - 1];
     _signatures = await connection.getSignaturesForAddress(address, {
       before: lastSignature.signature,
     });
    const signaturesData = await Promise.all(_signatures.map(sign => connection.getParsedTransaction(sign.signature, {maxSupportedTransactionVersion: 0})))
     signatures.push(...signaturesData);
     continue;
   }
   break;
 }
 return signatures;
}

(async () => {
  const data = await getSignaturesForAddress(solanaConnection, new solanaWeb3.PublicKey(walletToQuery), 100000)

  // write to json file
  fs.writeFileSync('data.json', JSON.stringify(data, null, 2));

  console.log(data)
})()

Transaction history statistics for a full wallet

I noticed trezor's history is pretty big, I didn't check if they were able to get the full history but I didn't struggle getting a 3+ years history.

While testing the RPCs, I noticed received SPL tokens transactions were not included by querying getSignaturesForAddress with the base wallet address, we need to call getSignaturesForAddress for each SPL Token account linked to get every TXs related with every tokens owned.

Achieved it by doing

async function getTokenAccounts(wallet, solanaConnection) {
    const filters = [
        {
          dataSize: 165,    //size of account (bytes)
        },
        {
          memcmp: {
            offset: 32,     //location of our query in the account (bytes)
            bytes: wallet,  //our search criteria, a base58 encoded string
          },            
        }];
    const accounts = await solanaConnection.getParsedProgramAccounts(
        TOKEN_PROGRAM_ID, //new PublicKey("TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA")
        {filters: filters}
    );

    return accounts;
}

// then
 for (const account of tokensAccounts) {
   const _signatures = await connection.getSignaturesForAddress(account.pubkey);
   const signaturesData = await Promise.all(_signatures.map(sign => connection.getParsedTransaction(sign.signature, {maxSupportedTransactionVersion: 0})))
   signatures.push(...signaturesData);
 }

With the current script, I've been using QuickNode as a provider where all their nodes are archive nodes containing the whole history:

Rate limit is 10 request by 1.5 second, according to quicknode, we might be able to upgrade to 15 request by second

1 year account with normal usage

Start time: 2024-09-05T15:19:09.314Z End time: 2024-09-05T15:20:22.220Z { numberRequests: 114, txNumber: 313 }

1.10m for a regular account

3 years account with pretty intensive usage

Start time: 2024-09-05T15:04:35.092Z End time: 2024-09-05T15:15:38.033Z { numberRequests: 2912, txNumber: 3797 }

9 mins and 2912 requests to retrieve full informations of every SPL accounts transactions including main account transactions

Node types

Resource: https://solana.com/developers/evm-to-svm/client-differences

There are 3 node types: Light node, Full Node, Archive Node image

Light node and Full node doesn't seems so reliable to retrieve the full transactions history as they are not able to retrieve the full history but only a recent portion.

Archive Node seems needed to retrieve the whole history, when testing I've been using QuickNode as a provider, but we have many solutions: Helius, QuickNode, dRPC...

I couldn't find any more informations on any other node types

0xean commented 6 days ago

Thanks @NeOMakinG -

Were you able to find any information re: websockets?

any WSS functionality gaps (next block, tx state, can we filter subscriptions by account)

NeOMakinG commented 6 days ago

Thanks @NeOMakinG -

Were you able to find any information re: websockets?

any WSS functionality gaps (next block, tx state, can we filter subscriptions by account)

Yep all details are inside https://github.com/shapeshift/web/issues/7565#issuecomment-2326403803

Websockets are working fine, I tested them every time I was looking for data, everything was working super fine using NowNodes, we can filter by account etc!

0xean commented 6 days ago

@kaladinlight - to take a look and getting some basic estimates on usage of tx history

we can look for a third part api for tx history and leverage NN for current data / RPC

NeOMakinG commented 4 days ago

Update:

https://discord.com/channels/554694662431178782/1283454541089603605/1283454541840253031

https://docs.helius.dev/solana-apis/enhanced-transactions-api/parsed-transaction-history#code-samples

We found out that Helius does index all the transactions on their side, using the Parse Transaction History endpoint, we can definitely reduce the usage and speed up the history retrieval!