search

holaplex / indexer

Index Solana data using a Geyser plugin (downstream service cluster)
https://holaplex.com
GNU Affero General Public License v3.0
162 stars 38 forks source link

Indexer is not consuming any data from validator #743

Closed Maverick9081 closed 2 years ago

Maverick9081 commented 2 years ago

i'm trying to setup holaplex indexer ,First a install diesel and compile the geyserplugin from their repo. then i run the 3 services simultaneously with following commands.

cargo run --bin holaplex-indexer-geyser --release --features geyser -- --amqp-url amqp://abhi:abhi@localhost --network mainnet --meili-url http://127.0.0.1:7700/ --meili-key f2d9822d8b40f1f05d6d92710c7c2e4051159d6c4572b6c3254ede368eb7f743 --database-url postgres://postgres:holap1ex@localhost:5337/holaplex-indexer


cargo run --bin holaplex-indexer-http --features http -- --entity metadata-json --amqp-url amqp://abhi:abhi@localhost --sender mainnet --meili-url http://127.0.0.1:7700 --meili-key f2d9822d8b40f1f05d6d92710c7c2e4051159d6c4572b6c3254ede368eb7f743 --database-url postgres://postgres:holap1ex@localhost:5337/holaplex-indexer --asset-proxy-endpoint https://assets[n].holaplex.tools/ --asset-proxy-count 5 --timeout 10 -- --release


cargo run --bin holaplex-indexer-http --features http -- --entity store-config --amqp-url amqp://abhi:abhi@localhost --sender mainnet --meili-url http://127.0.0.1:7700 --meili-key f2d9822d8b40f1f05d6d92710c7c2e4051159d6c4572b6c3254ede368eb7f743 --database-url postgres://postgres:holap1ex@localhost:5337/holaplex-indexer --asset-proxy-endpoint https://assets[n].holaplex.tools/ --asset-proxy-count 5 --timeout 10 -- --release

Then i run the Graphql server with this command 

cargo run --bin holaplex-indexer-graphql --asset-proxy-endpoint https://assets[n].holaplex.tools/ --asset-proxy-count 5 --meili-url http://127.0.0.1:7700/ --meili-key f2d9822d8b40f1f05d6d92710c7c2e4051159d6c4572b6c3254ede368eb7f743 --solana-endpoint https://holaplex.rpcpool.com/ --pre-query-search-limit 1000 --addr [::]:3000 --database-url postgres://postgres:holap1ex@localhost:5337/holaplex-indexer --follow-wallets-exclusions tsU33UT3K2JTfLgHUo7hdzRhRe4wth885cqVbM8WLiq,ho1aVYd4TDWCi1pMqFvboPPc3J13e4LgWkWzGJpPJty --featured-listings-auction-houses 9SvsTjqk3YoicaYnC4VW1f8QAN9ku7QCCk6AyfUdzc9t --marketplaces-store-address-exclusions 3doAaFs2VuTLnVTPLZwFAWsskqwwC4xLt31dZ24uwYsd

And everything is running fine and all are connected to the rabbitmq instance. Then i run the test-validator node on mainnet with this sample geyser config

{
  "amqp": {
    "network": "mainnet",
    "address": "amqp://abhi:abhi@localhost"
  },
  "jobs": {
    "limit": 16
  },
  "metrics": {
   "config": "host="
  },
  "accounts": {
    "owners": [
      "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s",
      "vau1zxA2LbssAUEF7Gpw91zMM1LvXrvpzJtmZ58rPsn",
      "auctxRXPeJoc4817jDhf4HbjnhEcr1cCXenosMhK5R8",
      "p1exdMJcjVao65QdewkaZRUnU6VPSXhus9n2GzWfh98",
      "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA",
      "hausS13jsjafwWwGqZTUQRmWyvyxn9EQpqMwV1PBBmk",
      "grphSXQnjAoPXSG5p1aJ7ZFw2A1akqP3pkXvjfbSJef",
      "cndy3Z4yapfJBmL3ShUp5exZKqR3z33thTzeNMm2gRZ",
      "namesLPneVptA9Z5rqUDD9tMTWEJwofgaYwp8cawRkX",
      "mgr99QFMYByTqGPWmNqunV7vBLmWWXdSrHUfV8Jf3JM",
      "pcaBwhJ1YHp7UDA7HASpQsRUmUNwzgYaLQto2kSj1fR",
      "tmeEDp1RgoDtZFtx6qod3HkbQmv9LMe36uqKVvsLTDE",
      "useZ65tbyvWpdYCLDJaegGK34Lnsi8S3jZdwx8122qp",
      "LocktDzaV1W2Bm9DeZeiyz4J9zs4fRqNiYqQyracRXw",
      "Govz1VyoyLD5BL6CSCxUJLVLsQHRwjfFj1prNsdNg5Jw",
      "GokivDYuQXPZCWRkwMhdH2h91KpDQXBEmpgBgs55bnpH",
      "TBondmkCYxaPCKG4CHYfVTcwQ8on31xnJrPzk8F8WsS",
      "nameXpT2PwZ2iA6DTNYTotTmiMYusBCYqwBLN2QgF4w",
      "HAbiTatJVqoCJd9asyr6RxMEdwtfrQugwp7VAFyKWb1g",
      "GovER5Lthms3bLBqWub97yVrMmEogzX7xNjdXpPPCVZw"
    ],
    "pubkeys": [],
    "startup": false
  },
  "instructions": {
    "programs": [
      "M2mx93ekt1fmXSVkTrUL9xVFHkmME8HTUi5Cyc5aF7K",
      "MEisE1HzehtrDpAAT8PnLHjpSSkRYakotTuJRPjTpo8",
      "hausS13jsjafwWwGqZTUQRmWyvyxn9EQpqMwV1PBBmk",
      "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA",
      "4kAkuX3eqqb6dFnpbBtbAi9g3tswyAEyns8kDE4nYuvo"
    ]
  },
  "libpath": "./libholaplex_indexer_rabbitmq_geyser.so"
}

The validator starts fine but even after 2 hours there is no datainput on the postgres database or no input or output of data on validator connected with rabbitmq. But it is upposed to work on test validator.

Then i try to run real validator on dev-net with the same config.It catches up with the current block then it gets some geyser samples then it just gets wipes out of terminal (crashes likely) but nothing about crash in logs. The logs ends with this following line 

[2022-09-12T13:44:23.389461986Z INFO  solana_metrics::counter] COUNTER:{"name": "geyser_recvs", "counts": 14928001, "samples": 14928000,  "now": 1662990263389, "events": 1}

I start the validator with this command 

solana-validator   --identity ~/validator-keypair.json   --vote-account ~/vote-account-keypair.json   --rpc-port 8899   --entrypoint entrypoint.devnet.solana.com:8001   --limit-ledger-size --no-port-check --no-poh-speed-test  --geyser-plugin-config ../Config.json

I get some movement on rabbit mq before the validator crashes. @imabdulbasit, @mpwsh , @kespinola please help, this project is urgent

kespinola commented 2 years ago

@mpwsh can you share the solana-validator command to run? The validtor shouldn't be voting but I see a --vote-account option specified.

You'll also need to provide your own --solana-endpoint https://holaplex.rpcpool.com/ as this one is only configured to work with our servers.

Other item to consider is the plugin should be built with the same version of solana that the validator is running.

Maverick9081 commented 2 years ago

ok, i will change the --solana-endpoint to my node rpc port, Should i change something from config as well?

Maverick9081 commented 2 years ago

https://github.com/holaplex/kubes/blob/dev/validator/mainnet/geyser-config.json#L11 Found this what should i Replace it with? or is it ok to leave it empty for now?

mpwsh commented 2 years ago

Hi @Maverick9081! You can leave the metrics config out for now if not using it. How's your resource consumption when running the validator? Looks like the server is going out of memory?

you can also check the startup commands we use for the validators for mainnet and devnet

A temporary solution (just to test things out) is to create a huge swap file, like 256GB or bigger. This will help you debug a bit more.
Your node might end up falling behind if it doesnt respond fast enough

Maverick9081 commented 2 years ago

Hey @mpwsh @kespinola , i Used a swap file for validator , now its running fine,It syncs up to the current version, and it keeps sending the geyser samples. Like image But the data is not being consumed.MY first service lokks like this 

     Running `target/release/holaplex-indexer-geyser --amqp-url 'amqp://guest:guest@localhost' --network mainnet --meili-url 'http://127.0.0.1:7700/' --meili-key a7038fd48402b1198a9530cf4253369718fa99200bb1470eedce84ebd78d553f --database-url 'postgres://postgres:holap1ex@localhost:5337/holaplex-indexer'`
[2022-09-16T11:50:49Z WARN  holaplex_indexer_core::db] Cannot determine if database is writable; assuming yes
[2022-09-16T11:50:50Z WARN  holaplex_indexer::geyser::client] Disabling Dialect integration

second and third both look like this for the whole duration.

[2022-09-16T12:42:34Z DEBUG lapin::channels] send heartbeat
[2022-09-16T12:42:36Z DEBUG lapin::channels] received heartbeat from server

and the graphql commad looks like this the wholetime

[2022-09-16T11:50:22Z INFO  holaplex_indexer_graphql] Listening on [::]:3000
[2022-09-16T11:50:22Z DEBUG holaplex_indexer_core::db] Connecting to db: "postgres://postgres:holap1ex@localhost:5337/holaplex-indexer"
[2022-09-16T11:50:22Z INFO  holaplex_indexer_core::db] Not running migrations over a read-only connection
[2022-09-16T11:50:23Z INFO  actix_server::builder] Starting 16 workers
[2022-09-16T11:50:23Z INFO  actix_server::server] Actix runtime found; starting in Actix runtime

But the rabbitmq queues are empty image I run the three servies on different terminals with the initial commands that i shared. And run the graphql crate with this command 

./holaplex-indexer-graphql --asset-proxy-endpoint https://assets[n].holaplex.tools/ --asset-proxy-count 5 --meili-url http://127.0.0.1:7700/ --meili-key a7038fd48402b1198a9530cf4253369718fa99200bb1470eedce84ebd78d553f --solana-endpoint localhost:8899 --pre-query-search-limit 1000 --addr [::]:3000 --database-url postgres://postgres:holap1ex@localhost:5337/holaplex-indexer --follow-wallets-exclusions tsU33UT3K2JTfLgHUo7hdzRhRe4wth885cqVbM8WLiq,ho1aVYd4TDWCi1pMqFvboPPc3J13e4LgWkWzGJpPJty --featured-listings-auction-houses 9SvsTjqk3YoicaYnC4VW1f8QAN9ku7QCCk6AyfUdzc9t --marketplaces-store-address-exclusions 3doAaFs2VuTLnVTPLZwFAWsskqwwC4xLt31dZ24uwYsd

after some hours validator is logging this 

[2022-09-16T13:18:52.348038159Z INFO  solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=7i no_ports_delta=0i in_errors_delta=0i out_datagrams_delta=8i rcvbuf_errors_delta=0i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=800688i rcvbuf_errors=800688i sndbuf_errors=0i
[2022-09-16T13:18:52.848187406Z INFO  solana_metrics::metrics] datapoint: memory-stats total=66707181568i swap_total=268435452i free_percent=13.964591405355776 used_bytes=26371301376i avail_percent=60.46707302553682 buffers_percent=0.08870845778378188 cached_percent=45.44951155085803 swap_free_percent=82.35578510695376
[2022-09-16T13:18:52.848662960Z INFO  solana_metrics::metrics] datapoint: cpu-stats cpu_num=32i cpu0_freq_mhz=2800i average_load_one_minute=1.45 average_load_five_minutes=1.34 average_load_fifteen_minutes=4.68 total_num_threads=2060i

and there are no data I/O on rabbitmq now

Maverick9081 commented 2 years ago

@mpwsh @kespinola @imabdulbasit Please help me with this

Maverick9081 commented 2 years ago

Finally solved, I only had to pass --startup all in the geyser indexer service.

Maverick9081 commented 2 years ago

@kespinola @mpwsh can you tell me the ideal specs to run the indexer ? Cause i ran it on 32v Cpu and 64 Ram with 256 GB swap memory, still Ram usage went to 250GB after sometime, And it was still increasing.

mpwsh commented 2 years ago

Hi @Maverick9081, glad you got that sorted out :) You should be able to handle the load with 256 GB of ram and a big swap. You also want to have 2 different SSDs, one for the ledger and another one for accounts, so you can keep up.

We run 2 geyser consumers at the same time, one with the --startup argument in all and the other one in normal so we can have the geyser consumer join in both queues, the startup messages one and the normal as well

Example:

#First -- Startup normal
./bin/holaplex-indexer-geyser --network=mainnet --startup=normal --meili-url $MEILI_URL --meili-key $MEILI_KEY --dialect-api-key $DIALECT_API_KEY --dialect-api-endpoint $DIALECT_API_ENDPOINT &
# Second -- startup all
./bin/holaplex-indexer-geyser --network=mainnet --startup=all --meili-url $MEILI_URL --meili-key $MEILI_KEY  --dialect-api-key $DIALECT_API_KEY --dialect-api-endpoint $DIALECT_API_ENDPOINT
Maverick9081 commented 2 years ago

@mpwsh is there any other way of not to run the validator .

Deploying my Auction house and replacing it with the holaplex AH address won't work right cause the new AH address won't be there in geyser config.Right? So is there any alternative to validator?