search

VolumeFi / paloma

The fast blockchain messenger protocol
Apache License 2.0
3 stars 2 forks source link

Paloma validators are jailed #2201

Closed maharifu closed 5 days ago

maharifu commented 6 days ago

Tracking ticket for https://github.com/palomachain/paloma/issues/1299

maharifu commented 6 days ago

Some pigeons keep reporting RPC errors like:

Sep 23 12:42:40 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.queue-type":"evm/base-main/evm-turnstone-message","msg.args.message-type":"*type
s.Message_UpdateValset","msg.args.message-id":"307220","msg.args.message-nonce":"0x000000000004b014","msg.args.message-error-data":"0x","msg.args.message-public-access-data":"0x","component":"pigeon-status-update","status":"original error
: 429 Too Many Requests: <!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 8]>    <html class=\"no-j
s ie8 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->\n<head>\n<title>Access denied | base.llamarpc.com used Cloudflare to restrict access</title>\n<meta charset=\"UTF-8\
" />\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\" />\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n<meta name=\"viewport\" content=\"width=
device-width,initial-scale=1\" />\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/main.css\" />\n\n\n<script>\n(function(){if(document.addEventListener&&window.XMLHttpRequest&&JSON&&JSON.stringify){var e=function(a){
var c=document.getElementById(\"error-feedback-survey\"),d=document.getElementById(\"error-feedback-success\"),b=new XMLHttpRequest;a={event:\"feedback clicked\",properties:{errorCode:1015,helpful:a,version:1}};b.open(\"POST\",\"https://s
parrow.cloudflare.com/api/v1/event\");b.setRequestHeader(\"Content-Type\",\"application/json\");b.setRequestHeader(\"Sparrow-Source-Key\",\"c771f0e4b54944bebf4261d44bd79a1e\");\nb.send(JSON.stringify(a));c.classList.add(\"feedback-hidden\
");d.classList.remove(\"feedback-hidden\")};document.addEventListener(\"DOMContentLoaded\",function(){var a=document.getElementById(\"error-feedback\"),c=document.getElementById(\"feedback-button-yes\"),d=document.getElementById(\"feedbac
k-button-no\");\"classList\"in a&&(a.classList.remove(\"feedback-hidden\"),c.addEventListener(\"click\",function(){e(!0)}),d.addEventListener(\"click\",function(){e(!1)}))})}})();\n</script>\n\n<script defer src=\"https://performance.rada
r.cloudflare.com/beacon.js\"></script>\n</head>\n<body>\n  <div id=\"cf-wrapper\">\n    <div class=\"cf-alert cf-alert-error cf-cookie-error hidden\" id=\"cookie-alert\" data-translate=\"enable_cookies\">Please enable cookies.</div>\n
<div id=\"cf-error-details\" class=\"p-0\">\n      <header class=\"mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-full mb-15 antialiased\">\n         <h1 class=\"inline-block md:block mr-2 md:mb-2 font-light text-60 md:text-3xl text-black-dark
leading-tight\">\n           <span data-translate=\"error\">Error</span>\n           <span>1015</span>\n         </h1>\n         <span class=\"inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed\">Ray ID:
8c7a9daa7b4abebe &bull;</span>\n         <span class=\"inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed\">2024-09-23 12:42:39 UTC</span>\n        <h2 class=\"text-gray-600 leading-1.3 text-3xl lg:text-2
xl font-light\">You are being rate limited</h2>\n      </header>\n\n      <section class=\"w-240 lg:w-full mx-auto mb-8 lg:px-8\">\n          <div id=\"what-happened-section\" class=\"w-1/2 md:w-full\">\n            <h2 class=\"text-3xl l
eading-tight font-normal mb-4 text-black-dark antialiased\" data-translate=\"what_happened\">What happened?</h2>\n            <p>The owner of this website (base.llamarpc.com) has banned you temporarily from accessing this website.</p>\n
          \n          </div>\n\n          \n      </section>\n\n      <div class=\"feedback-hidden py-8 text-center\" id=\"error-feedback\">\n    <div id=\"error-feedback-survey\" class=\"footer-line-wrapper\">\n        Was this page help
ful?\n        <button class=\"border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded\" id=\"feedback-button-yes\" type=\"button\">Yes</button>\n        <button class=\"border border-solid bg-white cf-button cursor-po
inter ml-4 px-4 py-2 rounded\" id=\"feedback-button-no\" type=\"button\">No</button>\n    </div>\n    <div class=\"feedback-success feedback-hidden\" id=\"error-feedback-success\">\n        Thank you for your feedback!\n    </div>\n</div>
\n\n\n      <div class=\"cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300\">\n  <p class=\"text-13\">\n    <span class=\"cf-footer-item sm:blo
ck sm:mb-1\">Cloudflare Ray ID: <strong class=\"font-semibold\">8c7a9daa7b4abebe</strong></span>\n    <span class=\"cf-footer-separator sm:hidden\">&bull;</span>\n    <span id=\"cf-footer-item-ip\" class=\"cf-footer-item hidden sm:block s
m:mb-1\">\n      Your IP:\n      <button type=\"button\" id=\"cf-footer-ip-reveal\" class=\"cf-footer-ip-reveal-btn\">Click to reveal</button>\n      <span class=\"hidden\" id=\"cf-footer-ip\">2001:41d0:2:9678::1</span>\n      <span class
=\"cf-footer-separator sm:hidden\">&bull;</span>\n    </span>\n    <span class=\"cf-footer-item sm:block sm:mb-1\"><span>Performance &amp; security by</span> <a rel=\"noopener noreferrer\" href=\"https://www.cloudflare.com/5xx-error-landi
ng\" id=\"brand_link\" target=\"_blank\">Cloudflare</a></span>\n    \n  </p>\n  <script>(function(){function d(){var b=a.getElementById(\"cf-footer-item-ip\"),c=a.getElementById(\"cf-footer-ip-reveal\");b&&\"classList\"in b&&(b.classList.
remove(\"hidden\"),c.addEventListener(\"click\",function(){c.classList.add(\"hidden\");a.getElementById(\"cf-footer-ip\").classList.remove(\"hidden\")}))}var a=document;document.addEventListener&&a.addEventListener(\"DOMContentLoaded\",d)
})();</script>\n</div><!-- /.error-footer -->\n\n\n    </div><!-- /#cf-error-details -->\n  </div><!-- /#cf-wrapper -->\n\n  <script>\n  window._cf_translation = {};\n  \n  \n</script>\n\n</body>\n</html>\n\nenriched fields:\nmessage id[u
int64] = 0x4b014\nmessage type[*types.Message_UpdateValset] = &types.Message_UpdateValset{UpdateValset:(*types.UpdateValset)(0xc00012cdc8)}\n","sender":"palomavaloper13uslh0y22ffnndyr3x30wqd8a6peqh25m8p743","time":"2024-09-23T12:42:40Z","
message":"original error: 429 Too Many Requests: <!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 8
]>    <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->\n<head>\n<title>Access denied | base.llamarpc.com used Cloudflare to restrict access</title>
\n<meta charset=\"UTF-8\" />\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\" />\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n<meta name=\"vie
wport\" content=\"width=device-width,initial-scale=1\" />\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/main.css\" />\n\n\n<script>\n(function(){if(document.addEventListener&&window.XMLHttpRequest&&JSON&&JSON.strin
gify){var e=function(a){var c=document.getElementById(\"error-feedback-survey\"),d=document.getElementById(\"error-feedback-success\"),b=new XMLHttpRequest;a={event:\"feedback clicked\",properties:{errorCode:1015,helpful:a,version:1}};b.o
pen(\"POST\",\"https://sparrow.cloudflare.com/api/v1/event\");b.setRequestHeader(\"Content-Type\",\"application/json\");b.setRequestHeader(\"Sparrow-Source-Key\",\"c771f0e4b54944bebf4261d44bd79a1e\");\nb.send(JSON.stringify(a));c.classLis
t.add(\"feedback-hidden\");d.classList.remove(\"feedback-hidden\")};document.addEventListener(\"DOMContentLoaded\",function(){var a=document.getElementById(\"error-feedback\"),c=document.getElementById(\"feedback-button-yes\"),d=document.
getElementById(\"f

And others fail to get any validator balance:

Sep 23 12:42:40 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.error":"all balance requests failed","msg.args.chain-reference-id":"bnb-main","component":"pigeon-status-update","status":
"error attesting messages","sender":"palomavaloper109l8jp4zx9g8udetnh5qwgsze82meeq5yq06ta","time":"2024-09-23T12:42:40Z","message":"error attesting messages"}

At around 12:40 UTC, before a large group of validators was jailed, we can see many errors on base-main:

Sep 23 12:41:58 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not fou
nd\n","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper1wm4y8yhppxud6j5wvwr7fyynhh09tmv5fy845g","time":"2024-09-23T12:41:58Z","message":"error attesting messages"}
Sep 23 12:41:59 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not fou
nd\n","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper1swa5kcf9cl5dx2ypx0c5r9e5qdfnzp9w0yj7uv","time":"2024-09-23T12:41:59Z","message":"error attesting messages"}
Sep 23 12:41:59 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not fou
nd\n","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper1s0lankh33kprer2l22nank5rvsuh9ksafx3mze","time":"2024-09-23T12:41:59Z","message":"error attesting messages"}
Sep 23 12:41:59 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not fou
nd\n","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper19svt6tkvnu9wcjfwz2daglnxtm0frcavwjwcqe","time":"2024-09-23T12:41:59Z","message":"error attesting messages"}
Sep 23 12:41:59 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not fou
nd\n","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper15gvyk43x406v7kcd4rff5qfutqmcnpj3w9wpnm","time":"2024-09-23T12:41:59Z","message":"error attesting messages"}
Sep 23 12:41:59 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not fou
nd\n","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper13uslh0y22ffnndyr3x30wqd8a6peqh25m8p743","time":"2024-09-23T12:41:59Z","message":"error attesting messages"}
Sep 23 12:41:59 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not fou
nd\n","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper10y227j9d09pckexy32v2gckerj9a0kcewgf7xy","time":"2024-09-23T12:41:59Z","message":"error attesting messages"}
Sep 23 12:41:59 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not fou
nd\n","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper109l8jp4zx9g8udetnh5qwgsze82meeq5yq06ta","time":"2024-09-23T12:41:59Z","message":"error attesting messages"}
Sep 23 12:42:00 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not fou
nd\n","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper1z9fgzh7mzqgu33pdkxw0dqmqgm9l8exj6rl5wj","time":"2024-09-23T12:42:00Z","message":"error attesting messages"}
Sep 23 12:42:00 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not fou
nd\n","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper1wm4y8yhppxud6j5wvwr7fyynhh09tmv5fy845g","time":"2024-09-23T12:42:00Z","message":"error attesting messages"}
Sep 23 12:42:00 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not fou
nd\n","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper1tdw23fpnxh2uk3djtteh7eaydymrfgnaepkpkz","time":"2024-09-23T12:42:00Z","message":"error attesting messages"}
Sep 23 12:42:00 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not fou
nd\n","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper1pyrugdyg58k4h788pptnzdvxsu5z66pwhu69gs","time":"2024-09-23T12:42:00Z","message":"error attesting messages"}
Sep 23 12:42:00 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not fou
nd\n","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper1j2zvqhxqycxlxj3stnmun8060wfhfw57n3znmt","time":"2024-09-23T12:42:00Z","message":"error attesting messages"}
Sep 23 12:42:00 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not fou
nd\n","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper1hsmnzawmnauy9qxkxzjqm2mpd20pmfaf5uj30x","time":"2024-09-23T12:42:00Z","message":"error attesting messages"}
Sep 23 12:42:00 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not found\n","msg.args.chain-reference-id":"base-
main","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper1c8uxkkz00qn97wsp9um8y3a8wlmrv3c546jw4q","time":"2024-09-23T12:42:00Z","message":"error attesting messages"}
Sep 23 12:42:00 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not found\n","msg.args.chain-reference-id":"base-
main","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper10d4g6m6tv9y6yr7xn3jkpdz9jek95lc3pkmln4","time":"2024-09-23T12:42:00Z","message":"error attesting messages"}
Sep 23 12:42:00 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not fou
nd\n","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper16jhsvx4zrukkjqd9akfawx2tzduyx4ux8m9wre","time":"2024-09-23T12:42:00Z","message":"error attesting messages"}
Sep 23 12:42:00 mainnet-validator palomad[534816]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"base-main","msg.args.error":"there are 2 errors in the group:\nerror  1: not found\nerror  2: not fou
nd\n","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper12c6hhdt9ly838pw6ftwctfdautr8rp288atqnf","time":"2024-09-23T12:42:00Z","message":"error attesting messages"}

These errors would have prevented attestation of messages on other chains as well.

taariq commented 5 days ago

Closing as fixed.