cardano-community / guild-operators

Artifacts and scripts created by Guild operators
https://cardano-community.github.io/guild-operators
MIT License
354 stars 177 forks source link

gLiveView 1.25.1 with Node 1.33.0 (EKG not providing some metrics) #1280

Closed sp33dy closed 2 years ago

sp33dy commented 2 years ago

Hi,

I've just upgraded to Node 1.33.0 (VEGAS Pool) yesterday and also upgraded to 1.25.1 of the gLiveView script. When I now run it there is no connection (dies after x retries), yet, if I press any button, the panel pops up and it looks correctly populated; until it stops with connection error.

I've now debugged it and when it makes the connection to EKG to get back metrics, not all are returned, such as the nodeStartTime which is used to increase the retry count and ultimately failure.

Is there something wrong with my installation of the latest node? Or am I missing some requirement? Any advice would be appreciated. I've not had a problem with this brilliant tool before.

To Reproduce Steps to reproduce the behavior:

Likely to be unique to my case. Installed 1.33.0 Node and gLiveView 1.25.1 and run.

Expected behavior For the view panel to show and not time out

Screenshots Sample of EKG output:

{"iohk-monitoring version":{"type":"l","val":"0.1.10.1"},"rts":{"gc":{"current_bytes_used":{"type":"g","val":3652444552},"init_wall_ms":{"type":"c","val":7},"init_cpu_ms":{"type":"c","val":4},"max_bytes_used":{"type":"g","val":2630276208},"mutator_wall_ms":{"type":"c","val":33187247},"par_max_bytes_copied":{"type":"g","val":19932770328},"cpu_ms":{"type":"c","val":3860787},"current_bytes_slop":{"type":"g","val":148147832},"num_bytes_usage_samples":{"type":"c","val":22},"gc_cpu_ms":{"type":"c","val":1030822},"num_gcs":{"type":"c","val":18584},"wall_ms":{"type":"c","val":33367786},"par_tot_bytes_copied":{"type":"g","val":49070195728},"par_avg_bytes_copied":{"type":"g","val":49070195728},"gc_wall_ms":{"type":"c","val":180539},"bytes_copied":{"type":"c","val":49070195728},"max_bytes_slop":{"type":"g","val":53324688},"peak_megabytes_allocated":{"type":"g","val":7319},"mutator_cpu_ms":{"type":"c","val":2829966},"cumulative_bytes_used":{"type":"c","val":12668897688},"bytes_allocated":{"type":"c","val":2146147205840}}},"cardano":{"node":{"metrics":{"forks":{"int":{"type":"g","val":81}},"Mem":{"resident":{"int":{"type":"g","val":7701094400}}},"served":{"header":{"counter":{"int":{"type":"c","val":60000}}},"block":{"count":{"int":{"type":"g","val":4011}},"latest":{"count":{"int":{"type":"g","val":3809}}}}},"density":{"real":{"type":"l","val":"4.678565239993069e-2"}},"Stat":{"cputicks":{"int":{"type":"g","val":386077}},"threads":{"int":{"type":"g","val":42}}},"slotInEpoch":{"int":{"type":"g","val":216722}},"RTS":{"gcMajorNum":{"int":{"type":"g","val":22}},"gcHeapBytes":{"int":{"type":"g","val":7674527744}},"mutticks":{"int":{"type":"g","val":282994}},"gcLiveBytes":{"int":{"type":"g","val":3652444552}},"gcticks":{"int":{"type":"g","val":103082}},"gcMinorNum":{"int":{"type":"g","val":18562}}},"epoch":{"int":{"type":"g","val":315}},"slotNum":{"int":{"type":"g","val":50933522}},"connectedPeers":{"int":{"type":"g","val":14}},"blockfetchclient":{"lateblocks":{"type":"c","val":3},"blockdelay":{"cdfThree":{"type":"l","val":"0.9638888888888889"},"s":{"type":"l","val":"0.520647632"},"cdfFive":{"type":"l","val":"0.9972222222222222"},"cdfOne":{"type":"l","val":"0.26666666666666666"}},"blocksize":{"type":"g","val":53619}},"blockNum":{"int":{"type":"g","val":6771180}}}}},"ekg":{"server_timestamp_ms":{"type":"c","val":1642499820518}}}

Version:

Additional context

None

sp33dy commented 2 years ago

Temporarily, if I set the nodeStartTime = 1, it appears to be running perfectly. I'm worried that my relays/bp aren't installed correctly, although it all is reporting perfectly now.

rdlrt commented 2 years ago

It sounds like you might be running a custom commit that's not associated with a release (there was one before which did not include particular metric). Check the version in "env" present on each line in node logs to be sure the node was started with expected release

sp33dy commented 2 years ago

Thanks @rdlrt. I picked up the pre-built binary here:

https://hydra.iohk.io/build/11777304

I'll build the binary myself later to see if it improves. Welcome to close this ticket. My 'hack' is good enough for me for now.

Many thanks.

rdlrt commented 2 years ago

That build is against current master branch, the node release 1.33 can be found in release notes (direct link here )