search

bcgov / indy-tails-server

This software stores and makes available tails files for use with Hyperledger Indy
Apache License 2.0
27 stars 53 forks source link

CoreDump in zmq #39

Open tkdp opened 2 years ago

tkdp commented 2 years ago

hello,

since weare behind a company firewall we are using the tails-server with the Socks-proxy pull request https://github.com/bcgov/indy-tails-server/pull/38; the tails-file server runs in OpenShift and tries to connect to the IDUnion indy-network (https://idunion.org/?lang=en);

Problem is that zmq creates a CoreDumped (currently we are not able to retrieve the coredump-file cause of company and OpenShift); log is given below:

remark: a similar problem we have on the agent side, where we switched from Indy to Asker...

=> has someone an idea how to investigate this problem further?

[2022-08-09T08:34:44Z DEBUG indy_vdr::ffi] Initialized default logger [2022-08-09T08:34:44Z DEBUG indy_vdr::ffi] Setting pool socks proxy: pxvip02.intranet.commerzbank.com:1080 [2022-08-09T08:34:44Z INFO indy_vdr::pool::genesis] Skipped: Node 'xxx' is not a validator [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::pool] New RequestHandle(1): reqId() [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] New ZMQConnectionHandle(1) [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Open new socket for node ING [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Use socks proxy: pxvip02.intranet.commerzbank.com:1080 [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Open new socket for node xxx [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Use socks proxy: pxvip02.intranet.commerzbank.com:1080 [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Open new socket for node xxx [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Use socks proxy: pxvip02.intranet.commerzbank.com:1080 [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Open new socket for node xxx [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Use socks proxy: pxvip02.intranet.commerzbank.com:1080 [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Open new socket for node xxx [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Use socks proxy: pxvip02.intranet.commerzbank.com:1080 [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Open new socket for node xxx [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Use socks proxy: pxvip02.intranet.commerzbank.com:1080 [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Open new socket for node xxx [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Use socks proxy: pxvip02.intranet.commerzbank.com:1080 [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Open new socket for node regio_iT [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Use socks proxy: pxvip02.intranet.commerzbank.com:1080 [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Open new socket for node xxx [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Use socks proxy: pxvip02.intranet.commerzbank.com:1080 [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Open new socket for node xxx [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Use socks proxy: pxvip02.intranet.commerzbank.com:1080 [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Open new socket for node xxx [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Use socks proxy: pxvip02.intranet.commerzbank.com:1080 [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Open new socket for node xxx [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Use socks proxy: pxvip02.intranet.commerzbank.com:1080 [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::helpers] Catchup target found 7YnD1B5iRy79QcBx1DFqiyu6j3fsyHiE13f4H4KEspLm 26 Some({"xxx": -1.0, "xxx": 0.290305, "xxx": -1.0, "xxx": -1.0, "xxx": -1.0, "xxx": 0.31385347, "xxx": 0.29052326, "xxx": -1.0, "xxx": 0.30031052, "xxx": -1.0, "xxx": -1.0, "xxx": -1.0}) [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::pool] New RequestHandle(2): reqId() [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] New ZMQConnectionHandle(2) [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Open new socket for node siemens [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Use socks proxy: pxvip02.intranet.commerzbank.com:1080 [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Open new socket for node tubzecm [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Use socks proxy: pxvip02.intranet.commerzbank.com:1080 [2022-08-09T08:34:44Z INFO indy_vdr::pool::helpers] Catchup completed Some({"siemens": 0.20512652, "tubzecm": -1.0}) [2022-08-09T08:34:44Z DEBUG indy_vdr::ffi::pool] Adding 11 new pool transactions [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::runner] Drop pool runner thread [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::runner] Pool thread ended [2022-08-09T08:34:44Z DEBUG indy_vdr::pool::networker::zmq] Drop networker thread [2022-08-09T08:34:44Z DEBUG zmq] socket dropped [2022-08-09T08:34:44Z DEBUG zmq] socket dropped [2022-08-09T08:34:44Z DEBUG zmq] socket dropped [2022-08-09T08:34:44Z DEBUG zmq] socket dropped [2022-08-09T08:34:44Z DEBUG zmq] socket dropped [2022-08-09T08:34:44Z DEBUG zmq] socket dropped [2022-08-09T08:34:44Z DEBUG zmq] socket dropped [2022-08-09T08:34:44Z DEBUG zmq] socket dropped [2022-08-09T08:34:44Z DEBUG zmq] socket dropped [2022-08-09T08:34:44Z DEBUG zmq] socket dropped [2022-08-09T08:34:44Z DEBUG zmq] socket dropped [2022-08-09T08:34:44Z DEBUG zmq] socket dropped [2022-08-09T08:34:44Z DEBUG zmq] socket dropped [2022-08-09T08:34:44Z DEBUG zmq] socket dropped [2022-08-09T08:34:44Z DEBUG zmq] socket dropped [2022-08-09T08:34:44Z DEBUG zmq] socket dropped [2022-08-09T08:34:44Z DEBUG zmq] context dropped Bad file descriptor (/github/home/.cargo/registry/src/github.com-1ecc6299db9ec823/zeromq-src-0.1.10+4.3.2/vendor/src/epoll.cpp:113) ./wrapper.sh: line 21: 12 Aborted (core dumped) tails-server $1 $2 $3 $4 $5

swcurran commented 2 years ago

@andrewwhitehead -- any thoughts on this?

andrewwhitehead commented 2 years ago

Interesting, haven't seen that before. Seems likely that it could be related to the SOCKS proxy support in either the Rust wrapper or the underlying zeromq library. My first guess would be that it's trying to close a file descriptor more than once, but it's hard to say.

tkdp commented 2 years ago

Hello, i solved the problem (at least for my environment).... i adapted the underlying indy-vdr build, in Detail i removed the zmq_vendored stuff from

[features]
 fetch = ["hyper-tls"]
-zmq_vendored = ["indy-vdr/zmq_vendored"]
-default = ["fetch", "zmq_vendored"]

after that i could not reproduce "my" core-dump anymore and my tails-file-server was running; this solved also the same acapy-agent core-dump-problem; the agent is now also running without problems;

=> i'm currently not sure how to deal with that situation, since i do not really / seriously can explain the core-dump-cause; => ?PR respective to indy-vdr?