ipbus / ipbus-firmware

Firmware that implements a reliable high-performance control link for particle physics electronics, based on the IPbus protocol
https://ipbus.web.cern.ch
Other
41 stars 31 forks source link

Request for hints on Debugging Control hub #40

Closed DavidCussans closed 6 years ago

DavidCussans commented 6 years ago

We are having troubles connecting via the control hub.

We have compiled IPBus and the control_hub on Ubuntu 14.04

IPBus communication direct to the hardware ( uri="ipbusudp-2.0://192.168.200.30:50001" ) works but is very slow.

The control hub seems to start up OK:

teleuser@azalea-runcontrol:~/software/eudaq-master$ /opt/cactus/bin/controlhub_info The ControlHub is running. General info ... started at 2018-04-18 14:19:27 UTC uptime: 0 days, 0 hours, 24 minutes, 19 seconds version: 2.6.0 config file: /etc/controlhub/default/sys.config

Device client parameters ... max packets in flight to each board = 16 timeout for replies from board = 20ms device client lifetime when inactive = 15000ms

TCP parameters ... Listening on port 10203


teleuser@azalea-runcontrol:~/software/eudaq-master$ /opt/cactus/bin/controlhub_status The ControlHub is up (version 2.6.0, pid=23498)


teleuser@azalea-runcontrol:~/software/eudaq-master$ /opt/cactus/bin/controlhub_stats Control Hub Stats Report

CLIENT All-time connections: 32 Active connections: 1 (peak: 1) @ 127.0.0.1:48440 Requests received: 1 (of which 0 were malformed) Responses sent: 1 Old clients Requests received: 31 (of which 0 were malformed) Responses sent: 31

Old UDP Packets Out: 32 Packets In: 0 Timeouts: 0 in normal operation (of which 0 were recovered) (0 request packets got lost, 0 reply packet) Other timeouts: 160 (status), 0 (in resends)


... however, when we try to connect via the control hub we see errors. Extract from the log:

18-04-18 16:21:14.006563 [7f231a191700] INFO - URI "chtcp-1.3://localhost:10203?target=192.168.200.30:50001" parsed as:

protocol : chtcp-1.3 hostname : localhost port : 10203 path : extension : arguments : target = 192.168.200.30:50001

18-04-18 16:21:14.006763 [7f231a191700] INFO - Converted IP address string "192.168.200.30:50001" to 192.168.200.30:50001 and converted this to IP 0xC0A8C81E, port 0xC351 18-04-18 16:21:14.007685 [7f231a191700] INFO - Attempting to create TCP connection to 'localhost' port 10203. 18-04-18 16:21:14.007847 [7f231a191700] INFO - TCP connection succeeded 18-04-18 16:21:14.113072 [7f231977e700] ERROR - The ControlHub did not receive any response from the target with URI "chtcp-1.3://localhost:10203?target=192.168.200.30:50001" 18-04-18 16:21:14.113114 [7f231977e700] ERROR - ControlHub returned error code 3 = 'no reply to status packet' 18-04-18 16:21:14.113311 [7f231a191700] WARNING - Closing TCP socket for device with URI "chtcp-1.3://localhost:10203?target=192.168.200.30:50001" since exception detected. 18-04-18 16:21:14.113375 [7f231a191700] INFO - function "void uhal::ControlHub::dispatchExceptionHandler() [with InnerProtocol = uhal::IPbus<1u, 3u>]" in /home/teleuser/software/ipbus-software/uhal/uhal/src/common/ProtocolControlHub.cpp, line 292. 18-04-18 16:21:14.113395 [7f231a191700] INFO - function "void uhal::IPbus<1u, IPbus_minor>::dispatchExceptionHandler() [with unsigned char IPbus_minor = 3u]" in /home/teleuser/software/ipbus-software/uhal/uhal/src/common/ProtocolIPbus.cpp, line 218. 18-04-18 16:21:14.113408 [7f231a191700] INFO - function "virtual void uhal::IPbusCore::dispatchExceptionHandler()" in /home/teleuser/software/ipbus-software/uhal/uhal/src/common/ProtocolIPbusCore.cpp, line 577.

Does anybody have any ideas about how to debug this issue?

We used almost identical firmware in the past with no problems, using a CentOS7 machine running the control hub and our IPBus application.

Many Thanks!

David

dpcsankey commented 6 years ago

chtcp-2.0 will probably work better.

DavidCussans commented 6 years ago

Dave, Thanks . Yes, indeed chtcp-2.0 works better

The online documentation confused me - I just cut and pasted from a line mentioning chtcp-1.3 in https://ipbus.web.cern.ch/ipbus/doc/user/html/software/uhalQuickTutorial.html#creating-a-connection-file , although it does mention chtcp-2.0 later on.

tswilliams commented 6 years ago

Fair point; putting chtcp-1.3 next to ipbusudp-2.0 did make that part of the documentation a bit confusing. I've updated it to make it clearer (you may need to force a refresh of the page in your browser to see the new copy).

Cheers, Tom

DavidCussans commented 6 years ago

Tom - Thanks! Never underestimate the stupidity of your users. Especially those in a hurry in a beam-test control room.