Exporter stops with "missed checkpoint, exiting "

egrigore commented 4 years ago

Hi,

We rebased to the labgrid version from 31 march: b0cc54dc2e2c9cf94c9c98e22f00a150ce475597 . All of our exporters are set up as services, since we made the rebase they are restarting at every 10 minutes with reason "missed checkpoint, exiting". There are many changes in exporter.py made in january - is any configuration that we have to make?

Below is the log of an exporter from when it starts to when it stops:

apr 08 10:54:04 exporter-workstation systemd[1]: Started labgrid exporter service.
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]: crossbar URL: ws://coordinatorIP:20408/ws
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]: crossbar realm: realm1
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]: exporter name: exporter-workstation
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]: exporter hostname: exporter-workstation.domain.net
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]: resource config file: /labgrid-location/../labgrid-config/exporters/TAG--exporter-workstation_exporter.yaml
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]: SessionDetails(realm=<realm1>,
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                session=6485400199526595,
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                authid=<exporter/exporter-workstation>,
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                authrole=<public>,
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                authmethod=ticket,
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                authprovider=dynamic,
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                authextra={'x_cb_node_id': None, 'x_cb_peer': 'tcp4:dummyIP:38508', 'x_cb_pid': 23189},
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                serializer=<cbor.batched>,
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                resumed=None,
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                resumable=None,
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                resume_token=None)
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]: ('rendered',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  'BOARD--TYPE1--1:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    location: BOARD--TYPE1--1\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    USBSerialPort:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '        match:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "            'ID_PATH': 'pci-0000:00:14.0-usb-0:3.4.4:1.0'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    UniversalUpdateUtility:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '        match:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "            'DEVPATH': "
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "'/devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3.4/3-3.4.3'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    NetworkPowerPort:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "        model: 'simplehttp'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "        host: 'dummyIP:8085/pduIP'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '        index: 8\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    NetworkService:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "        address: 'dummyIP'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "        username: 'bamboo'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  'BOARD--TYPE1--2:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    location: BOARD--TYPE1--2\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    USBSerialPort:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '        match:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "            'ID_PATH': 'pci-0000:00:14.0-usb-0:3.4.2:1.0'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    UniversalUpdateUtility:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '        match:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "            'DEVPATH': "
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "'/devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3.4/3-3.4.1'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    NetworkPowerPort:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "        model: 'simplehttp'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "        host: 'dummyIP:8085/pduIP'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '        index: 7\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    NetworkService:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "        address: 'dummyIP'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "        username: 'bamboo'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  'BOARD--TYPE2--1:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    location: BOARD--TYPE2--1\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    USBSerialPort:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '        match:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "            'ID_PATH': 'pci-0000:00:14.0-usb-0:3.3:1.0'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    UniversalUpdateUtility:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '        match:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "            'DEVPATH': '/devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3.2'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    NetworkPowerPort:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "        model: 'simplehttp'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "        host: 'dummyIP:8085/pduIP'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '        index: 6\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    NetworkService:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "        address: 'dummyIP'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "        username: 'bamboo'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  'BOARD--TYPE3--1:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    location: BOARD--TYPE3--1\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    USBSerialPort:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '        match:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "            'ID_PATH': 'pci-0000:00:14.0-usb-0:2.4.4:1.0'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    UniversalUpdateUtility:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '        match:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "            'DEVPATH': '/devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3.1'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    NetworkPowerPort:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "        model: 'simplehttp'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "        host: 'dummyIP:8085/pduIP'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '        index: 5\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  '    NetworkService:\n'
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "        address: 'dummyIP'\n"
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  "        username: 'bamboo'")
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]: ('loaded',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:  OrderedDict([('BOARD--TYPE1--1',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                OrderedDict([('location', 'BOARD--TYPE1--1'),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                             ('USBSerialPort',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                              OrderedDict([('match',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                            OrderedDict([('ID_PATH',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                                          'pci-0000:00:14.0-usb-0:3.4.4:1.0')]))])),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                             ('UniversalUpdateUtility',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                              OrderedDict([('match',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                            OrderedDict([('DEVPATH',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                                          '/devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3.4/3-3.4.3')]))])),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                             ('NetworkPowerPort',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                              OrderedDict([('model', 'simplehttp'),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                           ('host',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                            'dummyIP:8085/pduIP'),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                           ('index', 8)])),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                             ('NetworkService',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                              OrderedDict([('address', 'dummyIP'),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                           ('username', 'bamboo')]))])),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:               ('BOARD--TYPE1--2',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                OrderedDict([('location', 'BOARD--TYPE1--2'),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                             ('USBSerialPort',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                              OrderedDict([('match',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                            OrderedDict([('ID_PATH',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                                          'pci-0000:00:14.0-usb-0:3.4.2:1.0')]))])),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                             ('UniversalUpdateUtility',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                              OrderedDict([('match',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                            OrderedDict([('DEVPATH',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                                          '/devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3.4/3-3.4.1')]))])),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                             ('NetworkPowerPort',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                              OrderedDict([('model', 'simplehttp'),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                           ('host',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                            'dummyIP:8085/pduIP'),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                           ('index', 7)])),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                             ('NetworkService',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                              OrderedDict([('address', 'dummyIP'),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                           ('username', 'bamboo')]))])),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:               ('BOARD--TYPE2--1',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                OrderedDict([('location', 'BOARD--TYPE2--1'),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                             ('USBSerialPort',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                              OrderedDict([('match',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                            OrderedDict([('ID_PATH',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                                          'pci-0000:00:14.0-usb-0:3.3:1.0')]))])),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                             ('UniversalUpdateUtility',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                              OrderedDict([('match',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                            OrderedDict([('DEVPATH',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                                          '/devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3.2')]))])),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                             ('NetworkPowerPort',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                              OrderedDict([('model', 'simplehttp'),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                           ('host',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                            'dummyIP:8085/pduIP'),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                           ('index', 6)])),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                             ('NetworkService',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                              OrderedDict([('address', 'dummyIP'),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                           ('username', 'bamboo')]))])),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:               ('BOARD--TYPE3--1',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                OrderedDict([('location', 'BOARD--TYPE3--1'),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                             ('USBSerialPort',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                              OrderedDict([('match',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                            OrderedDict([('ID_PATH',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                                          'pci-0000:00:14.0-usb-0:2.4.4:1.0')]))])),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                             ('UniversalUpdateUtility',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                              OrderedDict([('match',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                            OrderedDict([('DEVPATH',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                                          '/devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3.1')]))])),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                             ('NetworkPowerPort',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                              OrderedDict([('model', 'simplehttp'),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                           ('host',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                            'dummyIP:8085/pduIP'),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                           ('index', 5)])),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                             ('NetworkService',
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                              OrderedDict([('address', 'dummyIP'),
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]:                                           ('username', 'bamboo')]))]))]))
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]: add resource BOARD--TYPE1--1/USBSerialPort: USBSerialPort/OrderedDict([('match', OrderedDict([('ID_PATH', 'pci-0000:00:14.0-usb-0:3.4.4:1.0')]))])
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkSerialPort', 'params': OrderedDict(), 'acquired': None, 'avail': False}
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]: 2020-04-08T10:54:05 started ser2net for /dev/ttyUSB0 on port 40637
apr 08 10:54:06 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkSerialPort', 'params': OrderedDict([('host', 'exporter-workstation.domain.net'), ('port', 40637), ('extra', {'path': '/dev/ttyUSB0', 'proxy_required': False, 'proxy': 'exporter-workstation'})]), 'acquired': 'BOARD--TYPE1--1', 'avail': True}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: add resource BOARD--TYPE1--1/UniversalUpdateUtility: UniversalUpdateUtility/OrderedDict([('match', OrderedDict([('DEVPATH', '/devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3.4/3-3.4.3')]))])
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkUniversalUpdateUtility', 'params': OrderedDict(), 'acquired': None, 'avail': False}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkUniversalUpdateUtility', 'params': OrderedDict([('host', 'exporter-workstation.domain.net'), ('busnum', 3), ('devnum', 73), ('path', '3-3.4.3'), ('vendor_id', 1234), ('model_id', 56789), ('extra', {'proxy_required': False, 'proxy': 'exporter-workstation'})]), 'acquired': 'BOARD--TYPE1--1', 'avail': True}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: add resource BOARD--TYPE1--1/NetworkPowerPort: NetworkPowerPort/OrderedDict([('model', 'simplehttp'), ('host', 'dummyIP:8085/pduIP'), ('index', 8)])
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkPowerPort', 'params': OrderedDict([('model', 'simplehttp'), ('host', 'dummyIP:8085/pduIP'), ('index', 8)]), 'acquired': None, 'avail': True}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkPowerPort', 'params': OrderedDict([('model', 'simplehttp'), ('host', 'dummyIP:8085/pduIP'), ('index', 8)]), 'acquired': 'BOARD--TYPE1--1', 'avail': True}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: add resource BOARD--TYPE1--1/NetworkService: NetworkService/OrderedDict([('address', 'dummyIP'), ('username', 'bamboo')])
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkService', 'params': OrderedDict([('address', 'dummyIP'), ('username', 'bamboo')]), 'acquired': None, 'avail': True}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkService', 'params': OrderedDict([('address', 'dummyIP'), ('username', 'bamboo')]), 'acquired': 'BOARD--TYPE1--1', 'avail': True}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: add resource BOARD--TYPE1--2/USBSerialPort: USBSerialPort/OrderedDict([('match', OrderedDict([('ID_PATH', 'pci-0000:00:14.0-usb-0:3.4.2:1.0')]))])
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkSerialPort', 'params': OrderedDict(), 'acquired': None, 'avail': False}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: add resource BOARD--TYPE1--2/UniversalUpdateUtility: UniversalUpdateUtility/OrderedDict([('match', OrderedDict([('DEVPATH', '/devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3.4/3-3.4.1')]))])
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkUniversalUpdateUtility', 'params': OrderedDict(), 'acquired': None, 'avail': False}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: add resource BOARD--TYPE1--2/NetworkPowerPort: NetworkPowerPort/OrderedDict([('model', 'simplehttp'), ('host', 'dummyIP:8085/pduIP'), ('index', 7)])
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkPowerPort', 'params': OrderedDict([('model', 'simplehttp'), ('host', 'dummyIP:8085/pduIP'), ('index', 7)]), 'acquired': None, 'avail': True}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: add resource BOARD--TYPE1--2/NetworkService: NetworkService/OrderedDict([('address', 'dummyIP'), ('username', 'bamboo')])
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkService', 'params': OrderedDict([('address', 'dummyIP'), ('username', 'bamboo')]), 'acquired': None, 'avail': True}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: add resource BOARD--TYPE2--1/USBSerialPort: USBSerialPort/OrderedDict([('match', OrderedDict([('ID_PATH', 'pci-0000:00:14.0-usb-0:3.3:1.0')]))])
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkSerialPort', 'params': OrderedDict(), 'acquired': None, 'avail': False}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: add resource BOARD--TYPE2--1/UniversalUpdateUtility: UniversalUpdateUtility/OrderedDict([('match', OrderedDict([('DEVPATH', '/devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3.2')]))])
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkUniversalUpdateUtility', 'params': OrderedDict(), 'acquired': None, 'avail': False}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: add resource BOARD--TYPE2--1/NetworkPowerPort: NetworkPowerPort/OrderedDict([('model', 'simplehttp'), ('host', 'dummyIP:8085/pduIP'), ('index', 6)])
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkPowerPort', 'params': OrderedDict([('model', 'simplehttp'), ('host', 'dummyIP:8085/pduIP'), ('index', 6)]), 'acquired': None, 'avail': True}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: add resource BOARD--TYPE2--1/NetworkService: NetworkService/OrderedDict([('address', 'dummyIP'), ('username', 'bamboo')])
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkService', 'params': OrderedDict([('address', 'dummyIP'), ('username', 'bamboo')]), 'acquired': None, 'avail': True}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: add resource BOARD--TYPE3--1/USBSerialPort: USBSerialPort/OrderedDict([('match', OrderedDict([('ID_PATH', 'pci-0000:00:14.0-usb-0:2.4.4:1.0')]))])
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkSerialPort', 'params': OrderedDict(), 'acquired': None, 'avail': False}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: add resource BOARD--TYPE3--1/UniversalUpdateUtility: UniversalUpdateUtility/OrderedDict([('match', OrderedDict([('DEVPATH', '/devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3.1')]))])
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkUniversalUpdateUtility', 'params': OrderedDict(), 'acquired': None, 'avail': False}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: add resource BOARD--TYPE3--1/NetworkPowerPort: NetworkPowerPort/OrderedDict([('model', 'simplehttp'), ('host', 'dummyIP:8085/pduIP'), ('index', 5)])
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkPowerPort', 'params': OrderedDict([('model', 'simplehttp'), ('host', 'dummyIP:8085/pduIP'), ('index', 5)]), 'acquired': None, 'avail': True}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: add resource BOARD--TYPE3--1/NetworkService: NetworkService/OrderedDict([('address', 'dummyIP'), ('username', 'bamboo')])
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkService', 'params': OrderedDict([('address', 'dummyIP'), ('username', 'bamboo')]), 'acquired': None, 'avail': True}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkSerialPort', 'params': OrderedDict([('host', 'exporter-workstation.domain.net'), ('port', None), ('extra', {'path': '/dev/ttyUSB4', 'proxy_required': False, 'proxy': 'exporter-workstation'})]), 'acquired': None, 'avail': True}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkUniversalUpdateUtility', 'params': OrderedDict([('host', 'exporter-workstation.domain.net'), ('busnum', None), ('devnum', None), ('path', None), ('vendor_id', None), ('model_id', None), ('extra', {'proxy_required': False, 'proxy': 'exporter-workstation'})]), 'acquired': None, 'avail': False}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkSerialPort', 'params': OrderedDict([('host', 'exporter-workstation.domain.net'), ('port', None), ('extra', {'path': '/dev/ttyUSB8', 'proxy_required': False, 'proxy': 'exporter-workstation'})]), 'acquired': None, 'avail': True}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkUniversalUpdateUtility', 'params': OrderedDict([('host', 'exporter-workstation.domain.net'), ('busnum', None), ('devnum', None), ('path', None), ('vendor_id', None), ('model_id', None), ('extra', {'proxy_required': False, 'proxy': 'exporter-workstation'})]), 'acquired': None, 'avail': False}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkSerialPort', 'params': OrderedDict([('host', 'exporter-workstation.domain.net'), ('port', None), ('extra', {'path': '/dev/ttyUSB12', 'proxy_required': False, 'proxy': 'exporter-workstation'})]), 'acquired': None, 'avail': True}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: {'cls': 'NetworkUniversalUpdateUtility', 'params': OrderedDict([('host', 'exporter-workstation.domain.net'), ('busnum', None), ('devnum', None), ('path', None), ('vendor_id', None), ('model_id', None), ('extra', {'proxy_required': False, 'proxy': 'exporter-workstation'})]), 'acquired': None, 'avail': False}
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: missed checkpoint, exiting (last was 300.0586256887764 seconds ago)
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: connection lost
apr 08 11:03:59 exporter-workstation service_start_exporter.sh[29060]: 2020-04-08T11:03:59 session closed with reason wamp.close.transport_lost [WAMP transport was lost without closing the session 6485400199526595 before]
apr 08 11:04:00 exporter-workstation systemd[1]: exporter.service: Main process exited, code=exited, status=100/n/a
apr 08 11:04:00 exporter-workstation systemd[1]: exporter.service: Failed with result 'exit-code'.
apr 08 11:04:05 exporter-workstation systemd[1]: exporter.service: Service hold-off time over, scheduling restart.
apr 08 11:04:05 exporter-workstation systemd[1]: exporter.service: Scheduled restart job, restart counter is at 7.
apr 08 11:04:05 exporter-workstation systemd[1]: Stopped labgrid exporter service.

Emantor commented 4 years ago

This looks like your coordinator is not able to ping your exporters. Please check on the coordinator log side whether the pings are send out to the exporters.

jluebbe commented 4 years ago

A possible reason would be that your coordinator is older than your exporters.

egrigore commented 4 years ago

Coordinator was updated at the same time as exporters, I am trying to distinguish the coordinator log only for that exporter but I can't since the rest of the exporters are also in there.

It seems coordinator responds - there is no failure or timeout.

We will try to switch back to the previous version.

Emantor commented 4 years ago

We will try to switch back to the previous version.

Note that this won't resolve your problem long-term. Please try to extract a usable coordinator and exporter log or, if prohibited by corporate policies, try to build a minimal reproduction case.

egrigore commented 4 years ago

We have switched back only the exporters (coordinator still runs with latest version) to previous version and they seem stable for at least 3 hours.

I will try to create a minimal reproduction case. It might be a configuration that is not compatible with the latest changes?

Emantor commented 4 years ago

We have switched back only the exporters (coordinator still runs with latest version) to previous version and they seem stable for at least 3 hours.

I will try to create a minimal reproduction case. It might be a configuration that is not compatible with the latest changes?

The exporter configuration did not change between older versions. What changed is that the crossbar configuration file location was moved, see c8756f2. For production deployments you should copy the configuration file and adjust the configuration to your environment. To me it sounds more like a firewalling issue, but YMMV. Note that for the exporter ping changes, crossbar and autobahn need to be updated to 19.11.0.

egrigore commented 4 years ago

Hi @Emantor , sorry for the late update, I didn't had time to debug more on this , we are still on the backup branch for exporter. Thank you for your suggestions :

I have looked and we have:

Crossbar.io : 19.11.1 txaio : 20.4.1 Autobahn : 19.11.1 UTF8 Validator : autobahn XOR Masker : autobahn JSON Codec : stdlib MsgPack Codec : msgpack-0.6.2 CBOR Codec : cbor-1.0.0 UBJSON Codec : ubjson-0.15.0 FlatBuffers : flatbuffers-?.?.? Twisted : 20.3.0-EPollReactor LMDB : 0.98/lmdb-0.9.22 Python : 3.6.8/CPython Frozen executable : no Operating system : Linux-4.15.0-72-generic-x86_64-with-Ubuntu-18.04-bionic Host machine : x86_64

Why would it be from firewall rules? Indeed we have the coordinator on a VM with restrictive firewall rules - but it has the 20408, TCP traffic allowed , and it worked like this.

Anyway I will try to find some time and recreate the issue on a smaller env and maybe engage the IT to take a look if it is any traffic blocked between exporter and coordinator.

Emantor commented 3 years ago

What we have seen a lot is that the Exporter/Coordinator VM is I/O blocked for some reason, which leads to the Coordinator not being able to respond in time. In our case we investigated this using ioping and logging the output into a file. This showed that under certain circumstances, we had IO ping times exceeding half a minute. Unfortunately the Websocket connections are quite sensitive to this kind of latency.

egrigore commented 3 years ago

Hi,

We have rebased in January to the latest code , met the same problem, but we changed from "age > 300 " to "age > 400" in labgrid/remote/exporter.py and since then exporters are stable from this point of view (some of them are still failing rarely "missed checkpoint, exiting (last was 400.20326938666403 seconds ago)"). Can this affect anything else? Or is it ok to keep it like this ?

jluebbe commented 3 years ago

Did this happen only on single exporters at a time or multiple at once?

Could you check that your network connection between exporter and coordinator is stable? (This was the cause in our lab for some cases where this triggered).

egrigore commented 3 years ago

After I have increased the timeout to 400 it happened very rarely (5-6 times per exporter in 6 weeks- not all at once , and half of them are indeed motivated by some IT network downtime). When it was set to 300 it happened every 10 minutes for each exporter (no matter their geo location).

Now we see on some of them only below message (without missing checkpoint part):

Feb 15 18:02:27 exporter service_start_exporter.sh[25258]: 2021-02-15T18:02:26 dropping connection to peer tcp4:<coordinator-IP>:20408 with abort=True: WebSocket ping timeout (peer did not respond with pong in time)
Feb 15 18:02:27 exporter service_start_exporter.sh[25258]: connection lost
Feb 15 18:02:27 exporter service_start_exporter.sh[25258]: 2021-02-15T18:02:26 session closed with reason wamp.close.transport_lost [WAMP transport was lost without closing the session 6852899289096070 before]
Feb 15 18:02:27 exporter systemd[1]: exporter.service: Main process exited, code=exited, status=100/n/a
Feb 15 18:02:27 exporter systemd[1]: exporter.service: Failed with result 'exit-code'.

For this exporter I have activated the --debug option from labgrid-exporter to see if we can see more when above happens. And for all the exporters we get the time to respond to ping from coordinator before exporter starts to monitor if there are changes in this also.

If you have other suggestions in how to monitor the connection between coordinator and exporter let me know.

Bastian-Krause commented 1 year ago

Is this still happening on current master?

ElenaGrigore commented 1 year ago

Hi, I haven't seen this error since we upgraded to the latest labgrid version, we kept the age to 400 in exporter.py (maybe this is a value that can be configurable ? ). I think we can close this ticket for now .

labgrid-project / labgrid

Exporter stops with "missed checkpoint, exiting " #600