1.9.1: SSL connection breaks after short time

[sgofferj]

Not sure if that's a bug or if I'm missing anything... I can't seem to get SSL to work. I have built the docker image from the current FreeTakServer-Docker main branch on an x86_64 server and it's running fine. tcp connectivity is fine too. I'm using an Android 8.1.0 phone and an Android 10 tablet for testing. Both have ATAK 4.3.1 installed. I created users in the WebUI and tried to deploy the datapackages. That was the first problem - the certificates don't seem to get installed in ATAK. At least they don't show up in /atak/certs. Additionally, the data package prefs refer to the internal docker network IP of the server, not to the external IP (which is set correctly throught the IP envvar) (see https://github.com/FreeTAKTeam/FreeTAKServer-Docker/issues/26). I pulled the certificates out of the ZIP files and uploaded them to the EUDs manually. Then I imported the certificates on the EUDs manually through the server settings. Which seems to work - at least I don't get an error. The EUDs try to connect and I immediately get "IOerror - reconnecting". When I additionally enter authentication credentials, the EUDs connect, apparently stay connected for like 20-30secs and then disconnect. Same error message on the EUDs. On the server I found the following logs...:

FTS_debug.log (repeating over and over)

INFO : 2021-07-31 17:35:18,456 : DataPackageServer.py:144 : sending client version json
INFO : 2021-07-31 17:35:18,457 : DataPackageServer.py:150 : sending client version info

FTS_error.log (repeating over and over)

ERROR : 2021-07-31 16:22:04,519 : ClientReceptionHandler.py:72 : Exception other than broken pipe in monitor for data function [Errno 11] Resource temporarily unavailable

Another thing I noticed are updated timestamps on the server certificates in the persistent storage after a container restart. Does the container recreate the server certificates after a restart or recreate without checking for exixting certificates?

[sgofferj]

Tried now with ATAK CIV 4.4.0RC from p!ay store. Same behavior, just instead of IOError, ATAK complains Reception timeout. Looked into the interface on the server with tcpdump and traffic is going between the tablet and the server port 8089.

[sgofferj]

Found the supervisor logs... FTS_stdout.log of one connect/disconnect cycle:

['  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main\n    return _run_code(code, main_globals, None,\n', '  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code\n    exec(code, run_globals)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/FTS.py", line 734, in <module>\n    FTS().startup(args.CoTPort, args.CoTIP, args.DataPackagePort, args.DataPackageIP, args.SSLDataPackagePort, args.SSLDataPackageIP, args.RestAPIPort, args.RestAPIIP, args.SSLCoTPort, args.SSLCoTIP, args.AutoStart, True, args.UI)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/FTS.py", line 601, in startup\n    self.start_all(StartupObject)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/FTS.py", line 350, in start_all\n    self.start_SSL_CoT_service(FTSServiceStartupConfigObject)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/FTS.py", line 197, in start_SSL_CoT_service\n    self.SSLCoTService.start()\n', '  File "/usr/lib/python3.8/multiprocessing/process.py", line 121, in start\n    self._popen = self._Popen(self)\n', '  File "/usr/lib/python3.8/multiprocessing/context.py", line 224, in _Popen\n    return _default_context.get_context().Process._Popen(process_obj)\n', '  File "/usr/lib/python3.8/multiprocessing/context.py", line 277, in _Popen\n    return Popen(process_obj)\n', '  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__\n    self._launch(process_obj)\n', '  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75, in _launch\n    code = process_obj._bootstrap(parent_sentinel=child_r)\n', '  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap\n    self.run()\n', '  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run\n    self._target(*self._args, **self._kwargs)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/SSLCoTServiceController.py", line 30, in start\n    self.mainRunFunction(clientData, receiveConnection, sock, pool, Event, clientDataPipe,\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/Orchestrator.py", line 436, in mainRunFunction\n    CoTOutput = self.handel_regular_data(clientDataOutput)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/Orchestrator.py", line 492, in handel_regular_data\n    CoTOutput = self.monitorRawCoT(clientDataOutputSingle)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/Orchestrator.py", line 362, in monitorRawCoT\n    output = function(CoT[1])\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/Orchestrator.py", line 276, in clientDisconnected\n    print(str(traceback.format_stack()))\n']
disconnecting client
stage 1 b
stage 1 c
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0bd670>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902314f0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f10b610>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0fe4f0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9a5c40>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9a5c10>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9a5c40>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0b8970>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f09a8b0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0fe4f0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f09a8b0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0bd670>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0bd5e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902314f0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0b77c0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0b8be0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9a5c40>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0fe4f0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f10b610>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0fe4f0>
handling conn data
adding client
client added
<FreeTAKServer.model.FTSModel.Chatgrp.Chatgrp object at 0x7f1a8f10f880>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f10b610>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0b73d0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9af700>

 disconnect C [Errno 11] Resource temporarily unavailable

handling reg data
['  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main\n    return _run_code(code, main_globals, None,\n', '  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code\n    exec(code, run_globals)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/FTS.py", line 734, in <module>\n    FTS().startup(args.CoTPort, args.CoTIP, args.DataPackagePort, args.DataPackageIP, args.SSLDataPackagePort, args.SSLDataPackageIP, args.RestAPIPort, args.RestAPIIP, args.SSLCoTPort, args.SSLCoTIP, args.AutoStart, True, args.UI)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/FTS.py", line 601, in startup\n    self.start_all(StartupObject)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/FTS.py", line 350, in start_all\n    self.start_SSL_CoT_service(FTSServiceStartupConfigObject)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/FTS.py", line 197, in start_SSL_CoT_service\n    self.SSLCoTService.start()\n', '  File "/usr/lib/python3.8/multiprocessing/process.py", line 121, in start\n    self._popen = self._Popen(self)\n', '  File "/usr/lib/python3.8/multiprocessing/context.py", line 224, in _Popen\n    return _default_context.get_context().Process._Popen(process_obj)\n', '  File "/usr/lib/python3.8/multiprocessing/context.py", line 277, in _Popen\n    return Popen(process_obj)\n', '  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__\n    self._launch(process_obj)\n', '  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75, in _launch\n    code = process_obj._bootstrap(parent_sentinel=child_r)\n', '  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap\n    self.run()\n', '  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run\n    self._target(*self._args, **self._kwargs)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/SSLCoTServiceController.py", line 30, in start\n    self.mainRunFunction(clientData, receiveConnection, sock, pool, Event, clientDataPipe,\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/Orchestrator.py", line 436, in mainRunFunction\n    CoTOutput = self.handel_regular_data(clientDataOutput)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/Orchestrator.py", line 492, in handel_regular_data\n    CoTOutput = self.monitorRawCoT(clientDataOutputSingle)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/Orchestrator.py", line 362, in monitorRawCoT\n    output = function(CoT[1])\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/Orchestrator.py", line 276, in clientDisconnected\n    print(str(traceback.format_stack()))\n']
disconnecting client
stage 1 b
stage 1 c
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0b66d0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9af700>

[sgofferj]

More stdout log: Deactivated server in ATAK, started logcapture, activated server, waited for connect-disconnect, stopped logcapture.

['  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main\n    return _run_code(code, main_globals, None,\n', '  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code\n    exec(code, run_globals)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/FTS.py", line 734, in <module>\n    FTS().startup(args.CoTPort, args.CoTIP, args.DataPackagePort, args.DataPackageIP, args.SSLDataPackagePort, args.SSLDataPackageIP, args.RestAPIPort, args.RestAPIIP, args.SSLCoTPort, args.SSLCoTIP, args.AutoStart, True, args.UI)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/FTS.py", line 601, in startup\n    self.start_all(StartupObject)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/FTS.py", line 350, in start_all\n    self.start_SSL_CoT_service(FTSServiceStartupConfigObject)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/FTS.py", line 197, in start_SSL_CoT_service\n    self.SSLCoTService.start()\n', '  File "/usr/lib/python3.8/multiprocessing/process.py", line 121, in start\n    self._popen = self._Popen(self)\n', '  File "/usr/lib/python3.8/multiprocessing/context.py", line 224, in _Popen\n    return _default_context.get_context().Process._Popen(process_obj)\n', '  File "/usr/lib/python3.8/multiprocessing/context.py", line 277, in _Popen\n    return Popen(process_obj)\n', '  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__\n    self._launch(process_obj)\n', '  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75, in _launch\n    code = process_obj._bootstrap(parent_sentinel=child_r)\n', '  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap\n    self.run()\n', '  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run\n    self._target(*self._args, **self._kwargs)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/SSLCoTServiceController.py", line 30, in start\n    self.mainRunFunction(clientData, receiveConnection, sock, pool, Event, clientDataPipe,\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/Orchestrator.py", line 436, in mainRunFunction\n    CoTOutput = self.handel_regular_data(clientDataOutput)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/Orchestrator.py", line 492, in handel_regular_data\n    CoTOutput = self.monitorRawCoT(clientDataOutputSingle)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/Orchestrator.py", line 362, in monitorRawCoT\n    output = function(CoT[1])\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/Orchestrator.py", line 276, in clientDisconnected\n    print(str(traceback.format_stack()))\n']
disconnecting client
stage 1 b
stage 1 c
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0b52e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0b66d0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f10b610>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f1571f0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a90da82b0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f1571f0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a90da82b0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0b52e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0a00a0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0d2310>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f1571f0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0cc2b0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f1571f0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0cc2b0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f1571f0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0cc2b0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f1571f0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0cc2b0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0d2c70>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f09e280>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a90da82b0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0d4430>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0d3520>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0b66d0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0b52e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f09e070>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0b52e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0a00a0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0d21c0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0a00a0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a90da82b0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0d2520>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0b52e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f10b610>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0b52e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0b66d0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0d2c70>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a90da82b0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0d2310>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0b52e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0a00a0>
handling conn data
handling conn data
adding client
client added
<FreeTAKServer.model.FTSModel.Chatgrp.Chatgrp object at 0x7f1a8f0d2f10>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f09e2e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0c9160>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0d4400>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9a5c10>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0c9910>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0d4eb0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f0c9160>

 disconnect C [Errno 11] Resource temporarily unavailable

handling reg data
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d98fe20>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902615e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902ba190>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c905040>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9add00>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902ba190>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c905c70>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9add30>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9b46a0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9add00>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c8ed4c0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d98f760>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c960dc0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902615e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d98fe50>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9add00>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c960f10>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c923fa0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c960dc0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902ba190>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d98ff70>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9b46a0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d98f760>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c8eddf0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c905c70>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902ba190>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d98f760>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c8ed400>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d98ff70>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9add00>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902615e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c8ed400>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9add00>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c923af0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9add30>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9b46a0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d98fe50>
handling reg data
handling reg data
handling conn data
handling reg data
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d9a6220>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c9230a0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9b46a0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c8eddf0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902615e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c905c70>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c8edfd0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9e8040>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902d0b20>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9e81f0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d9a6220>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d98fe20>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c905040>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902b05e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c960f10>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d9a6220>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9e8580>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902615e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9e8040>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9e81f0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c905040>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9e8580>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902ba190>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d98ff70>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902b05e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9b46a0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9e8580>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c905c70>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d9a6220>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902b05e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c905040>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902ba190>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d981070>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d98ff70>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9b46a0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d981d60>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c960f10>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d9a6220>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d981070>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d98fe50>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9b46a0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d981d60>
handling conn data
handling reg data
handling reg data
handling reg data
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9b46a0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902b05e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c8ed4c0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c960f10>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9e8580>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9b46a0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902311c0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c923940>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8d981070>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9add00>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8f9b46a0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c8ed4c0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902b05e0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c8edd90>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c8ed400>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a902ba190>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f1a8c8ed4c0>

[sgofferj]

Something new for a change... Debug log:

INFO : 2021-08-05 08:24:19,075 : ReceiveConnections.py:49 : client connected over ssl ('192.168.10.76', 55290) 1628151859.075033
INFO : 2021-08-05 08:24:19,133 : DataPackageServer.py:144 : sending client version json
INFO : 2021-08-05 08:24:19,135 : DataPackageServer.py:150 : sending client version info
INFO : 2021-08-05 08:24:19,376 : ReceiveConnections.py:59 : client connected
INFO : 2021-08-05 08:24:19,377 : Orchestrator.py:141 : client has connected
INFO : 2021-08-05 08:24:19,377 : Orchestrator.py:141 : client has connected
INFO : 2021-08-05 08:24:19,377 : Orchestrator.py:141 : client has connected
INFO : 2021-08-05 08:24:19,377 : Orchestrator.py:141 : client has connected
INFO : 2021-08-05 08:24:19,377 : Orchestrator.py:141 : client has connected
DEBUG : 2021-08-05 08:24:19,446 : Orchestrator.py:551 : connection data from client WOLFMAN successfully processed
DEBUG : 2021-08-05 08:24:19,446 : Orchestrator.py:551 : connection data from client WOLFMAN successfully processed
DEBUG : 2021-08-05 08:24:19,446 : Orchestrator.py:551 : connection data from client WOLFMAN successfully processed
DEBUG : 2021-08-05 08:24:19,446 : Orchestrator.py:551 : connection data from client WOLFMAN successfully processed
DEBUG : 2021-08-05 08:24:19,446 : Orchestrator.py:551 : connection data from client WOLFMAN successfully processed
ERROR : 2021-08-05 08:24:19,482 : ClientReceptionHandler.py:72 : Exception other than broken pipe in monitor for data function [Errno 11] Resource temporarily unavailable
INFO : 2021-08-05 08:24:19,551 : Orchestrator.py:346 : initiating client disconnection
INFO : 2021-08-05 08:24:19,551 : Orchestrator.py:346 : initiating client disconnection
INFO : 2021-08-05 08:24:19,551 : Orchestrator.py:346 : initiating client disconnection
INFO : 2021-08-05 08:24:19,551 : Orchestrator.py:346 : initiating client disconnection
INFO : 2021-08-05 08:24:19,551 : Orchestrator.py:346 : initiating client disconnection
INFO : 2021-08-05 08:24:19,553 : Orchestrator.py:355 : client disconnection has concluded WOLFMAN
INFO : 2021-08-05 08:24:19,553 : Orchestrator.py:355 : client disconnection has concluded WOLFMAN
INFO : 2021-08-05 08:24:19,553 : Orchestrator.py:355 : client disconnection has concluded WOLFMAN
INFO : 2021-08-05 08:24:19,553 : Orchestrator.py:355 : client disconnection has concluded WOLFMAN
INFO : 2021-08-05 08:24:19,553 : Orchestrator.py:355 : client disconnection has concluded WOLFMAN

Error log still:

ERROR : 2021-08-05 08:27:00,539 : ClientReceptionHandler.py:72 : Exception other than broken pipe in monitor for data function [Errno 11] Resource temporarily unavailable

And stdout log:

handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486caca0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486caca0>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486e1bb0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486e1bb0>
handling data shared with ssl
<FreeTAKServer.model.SpecificCoT.SendDisconnect.SendDisconnect object at 0x7f39486e1cd0>
handling data shared with ssl
<FreeTAKServer.model.SpecificCoT.SendTakPong.SendTakPong object at 0x7f39486c7730>
handling conn data
adding client
client added
<FreeTAKServer.model.FTSModel.Chatgrp.Chatgrp object at 0x7f39486c2310>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486c7130>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486c7130>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486cccd0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486cccd0>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486caf10>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486caf10>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486ccee0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486ccee0>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486c7eb0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486c7eb0>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486c7730>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486c7730>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486d7d30>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486d7d30>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f394871b190>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f394871b190>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486c7130>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486c7130>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486cccd0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486cccd0>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486caca0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486caca0>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486e1d00>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486e1d00>
handling reg data
data is empty
['  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main\n    return _run_code(code, main_globals, None,\n', '  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code\n    exec(code, run_globals)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/FTS.py", line 734, in <module>\n    FTS().startup(args.CoTPort, args.CoTIP, args.DataPackagePort, args.DataPackageIP, args.SSLDataPackagePort, args.SSLDataPackageIP, args.RestAPIPort, args.RestAPIIP, args.SSLCoTPort, args.SSLCoTIP, args.AutoStart, True, args.UI)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/FTS.py", line 601, in startup\n    self.start_all(StartupObject)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/FTS.py", line 350, in start_all\n    self.start_SSL_CoT_service(FTSServiceStartupConfigObject)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/FTS.py", line 197, in start_SSL_CoT_service\n    self.SSLCoTService.start()\n', '  File "/usr/lib/python3.8/multiprocessing/process.py", line 121, in start\n    self._popen = self._Popen(self)\n', '  File "/usr/lib/python3.8/multiprocessing/context.py", line 224, in _Popen\n    return _default_context.get_context().Process._Popen(process_obj)\n', '  File "/usr/lib/python3.8/multiprocessing/context.py", line 277, in _Popen\n    return Popen(process_obj)\n', '  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__\n    self._launch(process_obj)\n', '  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75, in _launch\n    code = process_obj._bootstrap(parent_sentinel=child_r)\n', '  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap\n    self.run()\n', '  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run\n    self._target(*self._args, **self._kwargs)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/SSLCoTServiceController.py", line 30, in start\n    self.mainRunFunction(clientData, receiveConnection, sock, pool, Event, clientDataPipe,\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/Orchestrator.py", line 444, in mainRunFunction\n    CoTOutput = self.handel_regular_data(clientDataOutput)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/Orchestrator.py", line 502, in handel_regular_data\n    CoTOutput = self.monitorRawCoT(clientDataOutputSingle)\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/Orchestrator.py", line 372, in monitorRawCoT\n    output = function(CoT[1])\n', '  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/Orchestrator.py", line 276, in clientDisconnected\n    print(str(traceback.format_stack()))\n']
disconnecting client
stage 1
stage 1 a
stage 1 b
stage 1 c
stage 2
stage 3
client disconnected
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486ccee0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486ccee0>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486d7d30>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486d7d30>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39490cf6d0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39490cf6d0>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486d7d30>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486d7d30>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39490cf6d0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39490cf6d0>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486caca0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486caca0>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486cda90>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486cda90>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39490cf6d0>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39490cf6d0>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486caf10>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486caf10>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486c7130>
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39486c7130>
handling data shared with ssl
<FreeTAKServer.model.ClientInformation.ClientInformation object at 0x7f39490cf6d0>

[Titou74]

Same problem for me, rollback to 1.9 solved the problem

[sgofferj]

I tried rollback to 1.9 but I'm getting the same issue. I'm looking at the moment into temporary deploying taky until this is fixed...

[Titou74]

Correct: it happens less often, but I still have the issue .... trying another rollback to 1.8.1

[Titou74]

Connection break still happen on 1.8.1 Rollbacking to 1.5.12 which work fine before I updated to 1.9.1

[naman108]

have you had this issue with 1.9.1.5 which can be found on pypi?

[Titou74]

have you had this issue with 1.9.1.5 which can be found on pypi?

Yes: I just retry with 1.9.1.5 and I have the issue

[naman108]

alright thank you,

1. this is running on ubuntu correct?
2. have you tried using the installer?
3. can you send the error logs?
4. can you send your yaml config located at /opt/FTSConfig.yaml?

[Titou74]

1. this is running on ubuntu correct?

Debian 10, python 3.8

2. have you tried using the installer?

I follow the Linux Pypi installation guide

3. can you send the error logs?

FTS_error.log

4. can you send your yaml config located at /opt/FTSConfig.yaml?

FTSConfig.yaml.txt

I remove IP and changed the extension to upload it to github (don't accept .yaml)

[naman108]

ok thank you and in between disconnects are you able to transfer data between clients?

[Titou74]

ok thank you and in between disconnects are you able to transfer data between clients?

Yes, but not for a long time: when I start ATAK, it connect correctly to the server and I can send and receive datas. But after a few seconds, the connection break with an "IO error". After that, the ATAK loop on trying to connect, connection success (having the welcome message with green circle on server icon), but the connection immediatly break with the "IO error" message.

[Titou74]

I don't know if the problem that I detail is linked to the first post of this issue, but it's look like the same

[naman108]

alright could you try to run it on an ubuntu VM to see if the behaviour persists, also have you checked that you dont have an email, xmpp or phone number attached?

[naman108]

was able to reproduce this issue directly however at this time we're not certain of the cause but this is being worked on.

[Titou74]

was able to reproduce this issue directly however at this time we're not certain of the cause but this is being worked on.

Ok great, let us know. It correctly work on v1.5.12, maybe a dependy update or a bug introduced since this version.

alright could you try to run it on an ubuntu VM to see if the behaviour persists, also have you checked that you dont have an email, xmpp or phone number attached?

I'll try, but my current FTS work on a VPS with other things, so I have to build another VPS to try on Ubuntu

[naman108]

was able to reproduce this issue directly however at this time we're not certain of the cause but this is being worked on.

Ok great, let us know. It correctly work on v1.5.12, maybe a dependy update or a bug introduced since this version.

ya it may have something to do with the updated Certificate system to support CRL's

alright could you try to run it on an ubuntu VM to see if the behaviour persists, also have you checked that you dont have an email, xmpp or phone number attached?

I'll try, but my current FTS work on a VPS with other things, so I have to build another VPS to try on Ubuntu alright but no rush as given we were able to reproduce it's unlikely to be the issue

[marktimmerman-dev]

I have the same problem. I build the docker image from git

git pull https://github.com/FreeTAKTeam/FreeTAKServer-Docker.git docker build -t freetakserver .

So that uses Ubuntu 20.04

Is there any version I can go back to, where the problem is not there? You mentioned updated cert systemen. When was that introduced?

Let me know if I can help to debug this.

[Titou74]

Is there any version I can go back to, where the problem is not there? You mentioned updated cert systemen. When was that introduced?

As I said on a previous message, it correctly work for me on 1.5.12

[oliviasculley]

This may be related, I get the errors below intermittently every 40 seconds, resulting in the connection disconnecting with the error "Data reception timeout" on ATAK.

INFO : 2021-09-19 03:51:39,206 : ReceiveConnections.py:49 : client connected over ssl ('IP ADDRESS', something) redacted_id
INFO : 2021-09-19 03:51:39,269 : ReceiveConnections.py:59 : client connected
INFO : 2021-09-19 03:51:39,269 : Orchestrator.py:141 : client has connected
INFO : 2021-09-19 03:51:39,269 : Orchestrator.py:141 : client has connected
INFO : 2021-09-19 03:51:39,269 : Orchestrator.py:141 : client has connected
INFO : 2021-09-19 03:51:39,269 : Orchestrator.py:141 : client has connected
INFO : 2021-09-19 03:51:39,269 : Orchestrator.py:141 : client has connected
WARNING : 2021-09-19 03:51:39,270 : Orchestrator.py:175 : there has been an error in a clients connection'int' object has no attribute 'socket'
WARNING : 2021-09-19 03:51:39,270 : Orchestrator.py:175 : there has been an error in a clients connection'int' object has no attribute 'socket'
WARNING : 2021-09-19 03:51:39,270 : Orchestrator.py:175 : there has been an error in a clients connection'int' object has no attribute 'socket'
WARNING : 2021-09-19 03:51:39,270 : Orchestrator.py:175 : there has been an error in a clients connection'int' object has no attribute 'socket'
WARNING : 2021-09-19 03:51:39,270 : Orchestrator.py:175 : there has been an error in a clients connection'int' object has no attribute 'socket'
INFO : 2021-09-19 03:51:39,306 : DataPackageServer.py:150 : sending client version info
INFO : 2021-09-19 03:51:39,307 : DataPackageServer.py:144 : sending client version json

[brothercorvo]

tested. it works in 1.9.5 closing this issue