FreeTAKServer on RPi 3 Error 98 Exceptions

[DaveyDave123]

Hi. I'm attempting to run FreeTAKServer on a RPi and experiencing issues with 'Address already in use' and exceptions. I'm using python 3.9 and what I believe to be an appropriate configured MainConfig.py. Installation as per: https://freetakteam.github.io/FreeTAKServer-User-Docs/Installation/Docker/Raspberry%20Pi/ with follow up execution guidance from here: https://www.youtube.com/watch?v=eMh9G96-npw

Any pointers would be greatly appreciated - thank you.

Dave

hh-takserver@raspberrypi:/usr/local/lib/python3.9/dist-packages/FreeTAKServer/controllers/configuration $ sudo python3 -m FreeTAKServer.controllers.services.FTS
would you like to use a yaml config file, 
 if yes you will be prompted for further configuration options [yes]: yes
where would you like to save the yaml config [/opt/FTSConfig.yaml]: y
enter ip [192.168.1.10]: 
192.168.1.10
192.168.1.10
enter the preferred database type (MySQL is highly experimental if you're not sure leave default) [SQLite]: 
enter the preferred database path [/opt/FTSDataBase.db]: 
/opt/FTSDataBase.db
enter the preferred main_path [/usr/local/lib/python3.9/dist-packages/FreeTAKServer]: 
/usr/local/lib/python3.9/dist-packages/FreeTAKServer
enter the preferred log file path [/usr/local/lib/python3.9/dist-packages/FreeTAKServer/Logs]: 
/usr/local/lib/python3.9/dist-packages/FreeTAKServer/Logs
CA found locally, not generating a new one
Certificate file exists, aborting.
Certificate file exists, aborting.
something
start 213
starting now
Exception ignored in: <function _after_fork at 0x72497028>
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 1490, in _after_fork
    thread._reset_internal_locks(True)
  File "/usr/lib/python3.9/threading.py", line 827, in _reset_internal_locks
Exception ignored in: <function _after_fork at 0x72497028>
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 1490, in _after_fork
    self._started._at_fork_reinit()
  File "/usr/lib/python3.9/threading.py", line 527, in _at_fork_reinit
    self._cond._at_fork_reinit()
  File "/usr/lib/python3.9/threading.py", line 253, in _at_fork_reinit
    self._lock._at_fork_reinit()
AttributeError: 'Semaphore' object has no attribute '_at_fork_reinit'
    thread._reset_internal_locks(True)
  File "/usr/lib/python3.9/threading.py", line 827, in _reset_internal_locks
running api
    self._started._at_fork_reinit()
  File "/usr/lib/python3.9/threading.py", line 527, in _at_fork_reinit
    self._cond._at_fork_reinit()
  File "/usr/lib/python3.9/threading.py", line 253, in _at_fork_reinit
    self._lock._at_fork_reinit()
AttributeError: 'Semaphore' object has no attribute '_at_fork_reinit'
0.0.0.0
8080
there has been an exception in Data Package service startup [Errno 98] Address already in use
start 213
starting SSL now
Exception ignored in: <function _after_fork at 0x72497028>
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 1490, in _after_fork
    thread._reset_internal_locks(True)
  File "/usr/lib/python3.9/threading.py", line 827, in _reset_internal_locks
    self._started._at_fork_reinit()
  File "/usr/lib/python3.9/threading.py", line 527, in _at_fork_reinit
    self._cond._at_fork_reinit()
  File "/usr/lib/python3.9/threading.py", line 253, in _at_fork_reinit
    self._lock._at_fork_reinit()
AttributeError: 'Semaphore' object has no attribute '_at_fork_reinit'
0.0.0.0
8443
(1917) wsgi starting up on https://0.0.0.0:8443
event event about to be created
CoTService started
Exception ignored in: <function _after_fork at 0x72497028>
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 1490, in _after_fork
    thread._reset_internal_locks(True)
  File "/usr/lib/python3.9/threading.py", line 827, in _reset_internal_locks
    self._started._at_fork_reinit()
  File "/usr/lib/python3.9/threading.py", line 527, in _at_fork_reinit
    self._cond._at_fork_reinit()
  File "/usr/lib/python3.9/threading.py", line 253, in _at_fork_reinit
    self._lock._at_fork_reinit()
AttributeError: 'Semaphore' object has no attribute '_at_fork_reinit'
SSL CoTService started
Exception ignored in: <function _after_fork at 0x72497028>
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 1490, in _after_fork
    thread._reset_internal_locks(True)
  File "/usr/lib/python3.9/threading.py", line 827, in _reset_internal_locks
    self._started._at_fork_reinit()
  File "/usr/lib/python3.9/threading.py", line 527, in _at_fork_reinit
    self._cond._at_fork_reinit()
  File "/usr/lib/python3.9/threading.py", line 253, in _at_fork_reinit
Exception ignored in: <function _after_fork at 0x72497028>
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 1490, in _after_fork
    thread._reset_internal_locks(True)
  File "/usr/lib/python3.9/threading.py", line 827, in _reset_internal_locks
    self._started._at_fork_reinit()
  File "/usr/lib/python3.9/threading.py", line 527, in _at_fork_reinit
    self._cond._at_fork_reinit()
  File "/usr/lib/python3.9/threading.py", line 253, in _at_fork_reinit
    self._lock._at_fork_reinit()
AttributeError: 'Semaphore' object has no attribute '_at_fork_reinit'
{}
    self._lock._at_fork_reinit()
AttributeError: 'Semaphore' object has no attribute '_at_fork_reinit'
number of CoT messages received by services: 0
there has been an exception in the start function of TCPCoTService [Errno 98] Address already in use
started federation federate service
ssl cot service starting
server started

 running ssl 

mainRunFunction is running
client dict: {}
{}
number of CoT messages received by services: 0
{}
number of CoT messages received by services: 0
time since last reset 0:00:30.014496
{}
number of CoT messages received by services: 0
mainRunFunction is running
client dict: {}

[hickey]

The error "address already in use" is caused by the TCP (or UDP) port already having a process listening for connections on the port. In simple terms there is already a process registered to receive any packets sent to that port.

It appears to me that port 8443 is already in use. Do you have another service that is running on this port? 8443 is usually used as an alternative/development port number for a SSL enabled webserver. Many things run on this port like dashboards and various open source projects web services run here. Even the Ubiquiti controller software uses 8443 as the port to connect to. So you may need to do a bit of digging to find what is already running.

If you have lsof installed, you can run as root lsof -i to see all the network ports that are being listened on and you should find a reference to the process that is listening on the port. You should be able to get the same information with netstat -ntpl, again run as root.

So then to fix this problem you need to move the service that is running on 8443 to another port or change the configuration of FTS so that the SSL DataPackage port is something other than 8443.

I was just checking the configuration code for the YAML config entry that would control this port and I don't see anything. Not certain at this moment if that is an oversight or if the port is hard coded since the TAK clients probably can not be changed to pull their data packages from another port. More investigation will be necessary to unwind this.

[DaveyDave123]

Thanks for your reply. I’ve used lsof –i and it appears that there’s some form of docker process docker-pr on that port. I have no idea what that is though. I can kill (kill -9) those processes off and re run FreeTAKServer. However, whilst it no longer references the address issues, the exceptions are still present ie:

Exception ignored in: <function _after_fork at 0x72497028> Traceback (most recent call last): File "/usr/lib/python3.9/threading.py", line 1490, in _after_fork self._started._at_fork_reinit() File "/usr/lib/python3.9/threading.py", line 527, in _at_fork_reinit self._cond._at_fork_reinit() File "/usr/lib/python3.9/threading.py", line 253, in _at_fork_reinit self._lock._at_fork_reinit() AttributeError: 'Semaphore' object has no attribute '_at_fork_reinit' thread._reset_internal_locks(True) File "/usr/lib/python3.9/threading.py", line 827, in _reset_internal_locks running api self._started._at_fork_reinit() File "/usr/lib/python3.9/threading.py", line 527, in _at_fork_reinit self._cond._at_fork_reinit() File "/usr/lib/python3.9/threading.py", line 253, in _at_fork_reinit self._lock._at_fork_reinit() AttributeError: 'Semaphore' object has no attribute '_at_fork_reinit'

Etc…….

So, to me, there’s still something wrong. Are there particular versions of FreeTAKServer that are known to work on the RPi? Interestingly, the server basically works. However, the GUI is only partially responsive, with it becoming unresponsive after a couple of pages, both on the RPi and through using an external client – this leads me to believe there are some issues.

Any ideas?

Thanks for any pointers, Dave

From: Gerard Hickey @.> Sent: Tuesday, November 15, 2022 7:00 PM To: FreeTAKTeam/FreeTakServer @.> Cc: Moore, Dave M (UK Rochester) @.>; Author @.> Subject: Re: [FreeTAKTeam/FreeTakServer] FreeTAKServer on RPi 3 Error 98 Exceptions (Issue #401)

PHISHING ALERT This email has been sent from an account outside of the BAE Systems network.

Please treat the email with caution, especially if you are requested to click on a link or open an attachment. For further information on how to spot and report a phishing email please access the Global Intranet then select / . If you think this is a phishing email, please report it by using the "Report Phishing" button in Outlook.

The error "address already in use" is caused by the TCP (or UDP) port already having a process listening for connections on the port. In simple terms there is already a process registered to receive any packets sent to that port.

It appears to me that port 8443 is already in use. Do you have another service that is running on this port? 8443 is usually used as an alternative/development port number for a SSL enabled webserver. Many things run on this port like dashboards and various open source projects web services run here. Even the Ubiquiti controller software uses 8443 as the port to connect to. So you may need to do a bit of digging to find what is already running.

If you have lsof installed, you can run as root lsof -i to see all the network ports that are being listened on and you should find a reference to the process that is listening on the port. You should be able to get the same information with netstat -ntpl, again run as root.

So then to fix this problem you need to move the service that is running on 8443 to another port or change the configuration of FTS so that the SSL DataPackage port is something other than 8443.

I was just checking the configuration code for the YAML config entry that would control this port and I don't see anything. Not certain at this moment if that is an oversight or if the port is hard coded since the TAK clients probably can not be changed to pull their data packages from another port. More investigation will be necessary to unwind this.

— Reply to this email directly, view it on GitHubhttps://github.com/FreeTAKTeam/FreeTakServer/issues/401#issuecomment-1315736306, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AE32MBYB5STVHMJAR5UQ4T3WIPMOPANCNFSM6AAAAAASBHRRZQ. You are receiving this because you authored the thread.Message ID: @.***>

---

This email and any attachments are confidential to the intended recipient and may also be privileged. If you are not the intended recipient please delete it from your system and notify the sender. You should not copy it or use it for any purpose nor disclose or distribute its contents to any other person.

---

BAE Systems may process information about you that may be subject to data protection laws. For more information about how we use your personal information, how we protect your information, our legal basis for using your information, your rights and who you can contact, please refer to our Privacy Notice at www.baesystems.com/en/privacy

[hickey]

docker-pr is actually docker-proxy--it is just getting truncated in the output. This is the part of docker that does some of the port redirection. So somewhere you have a Docker container running that is getting port 8443 redirected to a container port. Looking at the output of docker ps should give you a good idea which container it is.

As to the threading error that you are getting, I can not give you as much of a definitive answer. I just checked the FTS and UI code bases and there are no references I can find for Semaphore or the method after_fork. The other curious thing is that the exception that is occurring is in the Python standard library not any of the FTS code. Do you have any Python modules that modify how forking or threading operates?

[naman108]

My first thought is that given your version (3.9) it's possible that one the libraries FTS is dependent on is incompatible. We don't do much with semaphores as pointed out above, so my first troubleshooting recommendation would be to try do downgrade to the currently supported version 3.8 if at all possible otherwise we can escalate by selectively upgrading potentially culpable libraries.

[hickey]

@DaveyDave123 I was looking at the server output again and realized that I mis-read some of the output. I thought that I was reading that FTS was failing to open port 8443. What the output is actually saying is that 8443 was opened correctly, but it is the TCP COT port that is failing. By default that port is 8087. Sorry for sending you down the wrong path.