FTS Zero Touch core only install error after admin first admin login

[dtatum]

succeeded with a core only installation of latest FTS on clean ubuntu 20.04 server (minimal install, static IP, ESX VM).

web server connects and presents login page. login with default credentials returns with python error dump.

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 951, in send
    self.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 171, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5465066820>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='207.102.155.14', port=19023): Max retries exceeded with url: /AuthenticateUser?username=admin&password=password (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5465066820>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/eventlet/wsgi.py", line 573, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer-UI/app/base/routes.py", line 42, in login
    user = requests.get(f"http://{app.config['IP']}:{app.config['PORT']}/AuthenticateUser", params={"username": username, "password": password}, headers={"Authorization": f"{app.config['APIKEY']}"})
  File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='207.102.155.14', port=19023): Max retries exceeded with url: /AuthenticateUser?username=admin&password=password (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5465066820>: Failed to establish a new connection: [Errno 111] Connection refused'))

[SeawardDread]

I'm also experiencing this. Have tried all variations of external and internal IP in the configs I can think of.

In addition, using sudo python3 -m FreeTAKServer.controllers.services.FTS generates the following:

CA found locally, not generating a new one
Certificate file exists, aborting.
Certificate file exists, aborting.
something
start 213
starting now
running api
0.0.0.0
8080
(12036) wsgi starting up on http://0.0.0.0:8080
start 213
starting SSL now
0.0.0.0
8443
(12037) wsgi starting up on https://0.0.0.0:8443
event event about to be created
CoTService started
SSL CoTService started
{}
number of CoT messages received by services: 0
there has been an exception in the start function of TCPCoTService [Errno 98] Address already in use
ssl cot service starting
[Errno 98] Address already in use
there has been an exception thrown in the starting of the ssl service [Errno 98] Address already in use
started federation federate service

[brothercorvo]

which FTS version are you using?

[dtatum]

this was using the latest version at the time, [eae0b76].

same installation procedure as issue #56 except with --core

[ultrablazed]

Getting this issue too. I am using a fresh install of ubuntu server 20.04.4, no additional variables like --core. edit: i also used zerotouch install (wget -qO - bit.ly/ftszerotouch | sudo bash) at around 11 EST

[brothercorvo]

@ultrablazed the video server issue is a different one, please open a specific one to address it

[ultrablazed]

@ultrablazed the video server issue is a different one, please open a specific one to address it

apologies! im new to github. Moved my bug over to the correct thread

[numpad0]

I think the root cause is using public IP in config files. This leads to videoserver fail to start, causing Ansible to abort, causing Node-RED installation process to be skipped, etc., that results in a partial installation that can still display some web UIs. Is this a security consideration that daemons are not bound to 0.0.0.0?

[somefreaks]

Hi, I experienced the same issue. I have tried to install the zero-touch routine about 6 times. the login to the web interface always fails. Interestingly you can connect with ATAK clients to the server (we are using zerotier for for this). Basis is Ubuntu 20.04 (fresh install, no updates) on a virtual machine.

I`m using the default user/pw, which is stated (admin/password).

Andy idea what can be done?

Many thanks. Regards, Matt

[brothercorvo]

from our last tests the UI login works regularly if you are using ZTI, at the end of the jobs you should have a nice overview of all credentials. what are you using for the UI?

[somefreaks]

Hi Corvo,

thanks for taking care. Thats what I get and it brings the same mistake. Install is fresh from today.

[somefreaks]

that the error message:

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn conn = connection.create_connection( File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 84, in create_connection raise err File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 74, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/lib/python3.8/http/client.py", line 1256, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output self.send(msg) File "/usr/lib/python3.8/http/client.py", line 951, in send self.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 187, in connect conn = self._new_conn() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 171, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7effca2c0eb0>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen retries = retries.increment( File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='IP', port=19023): Max retries exceeded with url: /AuthenticateUser?username=admin&password=password (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7effca2c0eb0>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/eventlet/wsgi.py", line 573, in handle_one_response result = self.application(self.environ, start_response) File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2464, in call return self.wsgi_app(environ, start_response) File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2450, in wsgi_app response = self.handle_exception(e) File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1867, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise raise value File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise raise value File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1936, in dispatch_request return self.view_functionsrule.endpoint File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer-UI/app/base/routes.py", line 42, in login user = requests.get(f"http://{app.config['IP']}:{app.config['PORT']}/AuthenticateUser", params={"username": username, "password": password}, headers={"Authorization": f"{app.config['APIKEY']}"}) File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get return request('get', url, params=params, kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request resp = self.send(prep, send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send r = adapter.send(request, kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='IP', port=19023): Max retries exceeded with url: /AuthenticateUser?username=admin&password=password (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7effca2c0eb0>: Failed to establish a new connection: [Errno 111] Connection refused'))

[brothercorvo]

Your issue is that the ip are set wrong. So the ui cannot communicate with FTS. Whats is your network situation?

On Sun., Nov. 13, 2022, 7:41 p.m. Matt, @.***> wrote:

that the error message:

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn conn = connection.create_connection( File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 84, in create_connection raise err File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 74, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/lib/python3.8/http/client.py", line 1256, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output self.send(msg) File "/usr/lib/python3.8/http/client.py", line 951, in send self.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 187, in connect conn = self._new_conn() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 171, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7effca2c0eb0>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen retries = retries.increment( File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='IP', port=19023): Max retries exceeded with url: /AuthenticateUser?username=admin&password=password (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7effca2c0eb0>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/eventlet/wsgi.py", line 573, in handle_one_response result = self.application(self.environ, start_response) File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2464, in call return self.wsgi_app(environ, start_response) File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2450, in wsgi_app response = self.handle_exception(e) File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1867, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise raise value File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise raise value File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1936, in dispatch_request return self.view_functionsrule.endpoint http://**req.view_args File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer-UI/app/base/routes.py", line 42, in login user = requests.get(f"http://{app.config['IP']}:{app.config['PORT']}/AuthenticateUser", params={"username": username, "password": password}, headers={"Authorization": f"{app.config['APIKEY']}"}) File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get return request('get', url, params=params, kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request resp = self.send(prep, send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send r = adapter.send(request, kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='IP', port=19023): Max retries exceeded with url: /AuthenticateUser?username=admin&password=password (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7effca2c0eb0>: Failed to establish a new connection: [Errno 111] Connection refused'))

— Reply to this email directly, view it on GitHub https://github.com/FreeTAKTeam/FreeTAKHub-Installation/issues/57#issuecomment-1312854577, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOPIAPOSYZAHVKE447I2Z4LWIF4BRANCNFSM5VTFGT5A . You are receiving this because you commented.Message ID: @.***>

[somefreaks]

FTS is installed on a Synology NAS at a remote location (192.168.119.0/24). I`m connected via VPN into this network and part of the same IP range. I also tried to open the web interface (127.0.0.1) from the local machine with the same outcome.

For the later on operation, we will use zerotier VPN to connect the clients - this is actually working already.

[somefreaks]

@brothercorvo Would you have any idea how to fix this issue? Many thanks for your help

[brothercorvo]

127.0.0.1 means localhost and will not help you. If you are inside the same network you manually can set 192.168.119.0 as FTS IP in the UI configuration. If you need to access the Ui from a remote location you will need to create a rule in your router

[numpad0]

If that hadn't been fixed, this is a known issue that zerotouch installer aborting part-way yet reporting successful installation. I had notified the maintainer mid 2022, but he was understandably overwhelmed with the Rube Goldberg nature of the installer and the whole FreeTAKServer stack, and might not have been able to fix it.

The root cause of the problem I reported was that the installer tries to fill in configuration files with a global IP address obtained from an external website, and that that can be inappropriate. For technical reasons, IP addresses passed to server programs cannot be that of the NAT router's external interface(e.g. 123.45.678.90/0), but that of to the computer that actually run said programs(192.168.123.45/24). Inappropriate IP address in configuration files cause some daemons to refuse to start, and the zerotouch script aborts installation and skip over to a false "installation completed" messages when daemons fail to start.

This is not an issue with cloud instances with direct connections to the Internet, as IP addresses used for "local" communications by such cloud instances is the global IP address, which lead to the issue being overlooked by the previous maintainer, if I recall right.

I don't have an immediate workaround, sorry for that, maybe global IP can be temporarily added to eth0 and then the configurations can be manually fixed later, or installer can be modified to not try to start some servers, or use hardcoded IP address.

[Baldag]

Problem still exists. Running FTS from a VM should be the normal case as for most types of systems. It doesn't make it better that there are slightly different instructions scattered throughout all the documentation.

[HeroesLament]

@Baldag running on VMs is perfectly fine. The issue becomes the way certain programs are required to bind to specific IP addresses, and not simply to 0.0.0.0:9997, which listens on all IPs. That's just one example with the video server, but generally speaking the NAT/no-NAT install cases will require at least some modification of either the Zerotouch execution flags, which I have considered adding a --nat flag to the ZT bash script to handle this install case, but at the moment there does not exist to differentiate between whether it lives behind public IP NAT or not, or rather, the IP address clients will attempt to connect to is behind NAT.

If you could point out specific examples in the documentation you feel is not aligned well, I would be happy to help clarify them and update the docs with the clarified instruction.

[Baldag]

2 places I have spotted different install links that all point to easy_install.sh :

• https://freetakteam.github.io/FreeTAKServer-User-Docs/Installation/Ansible/ZeroTouchInstall/
  • uses bit.ly/freetakhub2 -https://github.com/FreeTAKTeam/FreeTAKHub-Installation/blob/main/README.md
  • uses bit.ly/ftszerotouch

Also the fact that easy_install and advancedinstall seems to be built idenpendantly from each other as some parts fail in advanced that work in easy. -Advanced seems to fail on the step "Delete PyYAML" because {{Python3_version}} does not contain a value.

[Baldag]

Also the linux manual install section seems to be very out of date: https://freetakteam.github.io/FreeTAKServer-User-Docs/Installation/Linux/1_Install/

The last step of configuring where you are supposed to set your websocketkey the FTS-config lacks a lot of the contents listen including the keys.

[Baldag]

https://freetakteam.github.io/FreeTAKServer-User-Docs/Installation/Linux/3_ManualConfiguration/

The MainConfig.py location is wrong

[phreed]

@brothercorvo I think this can be closed.

I will pick up the issue regarding the out-of-date linux documentation in that project.

[naman108]

If that hadn't been fixed, this is a known issue that zerotouch installer aborting part-way yet reporting successful installation. I had notified the maintainer mid 2022, but he was understandably overwhelmed with the Rube Goldberg nature of the installer and the whole FreeTAKServer stack, and might not have been able to fix it.

The root cause of the problem I reported was that the installer tries to fill in configuration files with a global IP address obtained from an external website, and that that can be inappropriate. For technical reasons, IP addresses passed to server programs cannot be that of the NAT router's external interface(e.g. 123.45.678.90/0), but that of to the computer that actually run said programs(192.168.123.45/24). Inappropriate IP address in configuration files cause some daemons to refuse to start, and the zerotouch script aborts installation and skip over to a false "installation completed" messages when daemons fail to start.

This is not an issue with cloud instances with direct connections to the Internet, as IP addresses used for "local" communications by such cloud instances is the global IP address, which lead to the issue being overlooked by the previous maintainer, if I recall right.

I don't have an immediate workaround, sorry for that, maybe global IP can be temporarily added to eth0 and then the configurations can be manually fixed later, or installer can be modified to not try to start some servers, or use hardcoded IP address.

As far as I understand we now use a different system for getting the system IP which is more reliable. Can someone confirm this?

[brothercorvo]

@phreed suggested that this is solved. However we still get the IP from the external service. There is no universal solution AFIK