Network port configuration not working?

juju4 commented 4 years ago

Changing port in /etc/kolide/kolide.yml does not affect port listening tested in ansible role inside travis https://travis-ci.org/juju4/ansible-kolide/jobs/635970338 default port allocation: 8080 OK change to 8888 NOK

Rereading through documentation, I didn't find another place where configuration should be changed for this.

If I review fleet config_dump, it seems that config /etc/kolide/kolide.yml is not used even if provided in command line /usr/local/bin/fleet serve --config /etc/kolide/kolide.yml and if syslog messages say it is https://travis-ci.org/juju4/ansible-kolide/jobs/635982933#L5771

-----> Execute command on default-8888-ubuntu-1804.
       ● kolide.service - kolide server
          Loaded: loaded (/lib/systemd/system/kolide.service; enabled; vendor preset: enabled)
          Active: active (running) since Sun 2020-01-12 15:09:44 UTC; 19s ago
        Main PID: 11938 (fleet)
           Tasks: 10 (limit: 4915)
          CGroup: /system.slice/kolide.service
           └─11938 /usr/local/bin/fleet serve --config /etc/kolide/kolide.yml

       Jan 12 15:09:44 default-8888-ubuntu-1804-1578841541 systemd[1]: Started kolide server.
       Jan 12 15:09:44 default-8888-ubuntu-1804-1578841541 kolide[11938]: Using config file:  /etc/kolide/kolide.yml
       Jan 12 15:09:45 default-8888-ubuntu-1804-1578841541 kolide[11938]: {"component":"service","err":null,"method":"ListUsers","took":"663.786µs","ts":"2020-01-12T15:09:45.007324742Z","user":"none"}
       Jan 12 15:09:45 default-8888-ubuntu-1804-1578841541 kolide[11938]: {"address":"0.0.0.0:8080","msg":"listening","transport":"https","ts":"2020-01-12T15:09:45.008980033Z"}

port displayed as not updated Config loading in https://github.com/kolide/fleet/blob/b524d813ca7ee2935ddc0f0ea86d0bd60fc35888/server/config/config.go#L501

I didn't find an option to validate config file just in case but anyway, no error message from fleet

few notes

port allocation seems to work fine with DetectionLab but fleet server part uses deprecated https://github.com/kolide/kolide-quickstart/blob/master/demo.sh#L33
if passing arguments through cli, that seems to work. strangely not in systemd (with reference to https://github.com/kolide/fleet/blob/545bc6fccb278d9489914f36b85d879b1a0ba17d/docs/infrastructure/systemd.md but unprivileged, ok cli+sudo). failing on "--mysql_password"...

What version of `fleet` are you using (`fleet version --full`)?

2.4.0

What operating system are you using?

tested on ubuntu bionic

What did you do?

ansible role set following /etc/kolide/kolide.yml

       #
       # Ansible managed
       #

       # https://github.com/kolide/fleet/blob/master/docs/infrastructure/configuring-the-fleet-binary.md
       mysql:
         address: localhost:3306
         database: kolidedb
         username: kolide
         password: Kolide!2017@
       redis:
         address: 127.0.0.1:6379
         password: redis_password_to_change
       server:
         server_address: 0.0.0.0:8888
         cert: /etc/ssl/default-8888-ubuntu-1804-1578838546.lxd.crt
         key: /etc/ssl/private/default-8888-ubuntu-1804-1578838546.lxd.key
         tls: true
       auth:
         jwt_key: BrkpHpfdRFInhf7AmWvgJ955Bge1ZSk9g2MeYaRqI
       logging:
         json: true
         debug: false
         disable_banner: false
       filesystem:
         result_log_file: /var/log/kolide/osquery_result
         status_log_file: /var/log/kolide/osquery_status
         enable_log_rotation: true
         label_query_update_interval: 1h
       app:
         token_key_size: 36
       #  invite_token_validity_period: 1d
       session:
         key_size: 64
       #  duration: 30d

What did you expect to see?

fleet daemon listing on port 8888 fleetctl can login to this port

What did you see instead?

listening on port 8080 https://travis-ci.org/juju4/ansible-kolide/jobs/635970338#L5569

       tcp6       0      0 :::8080                 :::*                    LISTEN      11776/fleet

fleectl can't connect to 8888 https://travis-ci.org/juju4/ansible-kolide/jobs/635970338#L5035

       fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["fleetctl", "login", "--email", "changethis@example.com", "--password", "admin1234#"], "delta": "0:00:00.044934", "end": "2020-01-12 14:19:39.185427", "msg": "non-zero return code", "rc": 1, "start": "2020-01-12 14:19:39.140493", "stderr": "error logging in: POST /api/v1/kolide/login: Post https://0.0.0.0:8888/api/v1/kolide/login: dial tcp 0.0.0.0:8888: connect: connection refused", "stderr_lines": ["error logging in: POST /api/v1/kolide/login: Post https://0.0.0.0:8888/api/v1/kolide/login: dial tcp 0.0.0.0:8888: connect: connection refused"], "stdout": "", "stdout_lines": []}

zwass commented 4 years ago

I am able to successfully configure the server address using flags, a config file, and environment variables. Are you sure you are using these tools correctly?

$ ./build/fleet config_dump --server_address=localhost:8888 | grep -B 1 8888
server:
  address: localhost:8888
$ cat > test.config
server:
  address: localhost:8888
$ ./build/fleet config_dump --config test.config | grep -B 1 8888
server:
  address: localhost:8888
$ KOLIDE_SERVER_ADDRESS=localhost:8888 ./build/fleet config_dump | grep -B 1 8888
server:
  address: localhost:8888

Further, the server does indeed start on the requested port when I use fleet serve with the config file.

juju4 commented 4 years ago

Trying in my side in an lxd container

root@default-ubuntu-1804-1578794933:~# fleet config_dump --server_address=localhost:8888 | grep -B 1 8888
server:
  address: localhost:8888
root@default-ubuntu-1804-1578794933:~# cat > test.config
server:
  address: localhost:8888
fleet config_dump --config test.config | grep -B 1 8888
root@default-ubuntu-1804-1578794933:~# fleet config_dump --config test.config | grep -B 1 8888
server:
  address: localhost:8888
root@default-ubuntu-1804-1578794933:~# KOLIDE_SERVER_ADDRESS=localhost:8888 fleet config_dump | grep -B 1 8888
server:
  address: localhost:8888

But from your comment, I understand that fleet has not a default config file that it reads or not the one I was supposing (/etc/kolide/kolide.yml or from running fleet). Have to give config arguments systematically.

Running manually, fleet serve works

root@default-ubuntu-1804-1578794933:~# fleet serve --config /etc/kolide/kolide.yml 
Using config file:  /etc/kolide/kolide.yml
{"component":"service","err":null,"method":"ListUsers","took":"1.910225ms","ts":"2020-01-19T14:59:24.38024072Z","user":"none"}
{"address":"0.0.0.0:8080","msg":"listening","transport":"https","ts":"2020-01-19T14:59:24.382842895Z"}

but if changing server_address port to 8888, it does not update

root@default-ubuntu-1804-1578794933:~# grep -B1 address /etc/kolide/kolide.yml 
mysql:
  address: localhost:3306
--
redis:
  address: 127.0.0.1:6379
--
server:
  server_address: 0.0.0.0:8888
root@default-ubuntu-1804-1578794933:~# fleet serve --config /etc/kolide/kolide.yml 
Using config file:  /etc/kolide/kolide.yml
{"component":"service","err":null,"method":"ListUsers","took":"1.910225ms","ts":"2020-01-19T14:59:24.38024072Z","user":"none"}
{"address":"0.0.0.0:8080","msg":"listening","transport":"https","ts":"2020-01-19T14:59:24.382842895Z"}
root@default-ubuntu-1804-1578794933:~# ss -tunap |grep fleet
tcp  ESTAB      0       0                  127.0.0.1:55756      127.0.0.1:3306   users:(("fleet",pid=14236,fd=3))
tcp  LISTEN     0       128                        *:8080               *:*      users:(("fleet",pid=14236,fd=5))

If I add environment variable, it works but config_dump is wrong if not giving path

root@default-ubuntu-1804-1578794933:~# KOLIDE_SERVER_ADDRESS=localhost:8888 fleet serve --config /etc/kolide/kolide.yml 
Using config file:  /etc/kolide/kolide.yml
{"component":"service","err":null,"method":"ListUsers","took":"541.844µs","ts":"2020-01-19T15:02:02.159419473Z","user":"none"}
{"address":"localhost:8888","msg":"listening","transport":"https","ts":"2020-01-19T15:02:02.16010335Z"}
root@default-ubuntu-1804-1578794933:~# ss -tunap |grep fleet
tcp  LISTEN     0       128                127.0.0.1:8888         0.0.0.0:*      users:(("fleet",pid=14325,fd=5))                                               
tcp  ESTAB      0       0                  127.0.0.1:55998      127.0.0.1:3306   users:(("fleet",pid=14325,fd=3))
root@default-ubuntu-1804-1578794933:~# fleet config_dump|grep -B1 address
mysql:
  address: localhost:3306
--
redis:
  address: localhost:6379
--
server:
  address: 0.0.0.0:8080

for reference

root@default-ubuntu-1804-1578794933:~# fleet version --full
fleet - version 2.4.0
  branch:       master
  revision:     70c6f5bd7d8a337a97f7702ee17088e74722896f
  build date:   2019-11-12T23:10:49Z
  build user:   zwass
  go version:   go1.12.8
root@default-ubuntu-1804-1578794933:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04 LTS
Release:        18.04
Codename:       bionic
root@default-ubuntu-1804-1578794933:~# uname -a
Linux default-ubuntu-1804-1578794933 4.4.0-170-generic #199-Ubuntu SMP Thu Nov 14 01:45:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

zwass commented 4 years ago

I think the issue is that you are using the key server_address in your kolide.yml when you need to be using the key address. We can see that it works in your test.config when you use the correct key.

If this doesn't fix it, please let me know and I will reopen this issue.

juju4 commented 4 years ago

good catch @zwass Sorry for the noise

kolide / fleet