Kolide Fleet, incoming host indefinitely

[TriflesT]

What version of fleet are you using (fleet version --full)?

What operating system are you using?

Ubuntu 16.04 for kolide, Windows Server 2012/Windows 10 hosts

What did you do?

I ran the following command to add the windows host with the parameters filled in:

./linux/launcher --enroll_secret=[secret from kolide] --hostname=[ip]:8080 --root_directory=[directory]/osq --insecure

My windows hosts have osquery version 4.4.0 installed.

What did you expect to see?

I expected to see the host added to kolide fleet

What did you see instead?

[zwass]

Can you run Launcher with the --debug flag and see if you can see any errors? Please paste the logs here.

[TriflesT]

Here is the launcher ran with the --debug flag.

[zwass]

I don't see anything unusual in the logs there. Is 192.168.1.61 (that osquery is connecting to) the same server as localhost (in your browser with the Fleet UI)? Can you connect to the DB and select * from hosts?

[TriflesT]

Yes 192.168.1.61 is the localhost server:

Here is the result of the query:

[zwass]

Can you actually run that query against your MySQL database that Fleet is connected to? I want to see what Fleet has stored about the hosts.

Also, please paste the actual text rather than screenshots. Thanks!

[TriflesT]

mysql> SELECT * FROM hosts -> ; +----+--------------------------------------+---------------------+---------------------+------------+---------+---------------------+----------------------------------+-----------+--------------------------------------+----------+-----------------+---------------+-------+---------------+-----------+----------------+-----------------+----------+-------------+------------------------------------------+--------------------+-------------------+-----------------+----------------+------------------+-----------------+---------------+---------------+---------------------+----------------------+-------------------+--------------------+------------+-------------------+---------------------+------------+--------------------+ | id | osquery_host_id | created_at | updated_at | deleted_at | deleted | detail_update_time | node_key | host_name | uuid | platform | osquery_version | os_version | build | platform_like | code_name | uptime | physical_memory | cpu_type | cpu_subtype | cpu_brand | cpu_physical_cores | cpu_logical_cores | hardware_vendor | hardware_model | hardware_version | hardware_serial | computer_name | primary_ip_id | seen_time | distributed_interval | logger_tls_period | config_tls_refresh | primary_ip | primary_mac | label_update_time | additional | enroll_secret_name | +----+--------------------------------------+---------------------+---------------------+------------+---------+---------------------+----------------------------------+-----------+--------------------------------------+----------+-----------------+---------------+-------+---------------+-----------+----------------+-----------------+----------+-------------+------------------------------------------+--------------------+-------------------+-----------------+----------------+------------------+-----------------+---------------+---------------+---------------------+----------------------+-------------------+--------------------+------------+-------------------+---------------------+------------+--------------------+ | 1 | f7ec49d1-1ee4-428e-a5e8-3ac36f2072b1 | 2020-08-04 13:16:41 | 2020-09-02 11:15:43 | NULL | 0 | 2020-08-04 15:16:53 | ccHJ5ph6XRd2TX0H8ZLEep1X5gKH18QL | ubuntu | 559225b7-8519-48e2-a029-29e50e666029 | ubuntu | 4.4.0 | Ubuntu 16.4.0 | | debian | | 15053000000000 | 4143108096 | x86_64 | 158 | Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz | 4 | 4 | | | | | ubuntu | NULL | 2020-09-02 11:15:44 | 10 | 10 | 300 | 10.0.2.15 | 08:00:27:26:69:1d | 2020-08-04 15:16:53 | {} | default | | 3 | 76973b88-a440-4286-9655-7e0537fe7635 | 2020-09-02 11:12:31 | 2020-09-02 11:15:45 | NULL | 0 | 1970-01-02 07:30:00 | 8Ij41jl4bE5vdmBFS2gpJ3xunuPzxV27 | | | | | | | | | 0 | 0 | | | | 0 | 0 | | | | | | NULL | 2020-09-02 11:15:45 | 10 | 10 | 0 | | | 1970-01-02 07:30:00 | NULL | default | +----+--------------------------------------+---------------------+---------------------+------------+---------+---------------------+----------------------------------+-----------+--------------------------------------+----------+-----------------+---------------+-------+---------------+-----------+----------------+-----------------+----------+-------------+------------------------------------------+--------------------+-------------------+-----------------+----------------+------------------+-----------------+---------------+---------------+---------------------+----------------------+-------------------+--------------------+------------+-------------------+---------------------+------------+--------------------+ 2 rows in set (0.00 sec)

[zwass]

We can see that host (id 3) in the database, and the detail_update_time is old enough that it should receive the detail queries. Your log screenshot indicates that it is not receiving those.

Does that same host work if you connect via plain osquery rather than Launcher?

[kevensen]

I am actually seeing similar behavior with all hosts enrolled.

fleet - version 3.1.0
  branch:   HEAD
  revision:     c6ce648fef3bb39b6e604333ec47cff0e625ff8e
  build date:   
  build user:   root
  go version:   go1.11.6

My osquery host is Ubuntu and running version osqueryd version 4.4.0

I did not use the launcher.

[zwass]

@kevensen can you please run osqueryd with the --verbose --tls_dump flags and paste the output here?

Also, please feel free to join the #kolide channel in osquery Slack where we can have a quicker back-and-forth.

[kevensen]

Oct 05 10:41:17 pop-os.home.fakedomain.com osqueryd[3252]:   "error": "failed to ingest result: ingesting query kolide_detail_query_system_info: strconv.Atoi: parsing \"67314212864\": value out of range"
Oct 05 10:41:17 pop-os.home.fakedomain.com osqueryd[3252]: }
Oct 05 10:41:19 pop-os.home.fakedomain.com osqueryd[3252]: {"queries":{"kolide_detail_query_osquery_flags":[{"name":"config_refresh","value":"10"},{"name":"distributed_interval","value":"10"},{"name":"logger_tls_period","value":"10"}],"kolide_detail_query_osquery_info":[{"pid":"3252","uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","instance_id":"9a60d2e0-2d19-4012-928e-a8b34143e4f5","version":"4.4.0","config_hash":"b01efbf375ac6767f259ae98751154fef727ce35","config_valid":"1","extensions":"active","build_platform":"1","build_distro":"centos7","start_time":"1601919469","watcher":"3250","platform_mask":"9"}],"kolide_detail_query_system_info":[{"hostname":"pop-os.home.fakedomain.com","uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","cpu_type":"x86_64","cpu_subtype":"165","cpu_brand":"Intel(R) Core(TM) i7-10875H CPU @ 2.30GHz","cpu_physical_cores":"8","cpu_logical_cores":"16","cpu_microcode":"0xc8","physical_memory":"67314212864","hardware_vendor":"System76","hardware_model":"Oryx Pro","hardware_version":"oryp6","hardware_serial":"123456789","board_vendor":"System76","board_model":"Oryx Pro","board_version":"oryp6","board_serial":"123456789","computer_name":"pop-os.home.fakedomain.com","local_hostname":"pop-os.home.fakedomain.com"}],"kolide_detail_query_uptime":[{"days":"0","hours":"0","minutes":"4","seconds":"43","total_seconds":"283"}],"kolide_label_query_6":[{"1":"1"}],"kolide_detail_query_network_interface":[{"address":"192.168.0.193","mac":"80:fa:5b:7f:f8:dd"},{"address":"fe80::54ca:d36:452c:d7b2%enp41s0","mac":"80:fa:5b:7f:f8:dd"},{"address":"192.168.86.122","mac":"c8:58:c0:24:fb:f3"},{"address":"fe80::eee8:69f4:a59:2460%wlp0s20f3","mac":"c8:58:c0:24:fb:f3"},{"address":"127.0.0.1","mac":"00:00:00:00:00:00"},{"address":"::1","mac":"00:00:00:00:00:00"},{"address":"172.16.23.1","mac":"00:50:56:c0:00:01"},{"address":"fe80::250:56ff:fec0:1%vmnet1","mac":"00:50:56:c0:00:01"},{"address":"172.16.194.1","mac":"00:50:56:c0:00:08"},{"address":"fe80::250:56ff:fec0:8%vmnet8","mac":"00:50:56:c0:00:08"}],"kolide_detail_query_os_version":[{"name":"Pop!_OS","version":"20.04 LTS","major":"20","minor":"4","patch":"0","build":"","platform":"pop","platform_like":"ubuntu debian","codename":"focal","arch":"x86_64I1005 10:41:19.127517  3261 config.cpp:1213] Refreshing configuration state
Oct 05 10:41:19 pop-os.home.fakedomain.com osqueryd[3252]: I1005 10:41:19.127760  3261 tls.cpp:253] TLS/HTTPS POST request to URI: https://kolide.home.fakedomain.com/api/v1/osquery/config
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: I1005 10:41:21.651185  3263 tls.cpp:253] TLS/HTTPS POST request to URI: https://kolide.home.fakedomain.com/api/v1/osquery/log
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: "}],"kolide_label_query_9":[]},"statuses":{"kolide_detail_query_osquery_flags":0,"kolide_detail_query_osquery_info":0,"kolide_detail_query_system_info":0,"kolide_detail_query_uptime":0,"kolide_label_query_6":0,"kolide_detail_query_network_interface":0,"kolide_detail_query_os_version":0,"kolide_label_query_9":0},"messages":{"kolide_detail_query_osquery_flags":"","kolide_detail_query_osquery_info":"","kolide_detail_query_system_info":"","kolide_detail_query_uptime":"","kolide_label_query_6":"","kolide_detail_query_network_interface":"","kolide_detail_query_os_version":"","kolide_label_query_9":""},"node_key":"v1v7cCrw2NgSKW9QteZovCHY98fwJd5A"}
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: {
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:   "error": "failed to ingest result: ingesting query kolide_detail_query_system_info: strconv.Atoi: parsing \"67314212864\": value out of range"
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: }
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: {"node_key":"v1v7cCrw2NgSKW9QteZovCHY98fwJd5A"}
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: {
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:   "decorators": {
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "load": [
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:       "SELECT uuid AS host_uuid FROM system_info;",
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:       "SELECT hostname AS hostname FROM system_info;"
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     ]
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:   },
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:   "options": {
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "disable_distributed": false,
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "distributed_interval": 10,
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "distributed_plugin": "tls",
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "distributed_tls_max_attempts": 3,
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "logger_plugin": "tls",
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "logger_tls_endpoint": "/api/v1/osquery/log",
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "logger_tls_period": 10,
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "pack_delimiter": "/"
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:   }
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: }
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: {"data":[{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:11 2020 UTC","unixTime":"1601919671","severity":"0","filename":"tls.cpp","line":"253","message":"TLS/HTTPS POST request to URI: https://kolide.home.fakedomain.com/api/v1/osquery/log","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"tls.cpp","line":"253","message":"TLS/HTTPS POST request to URI: https://kolide.home.fakedomain.com/api/v1/osquery/distributed/read","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"distributed.cpp","line":"120","message":"Executing distributed query: kolide_detail_query_network_interface: select address, mac\n                        from interface_details id join interface_addresses ia\n                               on ia.interface = id.interface where length(mac) > 0\n                               order by (ibytes + obytes) desc","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"distributed.cpp","line":"120","message":"Executing distributed query: kolide_detail_query_os_version: select * from os_version limit 1","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"distributed.cpp","line":"120","message":"Executing distributed query: kolide_detail_query_osquery_flags: select name, value from osquery_flags where name in (\"distributed_interval\", \"config_tls_refresh\", \"config_refresh\", \"logger_tls_period\")","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"distributed.cpp","line":"120","message":"Executing distributed query: kolide_detail_query_osquery_info: select * from osquery_info limit 1","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece7738I1005 10:41:21.740239  3266 tls.cpp:253] TLS/HTTPS POST request to URI: https://kolide.home.fakedomain.com/api/v1/osquery/distributed/write
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: 9-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"distributed.cpp","line":"120","message":"Executing distributed query: kolide_detail_query_system_info: select * from system_info limit 1","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"smbios_tables.cpp","line":"104","message":"Reading SMBIOS from sysfs DMI node","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"distributed.cpp","line":"120","message":"Executing distributed query: kolide_detail_query_uptime: select * from uptime limit 1","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"distributed.cpp","line":"120","message":"Executing distributed query: kolide_label_query_6: select 1;","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"distributed.cpp","line":"120","message":"Executing distributed query: kolide_label_query_9: select 1 from os_version where platform = 'centos' or name like '%centos%'","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"tls.cpp","line":"253","message":"TLS/HTTPS POST request to URI: https://kolide.home.fakedomain.com/api/v1/osquery/distributed/write","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:17 2020 UTC","unixTime":"1601919677","severity":"0","filename":"tls.cpp","line":"253","message":"TLS/HTTPS POST request to URI: https://kolide.home.fakedomain.com/api/v1/osquery/distributed/write","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:19 2020 UTC","unixTime":"1601919679","severity":"0","filename":"config.cpp","line":"1213","message":"Refreshing configuration state","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:19 2020 UTC","unixTime":"1601919679","severity":"0","filename":"tls.cpp","line":"253","message":"TLS/HTTPS POST request to URI: https://kolide.home.fakedomain.com/api/v1/osquery/config","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}}],"log_type":"status","node_key":"v1v7cCrw2NgSKW9QteZovCHY98fwJd5A"}

[zwass]

@kevensen Did you by chance make a custom build of Fleet for a 32 bit architecture? Looks like your host's memory value is overflowing a 32 bit int. We can certainly fix that by explicitly specifying 64 bit integers but I am wondering why/how you ended up in this position.

[kevensen]

I was actually thinking that as well. In my home lab I am attempting to run Fleet on a Raspberry Pi 3b+. Obviously not a production environment but an intellectual curiosity. So yeah, 32-bit.

[zwass]

@kevensen let's follow up in https://github.com/kolide/fleet/issues/2314 as I think this is unrelated to @TriflesT's issue (they seem to be using the official build).