search

clong / DetectionLab

Automate the creation of a lab environment complete with security tooling and logging best practices
MIT License
4.64k stars 987 forks source link

Fleet Service not starting / not getting installed on Logger #785

Closed haffa008 closed 2 years ago

haffa008 commented 2 years ago

Please verify that you are building from an updated Master branch before filing an issue: Yes, cloned the latest Master branch via GitHub desktop

Description of the issue:

While building the logger host, I'm running into the following error message that causes the build to hang:

    logger: The Splunk web interface is at https://logger:8000
    logger:
    logger: Init script installed at /etc/init.d/splunk.
    logger: Init script is configured to run at boot.
    logger: [12:51:06]: Downloading Palantir osquery configs...
    logger: Cloning into 'osquery-configuration'...
    logger: [12:51:07]: Installing Fleet...
    logger: 127.0.2.1 logger logger
    logger: mysql: [Warning] Using a password on the command line interface can be insecure.
    **_logger: No URLs found in -_**.
    **_logger: unzip:  cannot find or open fleet.zip, fleet.zip.zip or fleet.zip.ZIP._**
    logger: cp: cannot stat 'fleet/linux/fleetctl': No such file or directory
    logger: cp: cannot stat 'fleet/linux/fleet': No such file or directory
    logger: /tmp/vagrant-shell: line 301: fleet: command not found
    logger: cp: target '/opt/fleet/' is not a directory
    logger: Created symlink /etc/systemd/system/multi-user.target.wants/fleet.service → /etc/systemd/system/fleet.service.
    **_logger: [12:51:08]: Waiting for fleet service to start..._**
 # Always download the latest release of Fleet
    curl -s https://api.github.com/repos/fleetdm/fleet/releases | grep 'https://github.com' | grep "/fleet.zip" | cut -d ':' -f 2,3 | tr -d '"' | tr -d ' ' | head -1 | wget --progress=bar:force -i -
    unzip fleet.zip -d fleet
    cp fleet/linux/fleetctl /usr/local/bin/fleetctl && chmod +x /usr/local/bin/fleetctl
    cp fleet/linux/fleet /usr/local/bin/fleet && chmod +x /usr/local/bin/fleet

The actual file name in the URL can be seen below:

    "url": "https://api.github.com/repos/fleetdm/fleet/releases/assets/58816992",
    "id": 58816992,
    "node_id": "RA_kwDOEnd7fs4DgXng",
    "name": "**fleetctl_v4.11.0_linux.zip**",

Is because there is no actual "fleet.zip" in the URL: https://api.github.com/repos/fleetdm/fleet/releases being referred to in the "logger_bootstrap" script as seen in the code here:

Link to Gist Containing Build Logs:

NA. No local logs found in my Vagrant folder.

BSH77 commented 2 years ago

I am having some issues above. logger: The Splunk web interface is at https://logger:8000 logger: logger: Init script installed at /etc/init.d/splunk. logger: Init script is configured to run at boot. logger: [05:50:20]: Downloading Palantir osquery configs... logger: Cloning into 'osquery-configuration'... logger: [05:50:21]: Installing Fleet... logger: 127.0.2.1 logger logger logger: mysql: [Warning] Using a password on the command line interface can be insecure. logger: No URLs found in -. logger: unzip: cannot find or open fleet.zip, fleet.zip.zip or fleet.zip.ZIP. logger: cp: cannot stat 'fleet/linux/fleetctl': No such file or directory logger: cp: cannot stat 'fleet/linux/fleet': No such file or directory logger: /tmp/vagrant-shell: line 301: fleet: command not found logger: cp: target '/opt/fleet/' is not a directory logger: Created symlink /etc/systemd/system/multi-user.target.wants/fleet.service → /etc/systemd/system/fleet.service. logger: [05:50:22]: Waiting for fleet service to start...

haffa008 commented 2 years ago

vagrant@logger:~$ service fleet status ● fleet.service - fleet Fleet Loaded: loaded (/etc/systemd/system/fleet.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Wed 2022-03-09 07:51:51 UTC; 7s ago Process: 66651 ExecStart=/usr/local/bin/fleet serve --mysql_address=127.0.0.1:3306 --mysql_database=fleet --mysq> Main PID: 66651 (code=exited, status=203/EXEC)

Mar 09 07:51:51 logger systemd[1]: Started fleet Fleet. Mar 09 07:51:51 logger systemd[66651]: fleet.service: Failed to execute command: No such file or directory Mar 09 07:51:51 logger systemd[66651]: fleet.service: Failed at step EXEC spawning /usr/local/bin/fleet: No such fil> Mar 09 07:51:51 logger systemd[1]: fleet.service: Main process exited, code=exited, status=203/EXEC Mar 09 07:51:51 logger systemd[1]: fleet.service: Failed with result 'exit-code'.

mrpew commented 2 years ago

Same here. Looks similar to #604

haffa008 commented 2 years ago

Same here. Looks similar to #604

Yup! Looks to be a broken download URL / filename call in the script for Fleet. @clong Please advise.

mrpew commented 2 years ago

Vagrant/logger_bootstrap.sh L294

Quick fix:

    # fleetctl
    wget -O fleetctl.zip https://github.com/fleetdm/fleet/releases/download/fleet-v4.11.0/fleetctl_v4.11.0_linux.zip
    unzip fleetctl.zip
    cp fleetctl_v4.11.0_linux/fleetctl /usr/local/bin/fleetctl && chmod +x /usr/local/bin/fleetctl
    # fleet
    wget -O fleet.tgz https://github.com/fleetdm/fleet/releases/download/fleet-v4.11.0/fleet_v4.11.0_linux.tar.gz
    tar xfvz fleet.tgz
    cp fleet_v4.11.0_linux/fleet /usr/local/bin/fleet && chmod +x /usr/local/bin/fleet
    # create fleet dir for config files
    mkdir fleet
haffa008 commented 2 years ago

Vagrant/logger_bootstrap.sh L294

Quick fix:

    # fleetctl
    wget -O fleetctl.zip https://github.com/fleetdm/fleet/releases/download/fleet-v4.11.0/fleetctl_v4.11.0_linux.zip
    unzip fleetctl.zip
    cp fleetctl_v4.11.0_linux/fleetctl /usr/local/bin/fleetctl && chmod +x /usr/local/bin/fleetctl
    # fleet
    wget -O fleet.tgz https://github.com/fleetdm/fleet/releases/download/fleet-v4.11.0/fleet_v4.11.0_linux.tar.gz
    tar xfvz fleet.tgz
    cp fleet_v4.11.0_linux/fleet /usr/local/bin/fleet && chmod +x /usr/local/bin/fleet
    # create fleet dir for config files
    mkdir fleet

Awesome stuff @mrpew Thanks for that quick fix! It worked for me!

All seems to be good as per the Post build checks script!

PS C:\Users......\Documents\DetectionLab\Vagrant> .\post_build_checks.ps1 [*] Verifying that Splunk is reachable... [ √ ] Splunk is running and reachable!

[*] Verifying that Fleet is reachable... [ √ ] Fleet is running and reachable!

[*] Verifying that Microsoft ATA is reachable... [ √ ] Microsoft ATA is running and reachable!

[*] Verifying that Velociraptor is reachable... [ √ ] Velociraptor is running and reachable!

[*] Verifying that Guacamole is reachable... [ √ ] Guacamole is running and reachable!

haffa008 commented 2 years ago

@mrpew @clong everything seems fine now except that I don't have any devices reporting into the Fleet.

I noticed that /usr/bin/yq is erroring out in applying the enrollment secret with the below error during the "logger" setup:

" logger: mysql: [Warning] Using a password on the command line interface can be insecure. logger: Updated enrollment secret logger: Error: unknown command "w" for "yq" logger: Run 'yq --help' for usage. logger: Error: unknown command "w" for "yq" logger: Run 'yq --help' for usage. logger: Error: no spec field on "" document "

Did you also encounter this error by any chance?

haffa008 commented 2 years ago

Use "yq [command] --help" for more information about a command. vagrant@logger:~$ /usr/bin/yq -i '/tmp/options.yaml' Error: unknown command "/tmp/options.yaml" for "yq" Run 'yq --help' for usage. vagrant@logger:~$ /usr/bin/yq -i "/tmp/options.yaml" Error: unknown command "/tmp/options.yaml" for "yq" Run 'yq --help' for usage. vagrant@logger:~$ /usr/bin/yq -i </tmp/options.yaml> -bash: syntax error near unexpected token `newline' vagrant@logger:~$ /usr/bin/yq -i 'spec.config.options.enroll_secret= ""enrollmentsecret""' Error: unknown command "spec.config.options.enroll_secret= \"\"enrollmentsecret\"\"" for "yq" Run 'yq --help' for usage. vagrant@logger:~$ /usr/bin/yq -i 'spec.config.options.enroll_secret' 'enrollmentsecret' Error: unknown command "spec.config.options.enroll_secret" for "yq" Run 'yq --help' for usage.

haffa008 commented 2 years ago

Not able to figure out the syntax for "yq" to commit the config to YAML file!!

clong commented 2 years ago

Just pushed a fix. Update git and try again?

haffa008 commented 2 years ago

Did not work @clong 

logger: --2022-03-11 10:33:29--  https://github.com/fleetdm/fleet/releases/download/fleet-v4.11.0/fleet_v4.11.0_linux.tar.gz
logger: Resolving github.com (github.com)... 13.234.176.102
logger: Connecting to github.com (github.com)|13.234.176.102|:443... connected.
logger: HTTP request sent, awaiting response... 302 Found
logger: Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/309820286/38fd4833-7775-4071-a138-3a5c5a1714fd?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220311%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220311T103330Z&X-Amz-Expires=300&X-Amz-Signature=d50522864b2d7ce0d6f20f46f8909ed3c8dfb811d3595a14f08c91273c5db420&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=309820286&response-content-disposition=attachment%3B%20filename%3Dfleet_v4.11.0_linux.tar.gz&response-content-type=application%2Foctet-stream [following]
logger: --2022-03-11 10:33:29--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/309820286/38fd4833-7775-4071-a138-3a5c5a1714fd?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220311%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220311T103330Z&X-Amz-Expires=300&X-Amz-Signature=d50522864b2d7ce0d6f20f46f8909ed3c8dfb811d3595a14f08c91273c5db420&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=309820286&response-content-disposition=attachment%3B%20filename%3Dfleet_v4.11.0_linux.tar.gz&response-content-type=application%2Foctet-stream
logger: Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.111.133, ...
logger: Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
logger: HTTP request sent, awaiting response... 200 OK
logger: Length: 28278881 (27M) [application/octet-stream]
logger: Saving to: ‘fleet_v4.11.0_linux.tar.gz’
logger:

fleet_v4.11.0_linux 100%[===================>] 26.97M 3.49MB/s in 7.7s logger: logger: 2022-03-11 10:33:38 (3.50 MB/s) - ‘fleet_v4.11.0_linux.tar.gz’ saved [28278881/28278881] logger: logger: FINISHED --2022-03-11 10:33:38-- logger: Total wall clock time: 10s logger: Downloaded: 1 files, 27M in 7.7s (3.50 MB/s) logger: --2022-03-11 10:33:38-- https://github.com/fleetdm/fleet/releases/download/fleet-v4.11.0/fleetctl_v4.11.0_linux.tar.gz logger: Resolving github.com (github.com)... 13.234.176.102 logger: Connecting to github.com (github.com)|13.234.176.102|:443... connected. logger: HTTP request sent, awaiting response... 302 Found logger: Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/309820286/c13ae8a0-73c0-4646-a45f-4cf912d5fd36?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220311%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220311T103340Z&X-Amz-Expires=300&X-Amz-Signature=ce9d3ab972cca83a2859b748e0bfe438489ab13dbcfdf8de6cd434bb5c6485ce&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=309820286&response-content-disposition=attachment%3B%20filename%3Dfleetctl_v4.11.0_linux.tar.gz&response-content-type=application%2Foctet-stream [following] logger: --2022-03-11 10:33:38-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/309820286/c13ae8a0-73c0-4646-a45f-4cf912d5fd36?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220311%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220311T103340Z&X-Amz-Expires=300&X-Amz-Signature=ce9d3ab972cca83a2859b748e0bfe438489ab13dbcfdf8de6cd434bb5c6485ce&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=309820286&response-content-disposition=attachment%3B%20filename%3Dfleetctl_v4.11.0_linux.tar.gz&response-content-type=application%2Foctet-stream logger: Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.110.133, ... logger: Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected. logger: HTTP request sent, awaiting response... 200 OK logger: Length: 16350769 (16M) [application/octet-stream] logger: Saving to: ‘fleetctl_v4.11.0_linux.tar.gz’ logger: fleetctl_v4.11.0_li 100%[===================>] 15.59M 11.9MB/s in 1.3s logger: logger: 2022-03-11 10:33:40 (11.9 MB/s) - ‘fleetctl_v4.11.0_linux.tar.gz’ saved [16350769/16350769] logger: logger: FINISHED --2022-03-11 10:33:40-- logger: Total wall clock time: 2.6s logger: Downloaded: 1 files, 16M in 1.3s (11.9 MB/s) logger: fleet_v4.11.0_linux/CHANGELOG.md logger: fleet_v4.11.0_linux/LICENSE logger: fleet_v4.11.0_linux/README.md logger: fleet_v4.11.0_linux/fleet logger: fleetctl_v4.11.0_linux/CHANGELOG.md logger: fleetctl_v4.11.0_linux/LICENSE logger: fleetctl_v4.11.0_linux/README.md logger: fleetctl_v4.11.0_linux/fleetctl logger: Migrations completed. logger: cp: target '/opt/fleet/' is not a directory logger: Created symlink /etc/systemd/system/multi-user.target.wants/fleet.service → /etc/systemd/system/fleet.service. logger: [10:33:50]: Waiting for fleet service to start...

ostefano commented 2 years ago

Same issue here, getting the same output of @haffa008

clong commented 2 years ago

looks like yq changed its syntax in v4: https://mikefarah.gitbook.io/yq/v/v4.x/upgrading-from-v3#updating-writing-documents

I'm testing the fix now

clong commented 2 years ago

Fleet also no longer supports fleetctl get options and is now fleetctl get config

clong commented 2 years ago

Confirmed fixed in https://github.com/clong/DetectionLab/commit/0aeb9a477635e16fe9167c60f9a92e84e3fab859

haffa008 commented 2 years ago

Awesome stuff @clong I did see that the previous Fleet command was not exporting the config properly since the syntax changed with the upgrade and I am glad you figured that and the YQ as well for a permanent fix!

Attached is the confirmation that the Fleet config was updated successfully and logger deployment was successful too! Fleet

ostefano commented 2 years ago

@clong I think we might need to reopen this :( The ansible task is failing now as "curl -s https://api.github.com/repos/fleetdm/fleet/releases | grep 'https://github.com' | grep "/fleet.zip"" returns an empty string, so the service never gets installed.

ostefano commented 2 years ago

@clong here my pull request fixing this: https://github.com/clong/DetectionLab/pull/789

clong commented 2 years ago

Oh, whoops, I forgot to port the fixed code over to the ESXi section. Thanks for the PR!