clong / DetectionLab

Automate the creation of a lab environment complete with security tooling and logging best practices
MIT License
4.59k stars 979 forks source link

logger - fleet does not download and install properly #849

Closed kiyori-lw closed 1 year ago

kiyori-lw commented 1 year ago

Please verify that you are building from an updated Master branch before filing an issue.

Description of the issue:

While building the logger host, I'm running into the following error message that causes the build to stop:

    logger: [11:00:38]: Installing Fleet...
    logger: 127.0.2.1 logger logger
    logger: mysql: [Warning] Using a password on the command line interface can be insecure.
    logger: No URLs found in -.
    logger: No URLs found in -.
    logger: tar: fleet_*.tar.gz: Cannot open: No such file or directory
    logger: tar: Error is not recoverable: exiting now
    logger: tar: fleetctl_*.tar.gz: Cannot open: No such file or directory
    logger: tar: Error is not recoverable: exiting now
    logger: cp: cannot stat 'fleetctl_*/fleetctl': No such file or directory
    logger: cp: cannot stat 'fleet_*/fleet': No such file or directory
    logger: /tmp/vagrant-shell: line 307: fleet: command not found
    logger: Created symlink /etc/systemd/system/multi-user.target.wants/fleet.service → /etc/systemd/system/
fleet.service.
    logger: [11:00:40]: Waiting for fleet service to start...

Link to Gist Containing Build Logs:

Bisonito commented 1 year ago

I am having the same issue. I attempted to find a way to fix the issue but haven't been able to solve it yet.

kiyori-lw commented 1 year ago

as a workaround, i just added wget and the current release download link in logger_bootstrp.sh; i commented out the curl -s lines

GitHopper33 commented 1 year ago

Building on kiyori-lw's comment, I created a fairly clean workaround to successfully download the latest release of Fleet and Fleetctl (without hard-coding the version) :

The replacement download commands also use jq to help parse the curl request output (fortunately, jq is already installed in the logger vm).

In the logger_bootstrp.sh file, replace the original two lines (or comment out): curl -s https://github.com/fleetdm/fleet/releases | grep _linux.tar.gz | grep href | grep -v orbit | grep -v fleetctl | cut -d '"' -f 2 | head -1 | sed 's#^#https://github.com#g' | wget --progress=bar:force -i -

curl -s https://github.com/fleetdm/fleet/releases | grep _linux.tar.gz | grep href | grep fleetctl | cut -d '"' -f 2 | head -1 | sed 's#^#https://github.com#g' | wget --progress=bar:force -i -

with: curl -s https://api.github.com/repos/fleetdm/fleet/releases/latest | jq '.assets[] | select(.name|match("_linux.tar.gz$")) | .browser_download_url' | sed 's\"//g' | grep fleet_ | wget -q --show-progress --progress=bar:force -i -

curl -s https://api.github.com/repos/fleetdm/fleet/releases/latest | jq '.assets[] | select(.name|match("_linux.tar.gz$")) | .browser_download_url' | sed 's\"//g' | grep fleetctl_ | wget -q --show-progress --progress=bar:force -i -

Cheers :)

andreipintea97 commented 1 year ago

Hi, I encountered the same issue as kyiori-lw.

I tried your option, GitHopper33, but I am constantly getting an "The SSH command responded with a non-zero exit status". Not sure what else to try to install the logger.

Edit: It works now, just had to change sed 's\"//g' to sed 's/"//g"

GitHopper33 commented 1 year ago

Ok thanks for the info andrepintea97. I saw a separate ssh issue here (I don't recall the issue #) which I would typically resolve by destroying and re-creating the logger. In some cases, I had to delete the .vagrant folder as well.

Not sure if you had a typo or not in your example above (the before string doesn't match and the after string makes me wonder - lol). Perhaps using the "add code" option in the comment tool bar will help keep whatever you type intact (that was helpful for me to post my solution accurately). Either way, I'm glad your approach worked!

DYarizadeh commented 1 year ago

This works -

curl -s https://api.github.com/repos/fleetdm/fleet/releases/latest | jq '.assets[] | select(.name|match("_linux.tar.gz$")) | .browser_downloadurl' | sed 's/"//g' | grep fleet | wget -q --show-progress --progress=bar:force -i -

curl -s https://api.github.com/repos/fleetdm/fleet/releases/latest | jq '.assets[] | select(.name|match("_linux.tar.gz$")) | .browser_downloadurl' | sed 's/"//g' | grep fleetctl | wget -q --show-progress --progress=bar:force -i -

GitHopper33 commented 1 year ago

Thanks andreipintea97 and DYarizadeh - your examples show that that sed is flexible in the escaping character. I also found this helpful tidbit on a post elsewhere:

"The character directly after the s determines which character is the separator, which must appear three times in the s command."

Oh - fyi: Not sure if you had the underscore after fleet and fleetctl in your greps. When I didn't use the "add code" option, this comment interpreter stripped them out.

clong commented 1 year ago

Fixed in https://github.com/clong/DetectionLab/commit/b002e6115ef20324f48829477e1bae5e9fb714ca , thanks for the updated command, @DYarizadeh !