Open sjkeerthi opened 5 months ago
I found another strange after when I install it remove all my existing cronjob that what I set :( all the cronjob are gone.
Hi @sjkeerthi. Can you use this script to install the agents on Rocky Linux. While Rocky Linux is not supported yet meanwhile I did little change in script to add rocky.
#!/bin/bash
# MGMT_CONSOLE_URL: Example: threatmapper.customer.com or 65.65.65.65
export MGMT_CONSOLE_URL="${MGMT_CONSOLE_URL}"
export DEEPFENCE_KEY="${DEEPFENCE_KEY}"
if [[ -z "$MGMT_CONSOLE_URL" ]]; then
echo "env MGMT_CONSOLE_URL is not set"
exit 1
fi
if [[ -z "$DEEPFENCE_KEY" ]]; then
echo "env DEEPFENCE_KEY is not set"
exit 1
fi
export MGMT_CONSOLE_PORT="443"
export MGMT_CONSOLE_URL_SCHEMA="https"
export DF_HOSTNAME="$(hostname)"
export DF_LOG_LEVEL="info"
MANAGEMENT_CONSOLE_URL="$MGMT_CONSOLE_URL_SCHEMA://$MGMT_CONSOLE_URL:$MGMT_CONSOLE_PORT"
OS_ID=$(grep -oP '(?<=^ID=).+' /etc/os-release | tr -d '"')
if [[ "$OS_ID" == "amzn" || "$OS_ID" == "centos" ]]; then
# Do necessary installs for Amazon Linux
yum -y install logrotate jq curl
if [[ "$?" != "0" ]]; then
echo "Failed to install logrotate"
exit 1
fi
elif [[ "$OS_ID" == "rocky" ]]; then
# Do necessary installs for Rocky Linux
dnf -y install logrotate jq curl --skip-broken
if [[ "$?" != "0" ]]; then
echo "Failed to install logrotate"
exit 1
fi
else
# Do necessary installs for Ubuntu
apt-get -y install logrotate jq curl
if [[ "$?" != "0" ]]; then
echo "Failed to install logrotate"
exit 1
fi
fi
access_token_response=$(curl -m 5 -s -k "$MANAGEMENT_CONSOLE_URL/deepfence/auth/token" \
--header 'Content-Type: application/json' \
--data "{\"api_token\": \"$DEEPFENCE_KEY\"}")
if [[ $access_token_response == "" ]]; then
echo "Failed to connect to the management console"
exit 1
fi
access_token=$(jq -r '.access_token' <<< "$access_token_response")
if [[ $access_token == "" || $access_token == "null" ]]; then
echo "Failed to authenticate"
echo "$access_token_response"
exit 1
fi
download_url_response=$(curl -m 5 -s -k "$MANAGEMENT_CONSOLE_URL/deepfence/agent-deployment/binary/download-url" \
--header "Authorization: Bearer $access_token")
if [[ $download_url_response == "" ]]; then
echo "Failed to get agent binary download url"
exit 1
fi
start_agent_script_download_url=$(jq -r '.start_agent_script_download_url' <<< "$download_url_response")
if [[ $start_agent_script_download_url == "" ]]; then
echo "Failed to get agent binary download url"
echo "$download_url_response"
exit 1
fi
uninstall_agent_script_download_url=$(jq -r '.uninstall_agent_script_download_url' <<< "$download_url_response")
if [[ $uninstall_agent_script_download_url == "" ]]; then
echo "Failed to get agent binary download url"
echo "$download_url_response"
exit 1
fi
curl -k -o uninstall_deepfence.sh "$uninstall_agent_script_download_url"
chmod +x uninstall_deepfence.sh
echo "Uninstalling existing Deepfence agent installation, if any"
systemctl stop deepfence-agent.service
systemctl disable deepfence-agent.service
rm -f /etc/systemd/system/deepfence-agent.service
bash uninstall_deepfence.sh
if [[ ! -d "/opt/deepfence" ]]; then
mkdir -p /opt/deepfence /opt/deepfence/var/log/
fi
architecture=""
case $(uname -m) in
i386) architecture="386" ;;
i686) architecture="386" ;;
x86_64) architecture="amd64" ;;
arm) dpkg --print-architecture | grep -q "arm64" && architecture="arm64" || architecture="arm" ;;
esac
echo "Detected architecture: $architecture"
agent_binary_download_url=$(jq -r --arg architecture "agent_binary_${architecture}_download_url" '.[$architecture]' <<< "$download_url_response")
agent_binary_filename=$(basename "$agent_binary_download_url")
agent_binary_filename=$(cut -f1 -d"?" <<< "$agent_binary_filename")
if [[ $agent_binary_download_url == "" || $agent_binary_filename == "" ]]; then
echo "Failed to get agent binary download url"
echo "$download_url_response"
exit 1
fi
echo "Downloading agent binary from $agent_binary_download_url to /opt/deepfence/$agent_binary_filename"
curl -k -o "/opt/deepfence/$agent_binary_filename" "$agent_binary_download_url"
curl -k -o /opt/deepfence/start_deepfence_agent.sh "$start_agent_script_download_url"
chmod +x "/opt/deepfence/start_deepfence_agent.sh"
tar -xzf "/opt/deepfence/$agent_binary_filename" -C /opt/deepfence/
echo "MGMT_CONSOLE_URL: $MGMT_CONSOLE_URL"
echo "MGMT_CONSOLE_PORT: $MGMT_CONSOLE_PORT"
echo "DF_HOSTNAME: $DF_HOSTNAME"
echo "Installing Deepfence agent as daemon service"
cat << EOF > /etc/systemd/system/deepfence-agent.service
[Unit]
Description=Deepfence Agent Service
After=network.target
[Service]
Environment=MGMT_CONSOLE_URL="$MGMT_CONSOLE_URL"
Environment=DEEPFENCE_KEY="$DEEPFENCE_KEY"
Environment=MGMT_CONSOLE_PORT="$MGMT_CONSOLE_PORT"
Environment=MGMT_CONSOLE_URL_SCHEMA="$MGMT_CONSOLE_URL_SCHEMA"
Environment=DF_HOSTNAME="$(hostname)"
Environment=DF_LOG_LEVEL="$DF_LOG_LEVEL"
User=root
Group=root
Restart=on-failure
Type=forking
ExecStart=/opt/deepfence/start_deepfence_agent.sh
WorkingDirectory=/opt/deepfence
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable deepfence-agent.service
systemctl start deepfence-agent.service
systemctl status deepfence-agent.service
Thank you so much it works now.
There is one problem while we run this script it remove all my existing cronjob that I had in the VM. I hope either warning msg should be there in the script with sleep 60 before it proceed. Or please update in the document either to make sure to take a backup of the existing cronjob. Since I was very new I really have no idea one of production box all the cronjob have gone :( Kindly consider this.
I tried installing the agent on CentOS Linux release 7.5.1804 after agent installed and stared I could not see the agent appears in the deepfencer UI.
` deepfence-agent.service - Deepfence Agent Service Loaded: loaded (/etc/systemd/system/deepfence-agent.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2024-06-14 10:16:36 UTC; 1min 46s ago Process: 8955 ExecStart=/opt/deepfence/start_deepfence_agent.sh (code=exited, status=0/SUCCESS) Main PID: 9009 (start_deepfence) CGroup: /system.slice/deepfence-agent.service ├─9009 /bin/bash /opt/deepfence/df-agents/v4-services-vue-box/home/deepfence/start_deepfenced.sh ├─9011 /bin/sh -c ulimit -l unlimited; /opt/deepfence/df-agents/v4-services-vue-box/bin/deepfenced >> /opt/deepfence/df-agents/v4-services-vue-box/var/log/supervisor/deepfenced.log 2>&... ├─9013 /opt/deepfence/df-agents/v4-services-vue-box/bin/deepfenced ├─9019 /bin/bash -c /opt/deepfence/df-agents/v4-services-vue-box/home/deepfence/run_fluentbit.sh ├─9020 /opt/deepfence/df-agents/v4-services-vue-box/usr/local/discovery/deepfence-discovery --mode=probe --probe.log.level="info" --probe.spy.interval=5s --probe.publish.interval=10s -... ├─9023 /opt/deepfence/df-agents/v4-services-vue-box/home/deepfence/bin/package-scanner -socket-path /opt/deepfence/df-agents/v4-services-vue-box/tmp/package-scanner.sock -mode grpc-ser... └─9189 sleep 5 '
In the logs which I could see
2024/06/14 10:19:30 [DEBUG] POST https://"do-scan.xxxx.net":443/deepfence/auth/token 2024/06/14 10:19:30 [ERR] POST https://"do-scan.commusoft.net":443/deepfence/auth/token request failed: Post "https://\"do-scan.commusoft.net\":443/deepfence/auth/token": dial tcp: lookup "do-scan.xxx.net": no such host Fri, 14 Jun 2024 10:19:30 +0000 WRN prog/probe.go:175 Failed to authenticate. Retrying...
But if I run the same instead of the systemctl I run manually it works
/opt/deepfence/df-agents/v4-services-vue-box/usr/local/discovery/deepfence-discovery --mode=probe --probe.log.level="info" --probe.spy.interval=5s --probe.publish.interval=10s --probe.docker.interval=10s --probe.insecure=true --probe.docker=false --probe.podman=false --probe.cri=false --probe.token="ZGVmYXVsdDpkZTI1NmVlOC01YjY3LTQ4MWUtYmFhYi1lNzczMDU3NTlixxxxx" --probe.processes=true --probe.endpoint.report=true --probe.conntrack=false --probe.track.deploads=false https://"do-scan.xxxxxnet"
# journalctl -u deepfence-agent.service
-- Logs begin at Fri 2024-06-14 10:34:28 UTC, end at Fri 2024-06-14 10:36:48 UTC. --
Jun 14 10:34:35 v4-services-vue-box systemd[1]: Starting Deepfence Agent Service...
Jun 14 10:34:35 v4-services-vue-box start_deepfence_agent.sh[713]: Got hostname:
Jun 14 10:34:35 v4-services-vue-box start_deepfence_agent.sh[713]: v4-services-vue-box
Jun 14 10:34:35 v4-services-vue-box start_deepfence_agent.sh[713]: Deepfence agent install dir: /opt/deepfence/df-agents/v4-services-vue-box
Jun 14 10:34:35 v4-services-vue-box start_deepfence_agent.sh[713]: /opt/deepfence/df-agents/v4-services-vue-box/bin:/opt/deepfence/df-agents/v4-services-vue-box/usr/local/bin:/opt/deepfence/df-agent
Jun 14 10:34:35 v4-services-vue-box start_deepfence_agent.sh[713]: /opt/deepfence/start_deepfence_agent.sh: line 152: /bin/pidof: No such file or directory
Jun 14 10:34:36 v4-services-vue-box sudo[744]: root : TTY=unknown ; PWD=/opt/deepfence ; USER=root ; COMMAND=/bin/ln -sf bash /bin/sh
Jun 14 10:34:36 v4-services-vue-box sudo[760]: root : TTY=unknown ; PWD=/opt/deepfence ; USER=root ; COMMAND=/bin/ln -s / /fenced/mnt/host
Jun 14 10:34:36 v4-services-vue-box start_deepfence_agent.sh[713]: ln: failed to create symbolic link ‘/fenced/mnt/host’: No such file or directory
Jun 14 10:34:36 v4-services-vue-box start_deepfence_agent.sh[713]: Copying agent to DF installation dir
Jun 14 10:34:36 v4-services-vue-box start_deepfence_agent.sh[713]: cp: cannot create regular file '/opt/deepfence/df-agents/v4-services-vue-box/bin/./cp': Text file busy
Jun 14 10:34:40 v4-services-vue-box start_deepfence_agent.sh[713]: cp: cannot stat '/opt/deepfence/deepfence/*': No such file or directory
Jun 14 10:34:40 v4-services-vue-box start_deepfence_agent.sh[713]: Redirecting to /bin/systemctl start cron.service
Jun 14 10:34:40 v4-services-vue-box start_deepfence_agent.sh[713]: Failed to start cron.service: Unit not found.
Jun 14 10:34:40 v4-services-vue-box start_deepfence_agent.sh[713]: Starting agent...
Jun 14 10:34:40 v4-services-vue-box start_deepfence_agent.sh[713]: Deepfence agent install dir: /opt/deepfence/df-agents/v4-services-vue-box
Jun 14 10:34:40 v4-services-vue-box start_deepfence_agent.sh[713]: Deepfence agent base dir: /opt/deepfence
Jun 14 10:34:40 v4-services-vue-box start_deepfence_agent.sh[713]: Deepfence agent hostname: v4-services-vue-box
Jun 14 10:34:40 v4-services-vue-box start_deepfence_agent.sh[713]: Deepfence management console url: "do-scan.xxxx.net"
Jun 14 10:34:40 v4-services-vue-box start_deepfence_agent.sh[713]: Deepfence management console port: 443
Jun 14 10:34:40 v4-services-vue-box start_deepfence_agent.sh[713]: Deepfence key: "xxxxxxxxxxxxxxxxxxxxxx"
Jun 14 10:34:40 v4-services-vue-box systemd[1]: Started Deepfence Agent Service.
`
/opt/deepfence/df-agents/v4-services-vue-box/home/deepfence/bin/package-scanner -socket-path /opt/deepfence/df-agents/v4-services-vue-box/tmp/package-scanner.sock
INFO[2024-06-14T11:00:56Z] main.go:132 tools cache dir: /root/.cache/package-scanner-1943145639
INFO[2024-06-14T11:00:56Z] main.go:133 tools paths: /root/.cache/package-scanner-1943145639/syft /root/.cache/package-scanner-1943145639/grype /root/.cache/package-scanner-1943145639/grype.yaml
ERRO[2024-06-14T11:00:56Z] utils.go:46 cmd: podman --remote --url unix:///run/podman/podman.sock ps
ERRO[2024-06-14T11:00:56Z] utils.go:47 exec: "podman": executable file not found in $PATH
WARN[2024-06-14T11:00:56Z] autodetect.go:256 podman ps:exec: "podman": executable file not found in $PATH:
WARN[2024-06-14T11:01:04Z] main.go:178 error detecting container runtime: could not detect container runtime
WARN[2024-06-14T11:01:04Z] main.go:227 unsupported container runtime
FATA[2024-06-14T11:01:04Z] run-once.go:24 error: source is required
Do we expecting podman to be there on any agent machine
@ramanan-ravi I am trying to scan my Linux VM which is CentOS during that I found these issue after the agent is been setup. The one which you gave seems to be for the docker setup.
I am trying to scan my CentOS 7.5 VM for vulnerability scan after the agent installed by following below url https://community.deepfence.io/threatmapper/docs/sensors/linux-host
Then the service started the linux VM agent is not updating to the deepfencer threadmapper
When I tried in the Linux VM manually the
./package-scanner
INFO[2024-06-14T13:51:16Z] main.go:132 tools cache dir: /home/keerthi/.cache/package-scanner-3550680636
INFO[2024-06-14T13:51:16Z] main.go:133 tools paths: /home/keerthi/.cache/package-scanner-3550680636/syft /home/keerthi/.cache/package-scanner-3550680636/grype /home/keerthi/.cache/package-scanner-3550680636/grype.yaml
ERRO[2024-06-14T13:51:16Z] utils.go:46 cmd: /usr/bin/podman --remote --url unix:///run/podman/podman.sock ps
ERRO[2024-06-14T13:51:16Z] utils.go:47 exit status 125
WARN[2024-06-14T13:51:16Z] autodetect.go:256 podman ps:exit status 125: Error: unknown flag: --remote
WARN[2024-06-14T13:51:24Z] main.go:178 error detecting container runtime: could not detect container runtime
WARN[2024-06-14T13:51:24Z] main.go:227 unsupported container runtime
FATA[2024-06-14T13:51:24Z] run-once.go:24 error: source is required
/opt/deepfence/df-agents/v4-services-vue-box/home/deepfence/bin/package-scanner -socket-path /opt/deepfence/df-agents/v4-services-vue-box/tmp/package-scanner.sock INFO[2024-06-14T11:00:56Z] main.go:132 tools cache dir: /root/.cache/package-scanner-1943145639 INFO[2024-06-14T11:00:56Z] main.go:133 tools paths: /root/.cache/package-scanner-1943145639/syft /root/.cache/package-scanner-1943145639/grype /root/.cache/package-scanner-1943145639/grype.yaml ERRO[2024-06-14T11:00:56Z] utils.go:46 cmd: podman --remote --url unix:///run/podman/podman.sock ps ERRO[2024-06-14T11:00:56Z] utils.go:47 exec: "podman": executable file not found in $PATH WARN[2024-06-14T11:00:56Z] autodetect.go:256 podman ps:exec: "podman": executable file not found in $PATH: WARN[2024-06-14T11:01:04Z] main.go:178 error detecting container runtime: could not detect container runtime WARN[2024-06-14T11:01:04Z] main.go:227 unsupported container runtime FATA[2024-06-14T11:01:04Z] run-once.go:24 error: source is required
Do we expecting podman to be there on any agent machine
No podman is not expected! It's just a warning, you can ignore. Package Scanner runs a standby service that receives calls from other agent services when to scan.
@ramanan-ravi I am trying to scan my Linux VM which is CentOS during that I found these issue after the agent is been setup. The one which you gave seems to be for the docker setup.
I am trying to scan my CentOS 7.5 VM for vulnerability scan after the agent installed by following below url https://community.deepfence.io/threatmapper/docs/sensors/linux-host
Then the service started the linux VM agent is not updating to the deepfencer threadmapper
You have followed the right docs to install agents on VM. Can you confirm if you are able to see the VM in topology after you have installed Deepfence binaries in the VM?
dial tcp: lookup "do-scan.xxx.net": no such host
Looking at this, there seems to be extra double quotes in URL and PORT. Please check.
@ibreakthecloud if the question is for me as per instruction I followed by exporting the Variables and I installed. As mention below screenshot
you mean to say that when I do export the Variables should I need to avoid the double quotes.
I hope as per your shell script of install_deepfence.sh
#!/bin/bash
# MGMT_CONSOLE_URL: Example: threatmapper.customer.com or 65.65.65.65
export MGMT_CONSOLE_URL="${MGMT_CONSOLE_URL}"
export DEEPFENCE_KEY="${DEEPFENCE_KEY}"
if [[ -z "$MGMT_CONSOLE_URL" ]]; then
echo "env MGMT_CONSOLE_URL is not set"
exit 1
fi
if [[ -z "$DEEPFENCE_KEY" ]]; then
echo "env DEEPFENCE_KEY is not set"
exit 1
fi
export MGMT_CONSOLE_PORT="443"
export MGMT_CONSOLE_URL_SCHEMA="https"
export DF_HOSTNAME="$(hostname)"
export DF_LOG_LEVEL="info"
either one of that pass as double quotes into the url that my conclusion as per screenshot that while I do export I should avoid double quotes. If in that case the UI should avoid the quotes most of the users might copy and paste as per instruction.
or should remove the quotes in shall script since we pass quotes in export
With reference to the document below I am trying to install the sensor-agent on my Rocky Linux VM https://community.deepfence.io/threatmapper/docs/sensors/linux-host
Basically it fails to install because as per the script your script fails at this point
On rocky linux they mention /etc/os-release
NAME="Rocky Linux"
VERSION="9.4 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.4 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.4"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.4"
I bypass the issue and installed by after that the startup script fails with systemctl the host shows in only when I start manually under the path
/opt/deepfence/df-agents/rocky-vue-php74/home/deepfence/start_deepfenced.sh
After that when I tried to scan vulnerability it scan and I get the result.
But when I do other scan like malware / posture / secret it fails
In the malware log I could see as mention below.
For secret scan log