vladaurosh commented 6 months ago

Is there an existing issue for this?

[X] I have searched the existing open and closed issues

Is your feature request related to a problem? Please describe

Hi @jokob-sk

I've been using pialert for some time now, "normal" installation (outside of docker - don't ask me why :D), and I am looking to move to pialert in docker. I see that current debian based image is 900MB (907MB on arm64) which is a lot. I saw in one of closed issues that previously it was around 400MB. So in last couple of days I've been working on alpine based docker image. Currently, image size is 245MB, and initially it works fine, though I haven't done much testing. Now, my idea is to have this as optional image (Dockerfile), not to replace current debian based one. There are bunch of changes that I've done (removed unneeded stuff, changed startup script), so not just change in base image, I've done some optimizations as well.

If you are interested in this, let me know, I have some questions to ask around setup.

Describe the solution you'd like

N/A

Describe alternatives you've considered

N/A

Anything else?

N/A

vladaurosh commented 6 months ago

Forgot to add, I haven't touched code itself, all changes were around setup.

jokob-sk commented 6 months ago

Hi @vladaurosh ,

That sounds interesting!

The image size jumped because I needed to support bare-metal installations: https://github.com/jokob-sk/Pi.Alert/blob/main/docs/HW_INSTALL.md - Most of the installations are based on Debian.

If the above guide still works and all previous features are preserved, then I don't see a reason why rebasing would be an issue. We need to be more rigorous when testing this as there is a lot of functionality in the app,

Suggested test cases:

Blank setup with no DB or config
Existing DB / config
Sending a notification (e. g. Delete a device and wait for a scan to run) and testing all notification gateways, especially:
Email, Apprise (e.g. via Telegram), webhook (e.g. via Discord), MQTT (e.g. via HomeAssitant)
Saving settings
Test a couple of plugins
Check the Error log for anything unusual
Testing all front-end functionality (multi-edit, functionality in the Maintenance section such as restoring of Devices.csv)
Device-specific functionality (WOL, ad-hoc scans)
UI functionality (e.g. data retrieval in the System info section)

Unsure if I can trigger an alpine build with a different flow as part of the GitHub actions.

vladaurosh commented 6 months ago

Hi @jokob-sk

Thanks for quick reply.

I'll try to cover suggested test cases, already went through some of them, but I will do again, especially I have some other changes I want to try.

Couple of questions:

Where's perl being used? I searched for it in source files but couldn't find anything. I've removed it from alpine release, saves around 40MB in final image.
I was thinking if pi user is actually needed here, except to set ownership on pialert/config and pialert/db directories. In the end python is ran as root user. You could run it as regular user but then it needs sudo for bunch of commands. Any thoughts on this?

jokob-sk commented 6 months ago

Hey @vladaurosh ,

Thanks for the effort and help!

I don't think Perl is used anywhere - those are nice savings
The pi user is a legacy thing from the original source. I'm also not sure if you would change the installation directory, as the home/pi folder is defined in a few places. Not great, I know, just had different priorities for a while. Also, please note there is a Python environment that is established during installation, but you have probably noticed already. Here we have to make sure the user mapping functionality is preserved.

I was thinking more and more about this and it's probably a good idea to rebase to alpine completely and keed an installation directory for bare-metal users that is expecting debian - that should be doable... thoughts?

Thanks again for your help

vladaurosh commented 6 months ago

Hello again @jokob-sk .

Sorry for late reply, I had busy weekend. :)

Cool, thanks for confirmation on perl not being needed anymore.
Yes, one of the first things I've tried was to run from /opt, but quickly I've realized that /home/pi was hardcoded in multiple files. That's not a big deal now.
And yes, python environment is ok, I have used a trick to save some space there and make final image cleaner.

Now, I wanted to ask you about user mapping, as I did both options, with and without user mapping. To be honest, I don't see much benefits of having user mapping. Let me explain how I did it (and this is how it's handled in some community docker images that I've used in the past):

In alpine, nginx is running as nginx:www-data (id 101, gid 82), for php-fpm you can set user:group so I've set the same. Cause of this I've set ownership of ${INSTALL_DIR}/pialert to nginx:www-data.
Now, python we run as root, so there's not issue with access there. When docker container starts, before we run php-fpm, nginx and python we set ownership of "${INSTALL_DIR}"/pialert/{config,front/log,db} to nginx:www-data (currently similar is done in start.sh script).
On docker host this will translate to same id:gid, and there's no issue with that. On my test system (docker host - debian based) I see this: uuidd 82 4096 Feb 25 15:56 pialert_config where uuidd user (uuid daemon user) has id 101 and 82 is id of group (www-data in container). So I don't even need to have that user and group on docker host.
With this we don't even need to create user in docker image and do user mapping on docker host as container will set ownership on those directories (config, front/log and db).

Also I have tightened up security, as all in /home/pi/pialert is 750 (for directories) and 640 for files (except executables which are 750). Let me know what you think about above.

As for bare-metal users, installation directory should be portable, it's more about php-fpm, nginx and python setup on target distro.

jokob-sk commented 6 months ago

Legacy configuration

Sounds good. We just need to make sure existing configurations work - so if there are no issues with removing the user mapping variables, then great.

User mapping

There have been way too many permission issues in the past which took my time to support/troubleshoot so I'd prefer to have a blanket 755 permission for the files. Better security is fine, but if it is more difficult to maintain then I rather have an easier setup and maintenance.

The user mapping script was added to prevent permission issues with DB, config, and log access. If we prevent those, then I don't have an issue removing unnecessary code and complexity. But that should be solved with proper file ownership, which should be done during startup, I think.

Bare metal install

Cool, I don't have an issue with that if the script still works, I'm fine with it. I think most users run Docker, although it's nice to give options to others.

In general, I trust you with Alpine as it sounds like you have ample experience. I really only used Debian in the past.

Thanks again for the help!

vladaurosh commented 6 months ago

Hi @jokob-sk .

Again my apologies for late reply, been pretty busy this week.

I'm under impression that those permission issues were caused by funky setup, where files were owned by user who has nothing to do with the app (doesn't run anything). And there, rw applied to others helped to overcome those issues. But here, I believe proper way is to set ownership to user/group that actually runs the application (nginx in this case). Personally, I would have this alpine image as alternative (dev) image together with current image, and then if/when you feel comfortable, we can just move to alpine image.

Now, I just wanted to share some info on the goals I had when started to work in alpine image. Primary thing was reducing the size of the image, but also I wanted to simplify and improve setup, if I can and where I can. So I went through current setup and configuration to figure out how this currently works. Some improvements I came up with:

since this is multiprocess container, I've used s6-overlay as init system, it seems to be preferred way of running containers with multiple processes. It's small enough and easy to use. You can define in which order you want to start services, by defining dependencies. First I run setup script, it is very similar to current start.sh, though it doesn't start any services. In s6 this is defined as oneshot, basically a script that runs and exits. Next, we run php-fpm, as longrun process in s6 (basically service). After php-fpm we start nginx, again type is longrun, and last we run python (pialert) app. In s6 we define setup script as dependency for php-fpm, nginx and pialert, in similar way we define php-fpm to be dependency for nginx, and lastly we define nginx to be dependency for pialert app. All this is set up during docker image build. Also, we can easily configure s6 to stop container if any of above fails, for example if nginx crashes for some reason, s6 will exit basically stopping the running container. I've tested this, and we can implement this if you feel like, for now I've left it out. With above setup, app start pretty much instantly, in a second. :) I'll paste content of setup script, which sets timezone if needed, listen address and port, and sets correct ownership on config, db and front/logs directories. First line is a bit specific, needed for s6-overlay
```
#!/usr/bin/with-contenv bash
```

echo "---------------------------------------------------------" echo "[INSTALL] Run setup.sh" echo "---------------------------------------------------------"

INSTALL_DIR=/home/pi # Specify the installation directory here

DO NOT CHANGE ANYTHING BELOW THIS LINE!

WEB_UI_DIR=/home/pi/pialert/front NGINX_CONFIG_FILE=/etc/nginx/http.d/pialert.conf OUI_FILE="/usr/share/arp-scan/ieee-oui.txt" # Define the path to ieee-oui.txt and ieee-iab.txt FILEDB="${INSTALL_DIR}/pialert/db/pialert.db"

DO NOT CHANGE ANYTHING ABOVE THIS LINE!

Check if script is run as root

if [[ $EUID -ne 0 ]]; then echo "This script must be run as root. Please use 'sudo'." exit 1 fi

echo "[INSTALL] Copy starter pialert.db and pialert.conf if they don't exist"

Copy starter pialert.db and pialert.conf if they don't exist

cp -na "${INSTALL_DIR}/pialert/back/pialert.conf" "${INSTALL_DIR}/pialert/config/pialert.conf" cp -na "${INSTALL_DIR}/pialert/back/pialert.db" "${FILEDB}"

if custom variables not set we do not need to do anything

if [ -n "${TZ}" ]; then FILECONF="${INSTALL_DIR}/pialert/config/pialert.conf" echo "[INSTALL] Setup timezone" sed -i "#^TIMEZONE=#c\TIMEZONE='${TZ}'" "${FILECONF}" fi

echo "[INSTALL] Setup NGINX" echo "Setting webserver to address ($LISTEN_ADDR) and port ($PORT)" envsubst '$PORT $LISTEN_ADDR' < "${INSTALL_DIR}/pialert/install/pialert.conf.template" > "${NGINX_CONFIG_FILE}"

Run the hardware vendors update at least once

echo "[INSTALL] Run the hardware vendors update"

Check if ieee-oui.txt or ieee-iab.txt exist

if [ -f "${OUI_FILE}" ]; then echo "The file ieee-oui.txt exists. Skipping update_vendors.sh..." else echo "The file ieee-oui.txt does not exist. Running update_vendors..."

Run the update_vendors.sh script

if [ -f "${INSTALL_DIR}/pialert/back/update_vendors.sh" ]; then "${INSTALL_DIR}/pialert/back/update_vendors.sh" else echo "update_vendors.sh script not found in ${INSTALL_DIR}." fi fi

Create an empty log files

Create the execution_queue.log and pialert_front.log files if they don't exist

touch "${INSTALL_DIR}"/pialert/front/log/{execution_queue.log,pialert_front.log}

echo "[INSTALL] Fixing permissions after copied starter config & DB" chown -R nginx:www-data "${INSTALL_DIR}"/pialert/{config,front/log,db} chmod 750 "${INSTALL_DIR}"/pialert/{config,front/log,db} chmod 640 "${INSTALL_DIR}"/pialert/{config,front/log,db}/*

Check if buildtimestamp.txt doesn't exist

if [ ! -f "${INSTALL_DIR}/pialert/front/buildtimestamp.txt" ]; then

Create buildtimestamp.txt

date +%s > "${INSTALL_DIR}/pialert/front/buildtimestamp.txt"
chown nginx:www-data "${INSTALL_DIR}/pialert/front/buildtimestamp.txt"

fi

Since this script runs first, on container start up, proper permissions and ownership are applied (recursively) to config, db and front/logs directories. 
This is docker log when container is started:

s6-rc: info: service s6rc-oneshot-runner: starting s6-rc: info: service s6rc-oneshot-runner successfully started s6-rc: info: service fix-attrs: starting s6-rc: info: service SetupOneshot: starting s6-rc: info: service fix-attrs successfully started s6-rc: info: service legacy-cont-init: starting

[INSTALL] Run setup.sh

[INSTALL] Copy starter pialert.db and pialert.conf if they don't exist [INSTALL] Setup timezone s6-rc: info: service legacy-cont-init successfully started [INSTALL] Setup NGINX Setting webserver to address (0.0.0.0) and port (20211) [INSTALL] Run the hardware vendors update The file ieee-oui.txt exists. Skipping update_vendors.sh... [INSTALL] Fixing permissions after copied starter config & DB s6-rc: info: service SetupOneshot successfully started s6-rc: info: service php-fpm: starting s6-rc: info: service php-fpm successfully started s6-rc: info: service nginx: starting s6-rc: info: service nginx successfully started s6-rc: info: service pialert: starting s6-rc: info: service pialert successfully started s6-rc: info: service legacy-services: starting [INSTALL] 🚀 Starting app - navigate to your :20211 s6-rc: info: service legacy-services successfully started


- I've modified Dockerfile as well.  It's is 2 stage build now, where in first stage we create python virtual env in /opt/venv, copy all files in /home/pi/pialert, and set permissions to 640 for files and 750 for directories and executables. 
In second stage, from first stage we copy virtual env, add it to PATH (so we don't need to activate it), install needed packages, copy application files and set correct ownership.  pre-setup.sh is small script that configures s6-overlay, initially it was part of Dockerfile, but I moved commands into separate script to make Dockerfile more readable. 
And I've added healthcheck as well, I've used this endpoint:
${LISTEN_ADDR}:${PORT}/php/server/devices.php?action=getDevicesTotals
Let me know if there's some other endpoint that would be more appropriate for a healthcheck. 
At the end we start s6 (/init).

FROM alpine:3.19 as builder

ARG INSTALL_DIR=/home/pi ENV PYTHONUNBUFFERED 1

RUN apk add --no-cache bash python3 \ && python -m venv /opt/venv

Enable venv

ENV PATH="/opt/venv/bin:$PATH"

COPY . ${INSTALL_DIR}/pialert/

RUN pip install requests paho-mqtt scapy cron-converter pytz json2table dhcp-leases pyunifi speedtest-cli chardet \ && bash -c "find ${INSTALL_DIR} -type d -exec chmod 750 {} \;" \ && bash -c "find ${INSTALL_DIR} -type f -exec chmod 640 {} \;" \ && bash -c "find ${INSTALL_DIR} -type f ( -name '.sh' -o -name '.py' -o -name 'pialert-cli' -o -name 'speedtest-cli' ) -exec chmod 750 {} \;" \ && chmod 750 ${INSTALL_DIR}/pialert/

second stage

FROM alpine:3.19 as runner

ARG INSTALL_DIR=/home/pi

COPY --from=builder /opt/venv /opt/venv

Enable venv

ENV PATH="/opt/venv/bin:$PATH"

default port and listen address

ENV PORT=20211 LISTEN_ADDR=0.0.0.0

needed for s6-overlay

ENV S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0

RUN apk update --no-cache \ && apk add --no-cache bash zip gettext-envsubst sudo shadow mtr usbutils s6-overlay \ && apk add --no-cache curl arp-scan iproute2 iproute2-ss nmap traceroute net-tools net-snmp-tools bind-tools ca-certificates \ && apk add --no-cache sqlite php82 php82-fpm php82-cgi php82-curl php82-sqlite3 php82-session \ && apk add --no-cache python3 nginx \ && bash -c "install -d -m 750 -o nginx -g www-data ${INSTALL_DIR}" \ && rm -f /etc/nginx/http.d/default.conf

COPY --from=builder --chown=nginx:www-data ${INSTALL_DIR}/pialert/ ${INSTALL_DIR}/pialert/

RUN /home/pi/pialert/dockerfiles/pre-setup.sh

HEALTHCHECK --interval=30s --timeout=5s --start-period=0s --retries=2 \ CMD curl -sf -o /dev/null ${LISTEN_ADDR}:${PORT}/php/server/devices.php?action=getDevicesTotals

ENTRYPOINT ["/init"]



Let me know what you think about above.

vladaurosh commented 6 months ago

As for test cases, here's what I've done so far:

Blank setup with no DB or config - DONE, no issues
Existing DB / config - DONE, no issues
Sending a notification (e. g. Delete a device and wait for a scan to run) and testing all notification gateways, especially: Email, Apprise (e.g. via Telegram), webhook (e.g. via Discord), MQTT (e.g. via HomeAssitant) - Email and Pushover tested, working as expected
Saving settings - DONE, no issues
Test a couple of plugins - tested Internet speedtest plugin, along with already tested pushover
Check the Error log for anything unusual - nothing unusual
Testing all front-end functionality (multi-edit, functionality in the Maintenance section such as restoring of Devices.csv) - restored devices.csv exporting from other system, worked as expected
Device-specific functionality (WOL, ad-hoc scans) - haven't tested WOL, tried nmap scan and it worked fine
UI functionality (e.g. data retrieval in the System info section) - System info page looks good

jokob-sk commented 6 months ago

This looks amazing @vladaurosh.

For the health check - not sure if a locked database would cause issues. Maybe <url>/api/app_state.json would be better?

I couldn't spot any glaring issues, but I didn't step through the scripts line by line.

How would you like to proceed next? Do you want to submit a PR?

jokob-sk commented 6 months ago

And no rush - we are on no deadlines here :) - thanks a lot for the help 🙏

vladaurosh commented 6 months ago

Hi @jokob-sk

I've just opened a PR.

Found couple of bugs during my testing, not alpine related, they are in latest image as well. I'll create new issue here.

vladaurosh commented 6 months ago

Ah yes, forgot to add, I've updated healtcheck to /api/app_state.json , thanks for suggestion.

vladaurosh commented 6 months ago

Another question. Are mtr and usbutils packages needed?

vladaurosh commented 6 months ago

Searched source code files for mtr, didn't find anything. Maybe it is some leftover?

jokob-sk commented 6 months ago

hey @vladaurosh ,

I think usbutils might be needed for the systeminfo page and mtr I probably used previously for debugging, however, I think I also considered enabling traceroute on devices/services in the future.

I think, for now, the size is sufficiently optimized 😅 it's already a huge improvement from the previous setup. Thanks a lot for the contribution. 🙏

The pi.alert_dev image is already re-based so feel free to test it 😄

vladaurosh commented 6 months ago

Hi there @jokob-sk

Yeah, that makes sense. Though I've just realized that alpine comes with lsusb command, as it is busybox built-in. However you can install usbutils package which installs lsusb with more features.

This is without usbutils package:

/ # lsusb --help
BusyBox v1.36.1 (2023-11-07 18:53:09 UTC) multi-call binary.

No help available

And this is with usbutils package installed:

/ # lsusb --help
Usage: lsusb [options]...
List USB devices
  -v, --verbose
      Increase verbosity (show descriptors)
  -s [[bus]:][devnum]
      Show only devices with specified device and/or
      bus numbers (in decimal)
  -d vendor:[product]
      Show only devices with the specified vendor and
      product ID numbers (in hexadecimal)
  -D device
      Selects which device lsusb will examine
  -t, --tree
      Dump the physical USB device hierarchy as a tree
  -V, --version
      Show version of program
  -h, --help
      Show usage and help

I've just tested image without usbutils package and systeminfo page looks good: usb

This removes another 4MB from final image 😄

I'll open PR, if you are ok with this package being removed. I also have couple of smaller tweaks.

jokob-sk commented 6 months ago

Cool, thanks, merged 😁

vladaurosh commented 6 months ago

Hey @jokob-sk

Hope you will be able to test new alpine image more than I did. 😅

I've tested WOL yesterday, confirmed working.

jokob-sk commented 6 months ago

Hey @vladaurosh ,

Noticed the storage section being empty after the re-base:

Maybe related to these errors:

2024/03/03 00:37:07 [error] 98#98: *4275 FastCGI sent in stderr: "info.php on line 373; PHP message: PHP Warning:  Undefined array key 14 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 16 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 18 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 20 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 22 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 24 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 26 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 28 in /home/pi/pialert/front/systeminfo.php on line 373" while reading upstream, client: 127.0.0.1, server: , request: "GET /systeminfo.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.2-fpm.sock:", host: "111.111.111.111:20211", referrer: "http://111.111.111.111:20211/devices.php"
2024/03/03 00:37:07 [error] 98#98: *4275 FastCGI sent in stderr: "; PHP message: PHP Warning:  Undefined array key 30 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 32 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 34 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 36 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 38 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 40 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 42 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 44 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 46 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 48 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 50 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 52 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 54 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 56 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 58 in /home/pi/pialert/front/systeminfo.php on line 373" while reading upstream, client: 127.0.0.1, server: , request: "GET /systeminfo.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.2-fpm.sock:", host: "111.111.111.111:20211", referrer: "http://111.111.111.111:20211/devices.php"
2024/03/03 00:37:07 [error] 98#98: *4275 FastCGI sent in stderr: "; PHP message: PHP Warning:  Undefined array key 60 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 62 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 64 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 66 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 68 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 70 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 72 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 74 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 76 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 78 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 80 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 82 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 84 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 86 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 88 in /home/pi/pialert/front/systeminfo.php on line 373" while reading upstream, client: 127.0.0.1, server: , request: "GET /systeminfo.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.2-fpm.sock:", host: "111.111.111.111:20211", referrer: "http://111.111.111.111:20211/devices.php"
2024/03/03 00:37:08 [error] 98#98: *4275 FastCGI sent in stderr: "; PHP message: PHP Warning:  Undefined array key 90 in /home/pi/pialert/front/systeminfo.php on line 373" while reading upstream, client: 127.0.0.1, server: , request: "GET /systeminfo.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.2-fpm.sock:", host: "111.111.111.111:20211", referrer: "http://111.111.111.111:20211/devices.php"
2024/03/03 01:06:26 [error] 96#96: *4510 FastCGI sent in stderr: "PHP message: PHP Warning:  Undefined array key 6 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 8 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 10 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 12 in /home/pi/pialert/fron" while reading upstream, client: 127.0.0.1, server: , request: "GET /systeminfo.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.2-fpm.sock:", host: "111.111.111.111:20211", referrer: "http://111.111.111.111:20211/settings.php"
2024/03/03 01:06:26 [error] 96#96: *4510 FastCGI sent in stderr: "t/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 14 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 16 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 18 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 20 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 22 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 24 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 26 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 28 in /home/pi/pialert/front/systeminfo.php on line 373" while reading upstream, client: 127.0.0.1, server: , request: "GET /systeminfo.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.2-fpm.sock:", host: "111.111.111.111:20211", referrer: "http://111.111.111.111:20211/settings.php"
2024/03/03 01:06:26 [error] 96#96: *4510 FastCGI sent in stderr: "; PHP message: PHP Warning:  Undefined array key 30 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 32 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 34 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 36 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 38 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 40 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 42 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 44 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 46 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 48 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 50 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 52 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 54 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 56 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 58 in /home/pi/pialert/front/systeminfo.php on line 373" while reading upstream, client: 127.0.0.1, server: , request: "GET /systeminfo.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.2-fpm.sock:", host: "111.111.111.111:20211", referrer: "http://111.111.111.111:20211/settings.php"
2024/03/03 01:06:26 [error] 96#96: *4510 FastCGI sent in stderr: "; PHP message: PHP Warning:  Undefined array key 60 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 62 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 64 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 66 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 68 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 70 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 72 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 74 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 76 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 78 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 80 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 82 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 84 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 86 in /home/pi/pialert/front/systeminfo.php on line 373; PHP message: PHP Warning:  Undefined array key 88 in /home/pi/pialert/front/systeminfo.php on line 373" while reading upstream, client: 127.0.0.1, server: , request: "GET /systeminfo.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.2-fpm.sock:", host: "111.111.111.111:20211", referrer: "http://111.111.111.111:20211/settings.php"
2024/03/03 01:06:27 [error] 96#96: *4510 FastCGI sent in stderr: "; PHP message: PHP Warning:  Undefined array key 90 in /home/pi/pialert/front/systeminfo.php on line 373" while reading upstream, client: 127.0.0.1, server: , request: "GET /systeminfo.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.2-fpm.sock:", host: "111.111.111.111:20211", referrer: "http://111.111.111.111:20211/settings.php"

maybe the df utility is not available?

//HDD stats
$hdd_result = shell_exec("df | awk '{print $1}'");
$hdd_devices = explode("\n", trim($hdd_result));
$hdd_result = shell_exec("df | awk '{print $2}'");
$hdd_devices_total = explode("\n", trim($hdd_result));
$hdd_result = shell_exec("df | awk '{print $3}'");
$hdd_devices_used = explode("\n", trim($hdd_result));
$hdd_result = shell_exec("df | awk '{print $4}'");
$hdd_devices_free = explode("\n", trim($hdd_result));
$hdd_result = shell_exec("df | awk '{print $5}'");
$hdd_devices_percent = explode("\n", trim($hdd_result));
$hdd_result = shell_exec("df | awk '{print $6}'");
$hdd_devices_mount = explode("\n", trim($hdd_result));

vladaurosh commented 6 months ago

Hi @jokob-sk

Getting this on my end

I'd say it looks good.

Just tried "df | awk '{print $1}'" in container itself, getting expected result:

Filesystem
overlay
tmpfs
shm
/dev/nvme0n1p1
/dev/nvme0n1p1
/dev/nvme0n1p1
/dev/nvme0n1p1
/dev/nvme0n1p1
tmpfs
tmpfs
tmpfs
tmpfs
tmpfs
tmpfs

df output alone looks good as well:

Filesystem           1K-blocks      Used Available Use% Mounted on
overlay              123076904  11520224 106532692  10% /
tmpfs                    65536         0     65536   0% /dev
shm                      65536         0     65536   0% /dev/shm
/dev/nvme0n1p1       123076904  11520224 106532692  10% /etc/resolv.conf
/dev/nvme0n1p1       123076904  11520224 106532692  10% /etc/hostname
/dev/nvme0n1p1       123076904  11520224 106532692  10% /etc/hosts
/dev/nvme0n1p1       123076904  11520224 106532692  10% /home/pi/pialert/db
/dev/nvme0n1p1       123076904  11520224 106532692  10% /home/pi/pialert/config
tmpfs                  3936456         0   3936456   0% /proc/asound
tmpfs                    65536         0     65536   0% /proc/keys
tmpfs                    65536         0     65536   0% /proc/timer_list
tmpfs                    65536         0     65536   0% /proc/sched_debug
tmpfs                  3936456         0   3936456   0% /proc/scsi
tmpfs                  3936456         0   3936456   0% /sys/firmware

vladaurosh commented 6 months ago

Ah I see @jokob-sk

Just tried on other system, and got this when executing df:

/ # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  26.0G     19.4G      6.6G  75% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                     1.9G         0      1.9G   0% /sys/fs/cgroup
shm                      64.0M         0     64.0M   0% /dev/shm
/dev/mapper/centos_centos-root
                         26.0G     19.4G      6.6G  75% /etc/resolv.conf
/dev/mapper/centos_centos-root
                         26.0G     19.4G      6.6G  75% /etc/hostname
/dev/mapper/centos_centos-root
                         26.0G     19.4G      6.6G  75% /etc/hosts
tmpfs                     1.9G         0      1.9G   0% /proc/acpi
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/timer_stats
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                     1.9G         0      1.9G   0% /proc/scsi
tmpfs                     1.9G         0      1.9G   0% /sys/firmw

And "df | awk '{print $1}'" :

Filesystem
overlay
tmpfs
tmpfs
shm
/dev/mapper/centos_centos-root
27245572
/dev/mapper/centos_centos-root
27245572
/dev/mapper/centos_centos-root
27245572
tmpfs
tmpfs
tmpfs
tmpfs
tmpfs
tmpfs
tmpfs
tmpfs

Looks like when Filesystem is longer than some number of chars rest of the lines is printed in next line.

vladaurosh commented 6 months ago

Again, df is busybox built-in and output seems to be a bit different, we can install coreutils package to get "normal" df but let me see if we can format the output somehow with busybox df.

jokob-sk commented 6 months ago

Thanks a lot in advance!

vladaurosh commented 6 months ago

Ok, we have to options:

install coreutils to get non built-in df
use df -P instead of df.

-P here is POSIX output format, and works with "normal" df command: -P, --portability use the POSIX output format

Header line when using "normal" df executable without -P: Filesystem 1K-blocks Used Available Use% Mounted on

Header line when using "normal" df executable with -P: Filesystem 1024-blocks Used Available Capacity Mounted on

Except for different column names for 1K-blocks and Use (Capacity), output itself looks exactly the same. I've tested this on Centos 7 (outside of container).

So df -P would solve this issue and is portable with debian and other distros.

Let me know how you'd want to proceed.

vladaurosh commented 6 months ago

Example of df -P output in container:

/ # df -P
Filesystem           1024-blocks    Used Available Capacity Mounted on
overlay               27245572  20366532   6879040  75% /
tmpfs                    65536         0     65536   0% /dev
tmpfs                  2022580         0   2022580   0% /sys/fs/cgroup
shm                      65536         0     65536   0% /dev/shm
/dev/mapper/centos_centos-root            27245572  20366532   6879040  75% /etc/resolv.conf
/dev/mapper/centos_centos-root            27245572  20366532   6879040  75% /etc/hostname
/dev/mapper/centos_centos-root            27245572  20366532   6879040  75% /etc/hosts
tmpfs                  2022580         0   2022580   0% /proc/acpi
tmpfs                    65536         0     65536   0% /proc/kcore
tmpfs                    65536         0     65536   0% /proc/keys
tmpfs                    65536         0     65536   0% /proc/timer_list
tmpfs                    65536         0     65536   0% /proc/timer_stats
tmpfs                    65536         0     65536   0% /proc/sched_debug
tmpfs                  2022580         0   2022580   0% /proc/scsi
tmpfs                  2022580         0   2022580   0% /sys/firmware

I've run "df -P | awk '{print $1/2/3/4/5/6}'" and output looks good.

jokob-sk commented 6 months ago

Nice, I will use the -P option 👍

Hopefully last thing to solve is lsblk -io NAME,SIZE,TYPE,MOUNTPOINT,MODEL --list

Synology-NAS:/# lsblk -io NAME,SIZE,TYPE,MOUNTPOINT,MODEL --list
bash: lsblk: command not found
Synology-NAS:/#

vladaurosh commented 6 months ago

That is lsblk package, I've just created PR for that.

jokob-sk commented 6 months ago

🙌- merged :)

jokob-sk commented 5 months ago

released -> closing

jokob-sk / NetAlertX

Alpine based image - contribution offered #577

Is there an existing issue for this?

Is your feature request related to a problem? Please describe

Describe the solution you'd like

Describe alternatives you've considered

Anything else?

DO NOT CHANGE ANYTHING BELOW THIS LINE!

DO NOT CHANGE ANYTHING ABOVE THIS LINE!

Check if script is run as root

Copy starter pialert.db and pialert.conf if they don't exist

if custom variables not set we do not need to do anything

Run the hardware vendors update at least once

Check if ieee-oui.txt or ieee-iab.txt exist

Run the update_vendors.sh script

Create an empty log files

Create the execution_queue.log and pialert_front.log files if they don't exist

Check if buildtimestamp.txt doesn't exist

Create buildtimestamp.txt

s6-rc: info: service s6rc-oneshot-runner: starting s6-rc: info: service s6rc-oneshot-runner successfully started s6-rc: info: service fix-attrs: starting s6-rc: info: service SetupOneshot: starting s6-rc: info: service fix-attrs successfully started s6-rc: info: service legacy-cont-init: starting

[INSTALL] Run setup.sh

Enable venv

second stage

Enable venv

default port and listen address

needed for s6-overlay