Open AnalogJ opened 4 years ago
I was able to run it directly (no containers) on Win10 Pro without too much hassle.
@elliottback interesting. I'm assuming you're using the windows linux subsystem with the linux-amd64
variant of my binaries?
@AnalogJ no, I installed the golang ecosystem and am running the collector directly on a local windows build. It's able to get all the attributes except power-on time for a WDC WDS200T2B0B-00YS70.
@elliottback can you share more information on how to get the collector running on Windows?
I have a scheduled task that runs:
C:\Go\bin\go.exe run .\collector-metrics.go run --api-endpoint "http://192.168.1.75:8012"
and starts in C:\scrutiny\collector\cmd\collector-metrics\
Hi. Just wondering whether there is any update on this. I'd love to see Windows support for this tool.
This seems to work fine on windows. I can confirm what ellittback did.
However a bit more detail. And this assumes you have Go Lang installed on a windows system.
Unfortunately SMART doesnt seem compatible with our enterprise PERC raid controllers XD, but for diretly attach drives it seems to work.
I created new windows binaries for the Scrutiny web
and collector
in the latest release:
https://github.com/AnalogJ/scrutiny/releases/tag/0.3.13
Can I get some of you to take a look and verify that they are working?
Hi,
It does not work.
The collector throws the following error:
ERROR: DependencyMissingError: "smartctl is missing"
The web server throws the following error:
Start the scrutiny server
time="2021-10-25T17:20:45+01:00" level=fatal msg="\x1b[91mERROR: ConfigValidationError: \"Database parent directory does not exist. Please check path (\\\\scrutiny\\\\config)\"\x1b[0m"
Hi, It does not work. The collector throws the following error:
ERROR: DependencyMissingError: "smartctl is missing"
The web server throws the following error:Start the scrutiny server time="2021-10-25T17:20:45+01:00" level=fatal msg="\x1b[91mERROR: ConfigValidationError: \"Database parent directory does not exist. Please check path (\\\\scrutiny\\\\config)\"\x1b[0m"
Did you install smartmontools for windows?
Yes, but apparently I should have also restarted the computer after installing it. The collector now starts, but it gives the following errors:
time="2021-10-25T17:52:17+01:00" level=info msg="Verifying required tools" type=metrics
time="2021-10-25T17:52:17+01:00" level=info msg="Executing command: smartctl --scan -j" type=metrics
time="2021-10-25T17:52:17+01:00" level=info msg="Executing command: smartctl --info -j -d sat /dev/sdg" type=metrics
time="2021-10-25T17:52:17+01:00" level=error msg="Could not retrieve device information for /dev/sdg: exit status 2" type=metrics
time="2021-10-25T17:52:17+01:00" level=info msg="Executing command: smartctl --info -j -d sat /dev/sdl" type=metrics
time="2021-10-25T17:52:17+01:00" level=error msg="Could not retrieve device information for /dev/sdl: exit status 2" type=metrics
time="2021-10-25T17:52:17+01:00" level=info msg="Executing command: smartctl --info -j -d sat /dev/sdm" type=metrics
time="2021-10-25T17:52:18+01:00" level=info msg="Collecting smartctl results for /dev/sdg\n" type=metrics
time="2021-10-25T17:52:18+01:00" level=info msg="Executing command: smartctl -x -j -d sat /dev/sdg" type=metrics
time="2021-10-25T17:52:19+01:00" level=error msg="smartctl returned an error code (2) while processing /dev/sdg\n" type=metrics
time="2021-10-25T17:52:19+01:00" level=error msg="smartctl could not open device" type=metrics
time="2021-10-25T17:52:19+01:00" level=info msg="Publishing smartctl results for \n" type=metrics
time="2021-10-25T17:52:19+01:00" level=info msg="Collecting smartctl results for /dev/sdl\n" type=metrics
time="2021-10-25T17:52:19+01:00" level=info msg="Executing command: smartctl -x -j -d sat /dev/sdl" type=metrics
time="2021-10-25T17:52:19+01:00" level=error msg="smartctl returned an error code (2) while processing /dev/sdl\n" type=metrics
time="2021-10-25T17:52:19+01:00" level=error msg="smartctl could not open device" type=metrics
time="2021-10-25T17:52:19+01:00" level=info msg="Publishing smartctl results for \n" type=metrics
time="2021-10-25T17:52:19+01:00" level=info msg="Collecting smartctl results for /dev/sdm\n" type=metrics
time="2021-10-25T17:52:19+01:00" level=info msg="Executing command: smartctl -x -j -d sat /dev/sdm" type=metrics
time="2021-10-25T17:52:19+01:00" level=error msg="smartctl returned an error code (2) while processing /dev/sdm\n" type=metrics
time="2021-10-25T17:52:19+01:00" level=error msg="smartctl could not open device" type=metrics
time="2021-10-25T17:52:19+01:00" level=info msg="Publishing smartctl results for \n" type=metrics
time="2021-10-25T17:52:19+01:00" level=info msg="Collecting smartctl results for /dev/sdn\n" type=metrics
time="2021-10-25T17:52:19+01:00" level=info msg="Executing command: smartctl -x -j -d sat /dev/sdn" type=metrics
time="2021-10-25T17:52:19+01:00" level=error msg="smartctl returned an error code (2) while processing /dev/sdn\n" type=metrics
time="2021-10-25T17:52:19+01:00" level=error msg="smartctl could not open device" type=metrics
time="2021-10-25T17:52:19+01:00" level=info msg="Publishing smartctl results for \n" type=metrics
time="2021-10-25T17:52:19+01:00" level=info msg="Main: Completed" type=metrics
The web server now works (I didn't use the right file structure initially), but the web-page just says "No Devices Detected!" before proceeding to show an empty dashboard.
Fixed it. The anti-ransomware protection in Windows Defender was blocking smartctl.exe because it was trying to access "protected memory". It was sufficient to just go to Defender and "allow" the app the run.
The dashboard now loads all the drives. Some problems still remain:
Some more info: The device I said that appears blank is actually appearing twice. Once it appears normal with name and info and the other time blank with only /dev/sde shown; both devices have /dev/sde. No other sdX appears twice.
Hey @gabrielv33 Thanks for confirming that the Windows binaries work.
To solve your remaining issues, can you run the following commands and attach the outputs.
smartctl --scan --json
# The output of the above command will include a list of devices:
# {
# "name": "/dev/sda",
# "info_name": "/dev/sda",
# "type": "scsi",
# "protocol": "SCSI"
# },
# for your misbehaving devices, please run the following command,
# replacing ${DEVICE_ID} with the device name, eg. `smartctl -x --json /dev/sda`
smartctl -x --json ${DEVICE_ID}
Hi, is there a way to make scrutiny work on windows via docker? Thanks!
Hey @Issam2204
Unfortunately I don't have a windows machine, so it's a bit difficult to develop windows compatible Docker images.
Having said that, I don't see any reason why it wouldn't work. The web
and collector
binaries could definitely run in Windows.
The omnibus
image would be a bit more difficult, as I'm not sure if Windows Docker images have a service manager similar to "s6-overlay". This is required because the omnibus
image runs the webservice, the collector and Influxdb processes all at the same time.
I'm happy to review PRs and provide feedback if someone is interested in picking this up.
I’ve been attempting to get the collector working in Windows 10 with Docker Desktop installed using WSL2 but I can’t find a way to get the device mapping to work. The Windows drives are mounted at /mnt/c
/mnt/d
etc. but they are not available as block devices from /dev/
. In theory there might be a way to get this working under Windows 11 as per https://docs.microsoft.com/en-us/windows/wsl/wsl2-mount-disk but I’m not yet ready to make the move to Win 11….
I’ve been attempting to get the collector working in Windows 10 with Docker Desktop installed using WSL2 but I can’t find a way to get the device mapping to work. The Windows drives are mounted at
/mnt/c
/mnt/d
etc. but they are not available as block devices from/dev/
. In theory there might be a way to get this working under Windows 11 as per https://docs.microsoft.com/en-us/windows/wsl/wsl2-mount-disk but I’m not yet ready to make the move to Win 11….
So this is tricky... WSL2 is actually a very small and special hyperv virtual machine. So while the drives are "mounted" there. I do not believe they are the actual disk are. You could run the collector/server piece in a WSL2 docker container, and you could compile the golang client piece and run that on windows as mentioned above. This might work. However I do not believe that the WSL2 can actually see the disk devices... only the mounts.
@jaytay79 are you trying to create a native Windows (Core/Nano) docker image for the Scrutiny Collector? Or are you running the linux based Scrutiny collector docker image using Docker For Windows (which runs in a VM IIRC)?
I was trying the Linux collector container running in Windows Docker using WSL2. As I mentioned I believe that in Windows 11 you would be able to mount devices to /dev/sd* and if so then the collector container should work (well, in theory anyway) but unfortunately the --mount
option has not been added to WSL in Windows 10 yet - https://github.com/microsoft/WSL/issues/7940
@jaytay79 / @icsy7867 / anyone else
if you're willing to be testers, I can throw together a quick Windows Nano/Core based docker image and we can see if that works? We'll probably want to do this live, so I'll need you to jump on Discord with me: Selfhosted Discord - https://selfhosted.show/discord - #storage
channel
@jaytay79 / @icsy7867 / anyone else
if you're willing to be testers, I can throw together a quick Windows Nano/Core based docker image and we can see if that works? We'll probably want to do this live, so I'll need you to jump on Discord with me: Selfhosted Discord - https://selfhosted.show/discord -
#storage
channel
Would definitely be willing. But might be a couple weeks before I'll have time. Out of town this weekend, short week and out of town again.... But probably available after that if needed.
please try this native Windows Collector docker image when you get a chance:
docker pull ghcr.io/analogj/scrutiny_windows_test:latest
The image source code lives here: https://github.com/AnalogJ/scrutiny_windows_test
Once we verify its working, I can move it over to the main repo.
I can confirm that it runs (after switching Docker to Windows container mode) but I have no clue how I can get the drives/devices mapped or if that's even possible with Windows Docker images. I’ll try and have a further look when I have some more free time.
One more thing to note is that the image is pretty big - over 5GB!
@jaytay79 great, can you try to pass in devices using the instructions here?
You might also want to try passing in the --privileged
flag
Unfortunately there's a couple of issues: adding --isolation=process
results in a message that The container operating system does not match the host operating system
(I'm not sure why yet) and reading through the page I see that there are only 5 supported device types and DiskDrive devices are not included on the list. I'm beginning to think this may be a brick wall....
I think the process isolation error is because I only have one flavor of windows docker image -- based on windows-2019
. If your kernel is incompatible, it wont be able to run with process isolation (sharing the same windows kernel between host and container).
Which version of windows are you running?
I created 2 separate images for windows for testing.
docker pull ghcr.io/analogj/scrutiny_windows_test:ltsc2019
docker pull ghcr.io/analogj/scrutiny_windows_test:ltsc2022
Sorry, I’ve been rather busy the last couple of days so I haven’t had a chance to do much with this. I’m getting manifest unknown on both of those? I’m running Windows 10 Pro 21H2.
apologies @jaytay79 the build failed. I've fixed it, so you can try again :)
Ok so I can only use ltsc2019 (a Windows version 10.0.20348-based image is incompatible with a 10.0.19044 host
) but unfortunately I get the same message that The container operating system does not match the host operating system.
so I'm really not sure what else I can try at this point 😞
I can try building on some additional windows base images. Do you see anything that may match your system better in the Full Tag Listing
list from the link below? @jaytay79
https://hub.docker.com/_/microsoft-windows-servercore?tab=description
I had some time to fiddle around with a few things. My desktop I am testing on is 21H2, and apparently this has issues with the isolation mode requirement. https://github.com/microsoft/Windows-Containers/issues/163
However, I could successfully run:
docker run -itd ghcr.io/analogj/scrutiny_windows_test:ltsc2019
Also, it seems as though no one has tried to pass through an entire disk to a windows container... However I found a couple useful links: https://docs.microsoft.com/en-us/windows-hardware/drivers/install/system-defined-device-setup-classes-available-to-vendors https://www.lifewire.com/device-class-guids-for-most-common-types-of-hardware-2619208 You can find some of the windows GUID and class information for things like the SCSI drivers and whatnot. However I cannot currently use the --device flag and isolation mode presently.
However, if you could provide the compiled .exe, there is no reason someone couldnt run a linux web server, and use the collector.exe with task scheduler on windows. Still trying a few things though.
@icsy7867 oh, if you're just looking for a windows collector.exe file, that's already generated automatically and attached to each release.
Look here: https://github.com/AnalogJ/scrutiny/releases/latest for scrutiny-collector-metrics-windows-4.0-amd64.exe
@icsy7867 oh, if you're just looking for a windows collector.exe file, that's already generated automatically and attached to each release.
Look here: https://github.com/AnalogJ/scrutiny/releases/latest for
scrutiny-collector-metrics-windows-4.0-amd64.exe
No, I don't need it. I was just suggesting that anyone in this position could easily just run the collector.exe via task manager in windows and point it to a linux container running the web server.
I dont have this need, but was just giving my input trying to run your windows container.
No longer working on Windows with the latest updates. I'm getting the following when trying to run the web executable:
Start the scrutiny server
2022/06/09 02:56:44 Loading configuration file: C:\scrutiny\config\scrutiny.yaml
time="2022-06-09T02:56:44+01:00" level=info msg="Trying to connect to scrutiny sqlite db: C:\\scrutiny\\config\\scrutiny.db\n"
time="2022-06-09T02:56:44+01:00" level=info msg="Successfully connected to scrutiny sqlite db: C:\\scrutiny\\config\\scrutiny.db\n"
panic: failed to check influxdb setup status - Get "http://localhost:8086/api/v2/setup": dial tcp [::1]:8086: connectex: No connection could be made because the target machine actively refused it.
@gabrielv33 can you open a new issue for that? This thread is mostly focused on Windows Docker support now.
I'm not all too familiar with Docker and having trouble fixing the issue I am getting while trying to run the test windows image from this thread. I wasn't having this issue when I was running the linux container and it only happens when running the command within Window's containers
Command:
docker run -it --rm -p 8080:8080 -p 8086:8086 \
-v ./scrutiny:/opt/scrutiny/config \
-v ./influxdb2:/opt/scrutiny/influxdb \
-v /run/udev:/run/udev:ro \
--cap-add SYS_RAWIO \
--device=/dev/sda --device=/dev/sdb \
--name scrutiny ghcr.io/analogj/scrutiny_windows_test:ltsc2019
Error:
docker: Error response from daemon: invalid volume specification: 'E:\Documents\scrutiny:\opt\scrutiny\config'. See 'docker run --help'.
Also my E drive is not my main drive if that matters. It's my secondary hard drive. Though again, I had no issue when running Scrutiny within the linux containers.
Weird, not sure why docker is complaining about Windows paths. I guess you could try setting it up with a docker volume
instead of a bind mount. I'm not on windows so it's a bit hard for me to debug :(
To solve your remaining issues, can you run the following commands and attach the outputs.
smartctl --scan --json # The output of the above command will include a list of devices: # { # "name": "/dev/sda", # "info_name": "/dev/sda", # "type": "scsi", # "protocol": "SCSI" # }, # for your misbehaving devices, please run the following command, # replacing ${DEVICE_ID} with the device name, eg. `smartctl -x --json /dev/sda` smartctl -x --json ${DEVICE_ID}
I'm trying to get this working on Windows Server 2019, I installed smartmontools and the web server is running but with no devices listed. I tried the above code and got output for my device (/dev/sda). I also followed the instructions from the readme and I keep getting this output:
PS` C:> docker exec scrutiny-collector-1 /opt/scrutiny/bin/scrutiny-collector-metrics run
2022/08/10 02:25:40 No configuration file found at /opt/scrutiny/config/collector.yaml. Using Defaults.
___ ___ ____ __ __ ____ ____ _ _ _ _
/ __) / __)( _ \( )( )(_ _)(_ _)( \( )( \/ )
\__ \( (__ ) / )(__)( )( _)(_ ) ( \ /
(___/ \___)(_)\_)(______) (__) (____)(_)\_) (__)
AnalogJ/scrutiny/metrics dev-0.5.0
time="2022-08-10T02:25:40Z" level=info msg="Verifying required tools" type=metrics
time="2022-08-10T02:25:40Z" level=info msg="Executing command: smartctl --scan --json" type=metrics
time="2022-08-10T02:25:40Z" level=info msg="Executing command: smartctl --info --json /dev/sda" type=metrics
time="2022-08-10T02:25:40Z" level=error msg="Could not retrieve device information for sda: exit status 2" type=metrics
time="2022-08-10T02:25:40Z" level=info msg="Sending detected devices to API, for filtering & validation" type=metrics
time="2022-08-10T02:25:40Z" level=info msg="Main: Completed" type=metrics
PS C:>
I saw referenced above about allowing it through Windows Defender, but I am not getting any notifications regarding smartctl being blocked.
I had some time to fiddle around with a few things. My desktop I am testing on is 21H2, and apparently this has issues with the isolation mode requirement. microsoft/Windows-Containers#163
However, I could successfully run:
docker run -itd ghcr.io/analogj/scrutiny_windows_test:ltsc2019
Also, it seems as though no one has tried to pass through an entire disk to a windows container... However I found a couple useful links: https://docs.microsoft.com/en-us/windows-hardware/drivers/install/system-defined-device-setup-classes-available-to-vendors https://www.lifewire.com/device-class-guids-for-most-common-types-of-hardware-2619208 You can find some of the windows GUID and class information for things like the SCSI drivers and whatnot. However I cannot currently use the --device flag and isolation mode presently.
However, if you could provide the compiled .exe, there is no reason someone couldnt run a linux web server, and use the collector.exe with task scheduler on windows. Still trying a few things though.
Just wanted to say that i was able to get scrutiny working using this method. I'm currently running docker using WSL2 (thus was unable to use the omnibus image, no drives shown) and running just the influxdb
and web
image in conjunction with a scheduled task of the windows collector binary on my host machine, got the dashboard working as desired (no errors and correct information shown).
For anyone else having issues with running Scrutiny on docker using WSL2 on windows:
scrutiny-collector-metrics-windows-amd64.exe
from the releases page (under assets)action
tab), click Create Task...
General
Tab open, enter relevant information into the Name
and Description
fields
Triggers
tab and then click New...
(bottom left-hand side of the window)Daily
, start at TODAYS_DATE
12:00:00 AM
, Recur every 1
days,1 hour
for a duration of Indefinitely
30 minutes
Actions
tab and then click New...
(bottom left-hand side of the window)scrutiny-collector-metrics-windows-amd64.exe
run --api-endpoint "http://localhost:8080" --config collector.yaml
--config
param is optional is not needed if you just want to use the default collector config, see example.collector.yaml for more info on the collector config.scrutiny-collector-metrics-windows-amd64.exe
file
scrutiny-collector-metrics-windows-amd64.exe
in the pathConditions
tab and make sure that everything is unchecked (unless you want to specify otherwise)Settings
tab and check everything except for the last checkbox
5 minutes
3
times1 hour
Run
)Hope this helps anyone who comes across this thread.
@Robert-Zacchigna Thanks! this looks great 🥳
I'll go ahead and create a docs/INSTALL_MANUAL_WINDOWS.md
guide based on your steps.
Thanks again!
@Robert-Zacchigna Thanks! this looks great 🥳 I'll go ahead and create a
docs/INSTALL_MANUAL_WINDOWS.md
guide based on your steps.Thanks again!
Glad you find it useful, i can create the doc myself and open a pull request if you want. I'll do my best to follow the current doc format you have or if you would rather just do it yourself, either way is fine. Thanks.
@Robert-Zacchigna sorry I got distracted with some other work. If you could open a PR to add a docs/INSTALL_MANUAL_WINDOWS.md
guide, that would be greatly appreciated!
@AnalogJ Here you are, review at your leisure: https://github.com/AnalogJ/scrutiny/pull/372
I had some time to fiddle around with a few things. My desktop I am testing on is 21H2, and apparently this has issues with the isolation mode requirement. microsoft/Windows-Containers#163 However, I could successfully run:
docker run -itd ghcr.io/analogj/scrutiny_windows_test:ltsc2019
Also, it seems as though no one has tried to pass through an entire disk to a windows container... However I found a couple useful links: docs.microsoft.com/en-us/windows-hardware/drivers/install/system-defined-device-setup-classes-available-to-vendors lifewire.com/device-class-guids-for-most-common-types-of-hardware-2619208 You can find some of the windows GUID and class information for things like the SCSI drivers and whatnot. However I cannot currently use the --device flag and isolation mode presently. However, if you could provide the compiled .exe, there is no reason someone couldnt run a linux web server, and use the collector.exe with task scheduler on windows. Still trying a few things though.Just wanted to say that i was able to get scrutiny working using this method. I'm currently running docker using WSL2 (thus was unable to use the omnibus image, no drives shown) and running just the
influxdb
andweb
image in conjunction with a scheduled task of the windows collector binary on my host machine, got the dashboard working as desired (no errors and correct information shown).For anyone else having issues with running Scrutiny on docker using WSL2 on windows:
- Copy the influxdb and web sections of the example.hubspoke.docker-compose.yml
- Download the latest
scrutiny-collector-metrics-windows-amd64.exe
from the releases page (under assets)- docker-compose the file as normal to create the containers (your dashboard should be blank)
- On your windows host, open the Windows Task Scheduler as Administrator
- On the status bar (under the
action
tab), clickCreate Task...
- A new window should open with the
General
Tab open, enter relevant information into theName
andDescription
fieldsUnder Security Options check:
- Run whether user is logged on or not
- Run with highest privileges
- Next, click the
Triggers
tab and then clickNew...
(bottom left-hand side of the window)Here you can set how often you want this task to run, my settings are the following:
Settings:
Daily
, start atTODAYS_DATE
12:00:00 AM
, Recur every1
days,Advanced Settings:
- Repeat Task every:
1 hour
for a duration ofIndefinitely
- Stop task if it runs longer than:
30 minutes
- The above settings will trigger the task every day at midnight and run every hour after that (modify as needed)
- Click Ok when satisfied with your schedule
- Next, click the
Actions
tab and then clickNew...
(bottom left-hand side of the window)Action Settings:
- In the Program/Script field, put:
scrutiny-collector-metrics-windows-amd64.exe
In the Add arguments (optional) field, put:
run --api-endpoint "http://localhost:8080" --config collector.yaml
- NOTE: Make sure that you put the correct port number (as specified in the docker-compose file) for the dashboard (default is 8080)
- NOTE: The
--config
param is optional is not needed if you just want to use the default collector config, see example.collector.yaml for more info on the collector config.In the Start in (optional) field, put: FOLDER_PATH_TO_YOUR
scrutiny-collector-metrics-windows-amd64.exe
file
- NOTE: Must be exact and do not include
scrutiny-collector-metrics-windows-amd64.exe
in the path- Click Ok when finished
- Next, click the
Conditions
tab and make sure that everything is unchecked (unless you want to specify otherwise)Next, click the
Settings
tab and check everything except for the last checkbox
My limits for the following settings:
If the task fails, restart every:
5 minutes
Attempt restart up to:
3
timesStop the task if it runs longer than
1 hour
- Next, once satisfied with everything, click Ok
- Then, find your newly created task (by its name) and then manually run it (right click it and then click
Run
)- Finally, refresh your dashboard after a minute or two and your drive information should have populated the dashboard.
Hope this helps anyone who comes across this thread.
I have Unfortunately been unable to run this on my windows 11 machine with wsl. I am using the spoke method and I have both home assistant and Proxmox showing up but windows 11 just refuse.
Plan for eventual windows support.
API
Collector
Docker