dlandon / unassigned.devices

Unassigned Devices plugin for unRAID
Other
81 stars 39 forks source link

Unassigned devices causing high CPU with hosts_port_ping.sh #97

Closed ghTravis closed 7 months ago

ghTravis commented 7 months ago

Hi,

I recently was trying to debug a high CPU problem wiht my unraid setup. I have a fairly minimal installation, maybe around 10 docker containers but I just recently setup my server about 3 weeks ago, so nothing really crazy outside a few of the official plugins installed.

I noticed that there's a script that's taking up 80-90% CPU coming from the unassigned.devices plugin. See below output from ps auxf

root      6124  0.0  0.0  92772  9720 ?        Ss   Jan01   0:51 php-fpm: master process (/etc/php-fpm.conf)
root     29498  0.0  0.0  98040 16880 ?        S    Jan06   0:00  \_ php-fpm: pool www
root     32743 83.4  0.0  16444 15600 ?        S    Jan06 2005:24  |   \_ /bin/bash plugins/unassigned.devices/scripts/hosts_port_ping.sh 10.2.0.15  445
root     10878  0.0  0.0   2584   872 ?        S    11:50   0:00  |       \_ timeout -s 5 5 bash -c (echo >/dev/tcp/2.98.11.199/445) &>/dev/null && echo 2.98.11.199
root     10881  0.0  0.0   3980  2940 ?        S    11:50   0:00  |       |   \_ bash -c (echo >/dev/tcp/2.98.11.199/445) &>/dev/null && echo 2.98.11.199
root     10884  0.0  0.0   3980   296 ?        S    11:50   0:00  |       |       \_ bash -c (echo >/dev/tcp/2.98.11.199/445) &>/dev/null && echo 2.98.11.199
root     10883  0.0  0.0   2584   928 ?        S    11:50   0:00  |       \_ timeout -s 5 5 bash -c (echo >/dev/tcp/2.98.11.200/445) &>/dev/null && echo 2.98.11.200
root     10886  0.0  0.0   3980  2924 ?        S    11:50   0:00  |       |   \_ bash -c (echo >/dev/tcp/2.98.11.200/445) &>/dev/null && echo 2.98.11.200
root     10887  0.0  0.0   3980   292 ?        S    11:50   0:00  |       |       \_ bash -c (echo >/dev/tcp/2.98.11.200/445) &>/dev/null && echo 2.98.11.200
root     10888  0.0  0.0   2584   828 ?        S    11:50   0:00  |       \_ timeout -s 5 5 bash -c (echo >/dev/tcp/2.98.11.202/445) &>/dev/null && echo 2.98.11.202
root     10889  0.0  0.0   3980  2880 ?        S    11:50   0:00  |       |   \_ bash -c (echo >/dev/tcp/2.98.11.202/445) &>/dev/null && echo 2.98.11.202
root     10891  0.0  0.0   3980   292 ?        S    11:50   0:00  |       |       \_ bash -c (echo >/dev/tcp/2.98.11.202/445) &>/dev/null && echo 2.98.11.202
root     10890  0.0  0.0   2584   924 ?        S    11:50   0:00  |       \_ timeout -s 5 5 bash -c (echo >/dev/tcp/2.98.11.203/445) &>/dev/null && echo 2.98.11.203
root     10893  0.0  0.0   3980  2884 ?        S    11:50   0:00  |       |   \_ bash -c (echo >/dev/tcp/2.98.11.203/445) &>/dev/null && echo 2.98.11.203
root     10897  0.0  0.0   3980   296 ?        S    11:50   0:00  |       |       \_ bash -c (echo >/dev/tcp/2.98.11.203/445) &>/dev/null && echo 2.98.11.203
root     10895  0.0  0.0   2584   868 ?        S    11:50   0:00  |       \_ timeout -s 5 5 bash -c (echo >/dev/tcp/2.98.11.205/445) &>/dev/null && echo 2.98.11.205
root     10899  0.0  0.0   3980  2936 ?        S    11:50   0:00  |       |   \_ bash -c (echo >/dev/tcp/2.98.11.205/445) &>/dev/null && echo 2.98.11.205
root     10908  0.0  0.0   3980   292 ?        S    11:50   0:00  |       |       \_ bash -c (echo >/dev/tcp/2.98.11.205/445) &>/dev/null && echo 2.98.11.205
root     10898  0.0  0.0   2584   920 ?        S    11:50   0:00  |       \_ timeout -s 5 5 bash -c (echo >/dev/tcp/2.98.11.206/445) &>/dev/null && echo 2.98.11.206
root     10901  0.0  0.0   3980  3040 ?        S    11:50   0:00  |       |   \_ bash -c (echo >/dev/tcp/2.98.11.206/445) &>/dev/null && echo 2.98.11.206
root     10903  0.0  0.0   3980   292 ?        S    11:50   0:00  |       |       \_ bash -c (echo >/dev/tcp/2.98.11.206/445) &>/dev/null && echo 2.98.11.206
root     10907  0.0  0.0   2584   900 ?        S    11:50   0:00  |       \_ timeout -s 5 5 bash -c (echo >/dev/tcp/2.98.11.207/445) &>/dev/null && echo 2.98.11.207
root     10909  0.0  0.0   3980  3128 ?        S    11:50   0:00  |       |   \_ bash -c (echo >/dev/tcp/2.98.11.207/445) &>/dev/null && echo 2.98.11.207
root     10911  0.0  0.0   3980   288 ?        S    11:50   0:00  |       |       \_ bash -c (echo >/dev/tcp/2.98.11.207/445) &>/dev/null && echo 2.98.11.207
root     10910  0.0  0.0   2584   944 ?        S    11:50   0:00  |       \_ timeout -s 5 5 bash -c (echo >/dev/tcp/2.98.11.208/445) &>/dev/null && echo 2.98.11.208
root     10913  0.0  0.0   3980  3032 ?        S    11:50   0:00  |       |   \_ bash -c (echo >/dev/tcp/2.98.11.208/445) &>/dev/null && echo 2.98.11.208
root     10915  0.0  0.0   3980   288 ?        S    11:50   0:00  |       |       \_ bash -c (echo >/dev/tcp/2.98.11.208/445) &>/dev/null && echo 2.98.11.208
root     10912  0.0  0.0   2584   948 ?        S    11:50   0:00  |       \_ timeout -s 5 5 bash -c (echo >/dev/tcp/2.98.11.209/445) &>/dev/null && echo 2.98.11.209

I have no idea what IP addresses those are. It kind of looks like it's trying to do an address scan or something. There are literally hundreds of processes spawning from this script and this is only a small snippet.

Appreciate any insight. Thanks

ghTravis commented 7 months ago

Looking into this a bit more, I see that this script is called when doing something with samba shares, and was newly added a couple weeks ago. I did enable one of my shares as a samba share about two or so days ago, but it looks like there's something wrong with the logic here because it appears to be scanning the entire internet. I run entirely on a couple 10. /24 network ranges for my internal network.

dlandon commented 7 months ago

What version of UD do you have installed?

ghTravis commented 7 months ago

What version of UD do you have installed?

Well I just uninstalled it because I was trying to get it to stop doing what it was doing but I'm almost positive it was whatever the latest version was because unraid was not telling me to update it. Since I installed unraid I have been attempting to keep everything up to date since I'm looking at it every day while setting it up.

This is the plugin version on the unraid apps page Current Version | 2023.12.29

dlandon commented 7 months ago

And you probably just installed the latest version. That port scan occurs when searching for SMB servers on the LAN. Due to a strange situation that I haven't been able to reproduce, it will accidently scan outside the LAN. In the latest version it is blocked if the port scan is outside the LAN. The latest version should not be doing that scan.

ghTravis commented 7 months ago

I did update unassigned devices once. So I think my initial install of the plugin was on whatever the previous version was. Could this scan have been running that long? It appeared to be counting up, and it got to the 2. address range so seemingly I have been pinging a lot of IPs... Lol

dlandon commented 7 months ago

Yes, it could be that out of control. A reboot should clear it up for sure.

ghTravis commented 7 months ago

I just reinstalled the plugin along with Plus and Preclear. Will see if it acts up again.

ghTravis commented 7 months ago

OK I just got it to do the same thing again. If I go to the Main page and click on the Add Remote Share image

Select Windows icon, and then put nothing in the "Enter of Select Server" box and hit "Search for servers" It apparently starts at 0.0.0.0 and scans the entire internet.

image

It doesn't show any indicator that it's performing this scan or the progress other than the spinner, but you can navigate away from this screen and go about your day and not know that it's still scanning.

ghTravis commented 7 months ago

Also for anyone else that comes across this, you can simply start a console into the unraid OS and kill the main process and it will eventually quit out of all the subprocesses

Look for the process that is /bin/bash plugins/unassigned.devices/scripts/hosts_port_ping.sh and run kill <PID>

dlandon commented 7 months ago

Can you post your diagnostics. I suspect there is a mis-configuration that causes the issue and UD isn't catching it.

ghTravis commented 7 months ago

Which file is relevant to you? Or do you want the whole zip file?

dlandon commented 7 months ago

All of it please. I need to check multiple places for the IP address UD is picking up to do the scan.

Also, what does the GUI do after you click on the "Search for Servers" button? And what does the server field show? Does the button continue to show "Searching" and never go off?

ghTravis commented 7 months ago

All of it please. I need to check multiple places for the IP address UD is picking up to do the scan.

Also, what does the GUI do after you click on the "Search for Servers" button? And what does the server field show? Does the button continue to show "Searching" and never go off?

It just shows the spinner next to Searching for server and doesn't finish. I end up clicking away from the modal popup and doing something else after awhile. I can see the hosts_port_ping.sh start up in the console as soon as I hit the button and starts at 0.0.0.0.

dlandon commented 7 months ago

Go the the "Tools->PHP Settings" and see if there are any php errors.

I've found something and I know what is causing the issue, but I don't see how it can happen. I will make a change to block this condition.

Unzip and copy the attached file to /usr/local/emhttp/plugins/unassigned.devices/scripts/hosts_port_ping.sh

This should stop the crazy port scans. What will happen now is if the scan parameters are wrong, clicking the "Search for Servers" will not do anything. If the parameters are correct, the scan will occur.

hosts_port_ping.zip

ghTravis commented 7 months ago

I don't see any issues in the PHP logs. I set it to all categories and pressed the search button a few more times while watching the log, and got the scan to trigger on 0.0.0.0 again. Nothing output in the logs there.

I copied the script in after checking the logs but since I switched the script I have triggered the search a few more times and I can see it searching through 10.0.0.0/24. Triggered it a few more times and seems to be doing 10. consistently now. I get two entries though and I'm not sure why. I only made a single share.

image

ghTravis commented 7 months ago

I assume you were expecting the scan to do nothing, since scanning from 0.0.0.0 would indicate my parameters are wrong. I can't tell you why it's scanning the 10. now. I haven't really changed anything except deleting and reinstalling the addon a couple more times because I thought the script wouldn't trigger since I used a kill command on the server for the script process.

dlandon commented 7 months ago

That script is only part of the answer. I'm working on the rest and I suspect the duplicates would not appear. Once I am convinced that I have the correct solution, I'll release a new version of UD. Probably in a day or two.

dlandon commented 7 months ago

I just found the issue with the duplicate servers showing up. You have a virtual network on eth0 and UD found the server available on both networks. I'll filter out the duplicates.

dlandon commented 7 months ago

Update UD.