Closed Anto79-ops closed 1 year ago
I had the same issue and reported it as a standalone comment - https://github.com/timmo001/system-bridge/issues/1831#issuecomment-1176809830.
It's likely still an issue for me, but since I haven't gotten the service working yet for other reasons, I haven't ran it for an extended period of time recently which seems necessary to reproduce the issue.
I can still confirm and reproduce that this is still happening. Happens within the first 5 hrs of starting the service. Please let me know if you need more data from my end.
AND as for the HA side of things, looks like there is some requested changes to the PR for merging into HA:
https://github.com/home-assistant/core/pull/75362#pullrequestreview-1059870510
cheers @timmo001
AND as for the HA side of things, looks like there is some requested changes to the PR for merging into HA:
Unrelated
This appears to only occur on linux, which I don't have a dedicated system to test with
thanks, @timmo001
I've reached out the Ubuntu forums, perhaps they can shed some light on this
https://ubuntuforums.org/showthread.php?t=2477750&p=14106978#post14106978
also, I found this:
Do you think its just a matter of increasing the limit? Any ideas as to what it should be?
FYI, when I check my system
anto@GC676AA-ABA-m8150n:~$ ulimit -a
real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15425
max locked memory (kbytes, -l) 502296
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 15425
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
it seems I have a limit of 1024 files.
OR does system_bridge close the files after it opens them (if files are being opened)?
It should, but there may be something else causing this. Potentially a fix could be to increase this limit
It should, but there may be something else causing this. Potentially a fix could be to increase this limit
Okay I'll test it by doubling my maximum file open limit. If that basically just doubles the time and then crashes then you're right there could be something there but if it works, then it works.
One thing I did notice is that my Ubuntu it has the old spinning hard drive and whenever system bridge is running the hard drive does work often suggesting quite a bit of disk activity.
Keep you posted
just a quick update, I have not made too much progress in this. Turns out you can increase file open limits, but there are at least 3 different ways of doing this, from either a process point of view, user point of view and system point of view.
Being not so fluent in Linux, I just have to rely on others to direct me...so I'll post something when I get new information.
@ScottG489 I don't know if you are familiar with this, perhaps you can try on your Linux system?
EDIT: by the way, @ScottG489 what are your file open limits? 1024 like mine? typing ulimit -a
will tell you.
I don't have access to the exact machine at the moment, but another linux machine has a limit of 1024
for file descriptors (ulimit -n
).
However, increasing the systems limit on file descriptors isn't the appropriate solution. This is an application side issue and needs to be fixed here.
@ScottG489 you're possibly right, there could be a bug in the code, and @timmo001 mentioned that could very well be the case (and some on the Ubuntu forums are saying its the code)
The reason why I'm willing to try the file open increase, is because the system that I'm running system bridge on, also is my MQTT broker and InfluxDB database, so its doing some other stuff.
If your Linux system is barely being used and it still reaching the 1024 limit, than I'm more inclined to believe that their some code leak for system bridge, and not an issues with the OS. I guess we will figure this out eventually :)
This appears to only occur on linux, which I don't have a dedicated system to test with
Hi @timmo001
Can I make a suggestion? You can install Ubuntu 22.04 on Windows computer as dual boot. I've done this, its pretty simple Ubuntu will recognize the other OS and install alongside it, and then upon restart you'll get a grub screen to pick which OS you want boot up. This way, you might be able to get a better understanding of these 2 issues, here.
https://github.com/timmo001/system-bridge/issues/1928 https://github.com/timmo001/system-bridge/issues/1876
Unless its just my system, @ScottG489 have you gotten System bridge to work with the new updates?
This is a really awesome integration, its just to bad I can't get it to work with my linux box!
Hey @Anto79-ops, thanks for keeping on top of this.
I believe the last version I tried things on was 3.4.3
. I don't see any changes in the most recent version (3.4.4
) that would indicate a fix, but I'll try again soon.
I'm also curious if anyone on Linux has gotten this to work? Have you ever had it working on Linux, @Anto79-ops? @timmo001 not having availability to a Linux box, even in a VM, makes me wonder how stable this is on Linux. However, I haven't taken a look at the test setup for this project, so it could have adequate coverage there.
@ScottG489 you're the only other Linux user that I know that is using system Bridge.
I have the same version as you however recently just yesterday home assistant merged some of @timmo001 PRs, which was supposed to solve a second issue of not seeing entities in HA. Updating to the latest ha core and I'm still not seeing entities in HA.
If you're not using HA then it's not that important for you, but the first issue will be related to you.
I've been running System Bridge on a debian headless server, no issues. Added to HA and showing entities, which is why these issues are confusing to me
I think the next step would be to reach out to other forums.
I'll post my issues on the HA forums, perhaps some can confirm or spot something.
Its a great community, things usually get solved.
@timmo001 just curious, do the errors posted on my 2 issue make sense to you?
Also, you mentioned that its working on your headless Debian. Perhaps I can also try the headless Ubuntu...if that makes sense?
I mean it has a desktop when I connect a monitor to it, but, there is no monitor connected to it and I simply just ssh to it when I need to.
Could this be related?
I think at a high level it would be a good idea to say what distros and versions this has been formally tested against. To troubleshoot the existing issues, we should try looking at the versions of dependencies that differ between systems where this project works and not. Then try looking at any other environmental differences.
I'm not sure if there's a docker offering for this project, but that would eliminate a lot if not all of these kinds of problems.
Also I do use HA, but since I hadn't gotten the service running stably, I didn't bother trying to integrate it yet.
Same error, Linux Mint Debian Edition 5 + Cinnamon
after the error occured, systemmonitor shows Memory usage >500MB
After restart it was ~40MB
List of open files for python tasks systembridgebackend started with 40 files, now after 20min there are 186 files, most of them with type "unknown"
Memory used 185MB
There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!
Still an issue
There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!
Still an issue
Yes it's still an issue. I've removed the integration from both home assistant and my Ubuntu system.
This issue still persists along with increased disk activity and memory usage.
I'd be happy to try again later
This seems to be Linux specific and since I do not daily drive a Linux system, I would need help in diagnosing this issue.
It seems it could be a wider Python issue/limitation from what I've read so far
I have an idea for a workaround, maybe make a script that when [Errno 24] Too many open files detected, it automatically restart the app? Restarting app solved the issue.
Just some remarks:
ulimit -n
shows you the maximum number of open files per process. This means if you hit this limit systembridge has opened 1024 fileshttps://www.howtogeek.com/805629/too-many-open-files-linux/
I did a quick check on my system (Ubuntu 22.04, systembridgebackend --no-gui) and systembridge has less than 300 (~280-290) open "files". Many of them are network connections. I also have never seen this error.
Thanks for this information @jd1
I just updated system bridge to the latest and ran it as a no GUI.
Can you please tell me how I can check how many files system bridge opens? That way I can confirm whether it is system Bridge or not that is causing this issue. Using Ubuntu 22.04 Cheers
At first you need to find the process id of systembridge, e.g. have a look at ps aux | grep python
.
Then you can list all open files of this process (I will use 1337 as an example) by running: lsof -p 1337
. If you want only the number of open files, run lsof -p | wc -l
.
Remark: I'm writing this on my tablet, so there might be some typos in the command, but I hope I got them correct :) You can find most of them in the articles linked above or in the links of the articles.
Awesome! ITs been running for 1 day and 8 hrs. now (which is the longest its ever run on my system), and no more too many open file errors....open files ranges from 100 to 133.
This is great news!
For those of you we have experienced this in the past from Jul 2022....try updating to the new version.
2 days and still going, 100 open files. Nice!
I'll give this a couple more days and I'll close this issue. So far so good!
Now, the HA PR needs to be pushed. Brought it Frenck's attention and he labelled as smash...so hopefully soon
approaching 3 days now...still running! still only 100 open files.
I'm going to be closing this issue in 24 hrs.
thanks @timmo001 !
4 days and still running, strong. 100 open files
Ok, this problem is officially solved.
How it was solved...I have no idea BUT I just kepy updating Ubuntu (including distro updates) and updated to latest System Bridge and it works.
thanks all @jd1 for bring this up,, and @timmo001 for the development. Cheers
Description
Hi!
Running the program in UBuntu 22.04 and using the web url to access to data
└─5042 /usr/bin/python3.10 -m systembridgebackend --no-gui
everything runs fine for a few hours, data is being generated and upates but then stops with the error messag below:
What Platform / OS are you running?
Linux
What version are you running?
pyinstaller-5.2 sanic-ext-22.6.2 systembridgebackend-3.4.1 systembridgecli-3.4.1 systembridgefrontend-3.4.1 systembridgegui-3.4.1 systembridgeshared-3.4.1 systembridgewindowssensors-3.4.1 typer-0.6.1
Anything in the logs or a references that might be useful?
Additional information
No response