search

markqvist / Reticulum

The cryptography-based networking stack for building unstoppable networks with LoRa, Packet Radio, WiFi and everything in between.
https://reticulum.network
MIT License
2.04k stars 127 forks source link

Too Many Files Open #588

Open faragher opened 18 hours ago

faragher commented 18 hours ago

Describe the Bug

After ~28 days uptime, Reticulum crashes due to too many files being open, breaking ratchets and logging, and eventually crashing the entire transport instance.

To Reproduce

I don't think it wise to wait a month to see if I can duplicate, this is a fundamental issue.

Expected Behavior

File handles closed after use.

Logs & Screenshots

Typical: 

Oct 18 05:52:07 reticulum.betweentheborders.com rnsd[553]:   File "/home/xxx/.local/lib/python3.11/site-packages/RNS/Identity.py", line 288, in persist_job
Oct 18 05:52:07 reticulum.betweentheborders.com rnsd[553]: OSError: [Errno 24] Too many open files: '/home/xxx/.reticulum/storage/ratchets/xxx>

System Information

Additional context

Working theory is that there's an error inside a try/except cycle which leaves a handle dangling.

Suggestion: Either migrate to with open('file.ext','r') as f: format or move file.close() out of try: and into finally:

faragher commented 6 hours ago

Okay, I found the issue. Identity.py line 332 opens a file which is never closed. I'm halfway through changing Identity.py over into with open so I'm going to finish that up and fix the problem. If that's unacceptable, then it's a quick fix closing the file that was opened on 332.

liamcottle commented 1 hour ago

and eventually crashing the entire transport instance.

Heh, I've noticed now and then when I get up in the morning, MeshChat is no longer running and had closed itself overnight. I hadn't added any logging for this yet, but this could be related. Seemed to only happen once a month or so... :D

Looks like your error log is related to linux, whereas I'm running on Windows. So no idea until I investigate further, but thought I'd throw in some extra info...