search

gnuton / asuswrt-merlin.ng

Extends the support of Merlin firmware to more ASUS routers
Other
1.42k stars 82 forks source link

No space left on device #334

Open jorkzijlstra opened 1 year ago

jorkzijlstra commented 1 year ago

Router Model Affected Models: RT-AX92U

Firmware Version Affected RT-AX92U_388.1_0-gnuton1_ubi.w

Is this bug present in upstream Merlin releases too? Don't know

Describe the bug Wednesday Jan 25th I upgraded my system from 386_1 to the 388_1 firmware and the device became unresponsive Jan 26th 10PM.

After checking the logs I saw these:

Jan 26 22:04:58 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:05:58 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:06:58 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:07:58 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:08:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:09:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:10:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:11:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:12:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:13:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:14:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:15:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:16:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:17:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:18:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:19:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:20:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:21:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:22:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:23:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:24:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:25:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:26:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:27:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:28:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:29:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:30:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:31:59 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:32:57 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Jan 26 22:33:57 dnsmasq-dhcp[3146]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)

Apparently something is now filling up the disk but have no idea what exactly. I didn't notice anything like this in the 386_1 release not in the original Asus firmware.

To Reproduce Steps to reproduce the behavior: I'll go back to the 386_1 firmware to compare the logs output. Maybe something is now logging way more excessive in this new version?

Expected behavior To not crash by a full disk

gnuton commented 1 year ago

Hi, do you have scripts installed? have you reset to factory after flashing?

gnuton commented 1 year ago

could you ssh onto the router and run this? it will look for fatty files

find /var -type f -size +1M

jorkzijlstra commented 1 year ago

I'll try to ssh into the router to find that largest file.

I a suspect is that it seems that the romast is logging way more excessive:

In 386 I just see these 2 lines

Jan 27 10:17:25 roamast: sta 4E:0F:05:20:52:40 ap F0:2F:11:BB:AA:22 rcpi 98
Jan 27 10:17:25 roamast: after rssi -61
Jan 27 10:17:25 roamast: sta 4E:0F:05:20:52:40 ap F0:2F:11:BB:AA:11 rcpi 122
Jan 27 10:17:25 roamast: after rssi -49

while compared in 388

Jan 26 18:36:24 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 102 and rssi is -59
Jan 26 18:36:24 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 102 and rssi is -59
Jan 26 18:36:24 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 116 and rssi is -52
Jan 26 18:36:24 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 116 and rssi is -52
Jan 26 18:36:43 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 94 and rssi is -63
Jan 26 18:36:43 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 94 and rssi is -63
Jan 26 18:36:43 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 70 and rssi is -75
Jan 26 18:36:43 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 70 and rssi is -75
Jan 26 18:36:48 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 94 and rssi is -63
Jan 26 18:36:48 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 94 and rssi is -63
Jan 26 18:36:48 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 68 and rssi is -76
Jan 26 18:36:48 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 68 and rssi is -76
Jan 26 18:36:53 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 84 and rssi is -68
Jan 26 18:36:53 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 84 and rssi is -68
Jan 26 18:36:53 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 76 and rssi is -72
Jan 26 18:36:53 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 76 and rssi is -72
Jan 26 18:37:03 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 80 and rssi is -70
Jan 26 18:37:03 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 80 and rssi is -70
Jan 26 18:37:03 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 70 and rssi is -75
Jan 26 18:37:03 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 70 and rssi is -75
Jan 26 18:37:08 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 88 and rssi is -66
Jan 26 18:37:08 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 88 and rssi is -66
Jan 26 18:37:08 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 82 and rssi is -69
Jan 26 18:37:08 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 82 and rssi is -69
Jan 26 18:37:13 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 88 and rssi is -66
Jan 26 18:37:13 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 88 and rssi is -66
Jan 26 18:37:13 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 76 and rssi is -72
Jan 26 18:37:13 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 76 and rssi is -72
Jan 26 18:37:23 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 86 and rssi is -67
Jan 26 18:37:23 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 86 and rssi is -67
Jan 26 18:37:23 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 86 and rssi is -67
Jan 26 18:37:23 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 86 and rssi is -67
Jan 26 18:37:49 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 78 and rssi is -71
Jan 26 18:37:49 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 78 and rssi is -71
Jan 26 18:37:49 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 78 and rssi is -71
Jan 26 18:37:49 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 78 and rssi is -71
Jan 26 18:37:53 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 64 and rssi is -78
Jan 26 18:37:53 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 64 and rssi is -78
Jan 26 18:37:53 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 64 and rssi is -78
Jan 26 18:37:53 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 64 and rssi is -78
Jan 26 18:37:58 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 68 and rssi is -76
Jan 26 18:37:58 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 68 and rssi is -76
Jan 26 18:37:58 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 78 and rssi is -71
Jan 26 18:37:58 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 78 and rssi is -71
Jan 26 18:38:08 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 82 and rssi is -69
Jan 26 18:38:08 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 82 and rssi is -69
Jan 26 18:38:08 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 86 and rssi is -67
Jan 26 18:38:08 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 86 and rssi is -67
Jan 26 18:38:13 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 88 and rssi is -66
Jan 26 18:38:13 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 88 and rssi is -66
Jan 26 18:38:13 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 68 and rssi is -76
Jan 26 18:38:13 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 68 and rssi is -76
Jan 26 18:38:23 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 108 and rssi is -56
Jan 26 18:38:23 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 108 and rssi is -56
Jan 26 18:38:23 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 90 and rssi is -65
Jan 26 18:38:23 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 90 and rssi is -65
Jan 26 18:38:43 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 80 and rssi is -70
Jan 26 18:38:43 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 80 and rssi is -70
Jan 26 18:38:43 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 74 and rssi is -73
Jan 26 18:38:43 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 74 and rssi is -73
Jan 26 18:38:53 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 80 and rssi is -70
Jan 26 18:38:53 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 80 and rssi is -70
Jan 26 18:38:53 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 60 and rssi is -80
Jan 26 18:38:53 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 60 and rssi is -80
Jan 26 18:38:58 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 82 and rssi is -69
Jan 26 18:38:58 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 82 and rssi is -69
Jan 26 18:38:58 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 66 and rssi is -77
Jan 26 18:38:58 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 66 and rssi is -77
Jan 26 18:39:08 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 72 and rssi is -74
Jan 26 18:39:08 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 72 and rssi is -74
Jan 26 18:39:08 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 58 and rssi is -81
Jan 26 18:39:08 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 58 and rssi is -81
Jan 26 18:39:28 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 52 and rssi is -84
Jan 26 18:39:28 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 52 and rssi is -84
Jan 26 18:39:28 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 50 and rssi is -85
Jan 26 18:39:28 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 50 and rssi is -85
Jan 26 18:39:38 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 56 and rssi is -82
Jan 26 18:39:38 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 56 and rssi is -82
Jan 26 18:39:38 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 48 and rssi is -86
Jan 26 18:39:38 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 48 and rssi is -86
Jan 26 18:39:48 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 70 and rssi is -75
Jan 26 18:39:48 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:11], rcpi is 70 and rssi is -75
Jan 26 18:39:48 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 68 and rssi is -76
Jan 26 18:39:48 roamast: sta[4E:0F:05:20:52:40] on ap[F0:2F:11:BB:AA:22], rcpi is 68 and rssi is -76
jorkzijlstra commented 1 year ago

could you ssh onto the router and run this? it will look for fatty files

find /var -type f -size +1M

The find command with these particular arguments doesn't seem to be supported.

jorkzijlstra commented 1 year ago

This is the result for firmware 386 and will try 388 later command du -ahx / | sort -rg | head -20

result:

1020.0K /usr/sbin/sqlite3
944.0K  /www/js
940.0K  /www/images/New_ui/amesh/location
920.0K  /usr/bin
908.0K  /usr/lib/libiconv.so.2.5.1
900.0K  /usr/lib/libsqlite3.so.0.8.6
896.0K  /usr/bwdpi/rule.trf
820.0K  /lib/modules/4.1.51/kernel/net/netfilter
768.0K  /usr/sbin/fatfsck
756.0K  /www/mobile
740.0K  /usr/lib/ipsec/libcharon.so.0.0.0
732.0K  /usr/sbin/mkfatfs
712.0K  /lib/modules/4.1.51/kernel/fs/tntfs.ko
700.0K  /usr/lib/libdb-4.8.so
700.0K  /lib/libm.so.6
676.0K  /rom/usr/share/zoneinfo/America
668.0K  /usr/sbin/openvpn
636.0K  /usr/sbin/httpd
636.0K  /usr/sbin/cfg_server
604.0K  /sbin/ldconfig
Snid0 commented 1 year ago

I have experienced this issue too a few days ago (also on RT-AX92U). The router froze up and the logs showed the same error about space. I rebooted the router and the issue resolved itself. I am going to keep monitoring the space to see if it reocccurs and try and identify what is using up the space, but so far it hasnt reoccured.

DrDagless commented 1 year ago

Yep, same for me on latest fw with RT-AX92U. Happens in the syslog every minute:

Jan 27 09:14:07 dnsmasq-dhcp[1850]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s) Jan 27 09:15:07 dnsmasq-dhcp[1850]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s) Jan 27 09:16:07 dnsmasq-dhcp[1850]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s) Jan 27 09:17:07 dnsmasq-dhcp[1850]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)

I had noticed something similar on the beta for 388 (and also had some other issues like not being able to access the web admin page.

jorkzijlstra commented 1 year ago

Could you check which file / directory is filling up the disk?

I just re-uploaded the 388 firmware so it will be a few while before I can check this again on my router.

MrRojano97 commented 1 year ago

Try running this command via SSH after starting your router. service stop_conn_diag

The issue is present in all Asus Routers using even the vanilla version of 388.

balilu91 commented 1 year ago

EDIT: actually ignore my comment .. I was on the node.. cant SSH to the main one to confirm what it looks like

Having the same issue and tried also looking / following at the discussion over on https://www.snbforums.com/threads/out-of-memory-errors-388-1-no-space-left-on-device.82406

One of the fixes suggested over in the thread is the same as @MrRojano97 suggested.

Assuming I'll need to run it as soon as I start the router again., as currently it's "broken" and stopping that service will not resolve the issue

What I'm finding weird (unless I'm missing something) is that when checking disk space it's not full 

Filesystem                Size      Used Available Use% Mounted on
ubi:rootfs_ubifs         69.4M     65.6M      3.9M  94% /
devtmpfs                185.5M         0    185.5M   0% /dev
tmpfs                   185.6M    220.0K    185.4M   0% /var
tmpfs                   185.6M      1.8M    183.9M   1% /tmp/mnt
mtd:bootfs                5.1M      4.2M    980.0K  81% /bootfs
mtd:data                  8.0M    632.0K      7.4M   8% /data
tmpfs                   185.6M      1.8M    183.9M   1% /tmp/mnt
tmpfs                   185.6M      1.8M    183.9M   1% /tmp
/dev/mtdblock9           63.0M      1.9M     61.1M   3% /jffs

I also seem to have enough memory 

Mem: 256152K used, 124036K free, 2184K shrd, 0K buff, 18724K cached

Output of du

du -ahx / | sort -rg | head -20 

1020.0K /usr/sbin/sqlite3
952.0K  /usr/bin
940.0K  /www/images/New_ui/amesh/location
908.0K  /usr/lib/libiconv.so.2.5.1
900.0K  /usr/lib/libsqlite3.so.0.8.6
896.0K  /usr/bwdpi/rule.trf
820.0K  /lib/modules/4.1.51/kernel/net/netfilter
772.0K  /usr/lib/ipsec/libcharon.so.0.0.0
768.0K  /www/mobile
768.0K  /usr/sbin/fatfsck
732.0K  /usr/sbin/mkfatfs
712.0K  /lib/modules/4.1.51/kernel/fs/tntfs.ko
708.0K  /usr/sbin/httpd
700.0K  /usr/lib/libdb-4.8.so
700.0K  /lib/libm.so.6
688.0K  /www/images/New_ui/weekSchedule
676.0K  /rom/usr/share/zoneinfo/America
668.0K  /usr/sbin/openvpn
664.0K  /usr/lib/libavformat.so.58
644.0K  /usr/sbin/cfg_server

am I missing something ?

Nehaxak commented 1 year ago

Hey, maybe related, was a post on the snb forums from Rmerlin...

Post in thread 'Out of Memory Errors 388.1 / No space left on device' https://www.snbforums.com/threads/out-of-memory-errors-388-1-no-space-left-on-device.82406/post-819862

"Asus confirmed that the conn_diag database issues were already fixed on their end, and should therefore be included in the next code release."

So maybe Asus have recognised the issue and the upcoming patch/release will sort it out.

mmurza commented 1 year ago

I had strange issues with 388 firmwares on my RT-AX92U (beta and stable). Running out of space could explain this. Router got stuck/unstable after ~1 day of operation: slow or no internet for connected devices, no new WiFi conneections accepted. I couldn't connect to router to look at logs / troubleshoot - just reboot helps. Tried to schedule reboot every night. But that didn't help either (running out of space in <24h?)

Therefore - rolled back to 386. Looking forward for next beta!

DrDagless commented 1 year ago

Just following up, had a really severe issue this morning. Noticed when I woke up that there was no internet, so I rebooted the router and node. Internet still wasn't working so I went to the admin page and it was prompting me to set up the router as a new device. Had to go through all the steps and set it up as new. Obviously very frustrating since all my settings were wiped out.

It seems like 388 has some stability issues with the ax-92u, so I reverted back to 386. I was having issues with 386 as well (weekly disconnects where the internet would stop working, even thought the modem connection itself was fine), but maybe they'll be resolved with the factory reset.

megosugit commented 1 year ago

Hello, I have the same issue with the same model and firmware (RT-AX92U_388.1_0-gnuton1_ubi.w). After 2 to 3 days, I have issues connecting to the network. If I assign myself a manual IP address, I can use Internet. I cannot SSH when it happens, as the SSH connection is almost impossible, and when it lets me, it refuses my usual credentials. I cannot get the http or https login page.

I tried to check the /var space every morning and evening after a reboot, I didn't see any big files and space left was far from being full (664B used only). I couldn't check this morning, and the message started to appear around 3:29 AM. Too bad I didn't know and didn't check the /var used space today before being locked out of ssh.

After a hardware reboot, when I look at the system logs, I see also the following message starting to appear several hours before total outage: Feb 19 11:26:20 dnsmasq-dhcp[11057]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)

Having found this thread after today outage, I have now input the service stop_conn_diag hoping the issue might be fixed while waiting for a new release.

Do you have any ETA about the fix ? Is the service stop_conn_diag supposed to help ?

Thank you

DrDagless commented 1 year ago

@megosugit

I tried the service stop_conn_diag and it didn't help. In fact it seemed to really mess things up (as per the post above), but not sure it was directly related.

Ended up having to go back to 386 for now. Hopefully the bugs will be ironed out.

megosugit commented 1 year ago

@DrDagless scary... I hope mine will survive without having to reconfigure everything. I did a new backup just in case. Didn't you have one ?

I'm disappointed, because I was expecting Merlin to be available for our model but it was not compatible. Then I discovered it finally came and that it was not beta anymore and that stability was supposed to be the main goal, I'm now wondering if stock was not a better option :(

rlfrank165 commented 1 year ago

Not sure why I feel compelled to "defend" the Merlin effort and Gnuton, but somehow feel it's become my job to remind everyone that almost without exception all of the bugs being reported with Merlin and Gnuton versions of the 388 releases also happen with the Asus official 388 releases.

My understanding (and hope) is that there may be some newer releases from Asus that at least acknowledge and fix some of these issues and that hopefully there will be a new GPL release from Asus that will hopefully make its way into Merlin and then Gnuton in the not too distant future.

Randy Frank

On Mon, Feb 20, 2023 at 12:40 PM megosugit @.***> wrote:

@DrDagless https://github.com/DrDagless scary... I hope mine will survive without having to reconfigure everything. I did a new backup just in case. Didn't you have one ?

I'm disappointed, because I was expecting Merlin to be available for our model but it was not compatible. Then I discovered it finally came and that it was not beta anymore and that stability was supposed to be the main goal, I'm now wondering if stock was not a better option :(

— Reply to this email directly, view it on GitHub https://github.com/gnuton/asuswrt-merlin.ng/issues/334#issuecomment-1437361332, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2TBKSV2CASEYCM6VUTCTTTWYOUBTANCNFSM6AAAAAAUIMKCOM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

balilu91 commented 1 year ago

I'm using the workaround documented on SNB Forums to setup a script to clean the DBs in /tmp/.diag for my RT-AX92U

https://www.snbforums.com/threads/out-of-memory-errors-388-1-no-space-left-on-device.82406

Router ran fine for around 20 hours and then it didn't

Currently the script is not completing the whole run for 99% of the times.

As can be seen by the below output router is saying there is no space for dnsmasq however there is plenty of space.

 |   |2023-02-20 15:00:00 | *** Starting /jffs/scripts/clean-dbs.sh ***
  |   | 2023-02-20 15:00:00 | *** Before Disk Free:***
  |   | 2023-02-20 15:00:00 | Filesystem                Size      Used Available Use% Mounted on
  |   | 2023-02-20 15:00:00 | ubi:rootfs_ubifs         69.4M     65.6M      3.9M  94% /
  |   | 2023-02-20 15:00:00 | devtmpfs                185.5M         0    185.5M   0% /dev
  |   | 2023-02-20 15:00:00 | tmpfs                   185.6M    508.0K    185.1M   0% /var
  |   | 2023-02-20 15:00:00 | tmpfs                   185.6M      3.2M    182.5M   2% /tmp/mnt
  |   | 2023-02-20 15:00:00 | mtd:bootfs                5.1M      4.2M    980.0K  81% /bootfs
  |   | 2023-02-20 15:00:00 | mtd:data                  8.0M    600.0K      7.4M   7% /data
  |   | 2023-02-20 15:00:00 | tmpfs                   185.6M      3.2M    182.5M   2% /tmp/mnt
  |   | 2023-02-20 15:00:00 | tmpfs                   185.6M      3.2M    182.5M   2% /tmp
  |   | 2023-02-20 15:00:00 | /dev/mtdblock9           63.0M      3.0M     60.0M   5% /jffs
  |   | 2023-02-20 15:00:00 | *** ---------------------------------- ***
  |   | 2023-02-20 15:00:00 | *** Before Sizes in /tmp/.diag:***
  |   | 2023-02-20 15:00:34 | [check_watchdog] restart watchdog for no heartbeat
  |   | 2023-02-20 15:00:34 | check_watchdog 1277:notify_rc restart_watchdog
  |   | 2023-02-20 15:00:39 | failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)

Currently getting bursts of tainted processes every couple of hours. mostly networkmap and cfg_server right after each other. Got around 80 hits in less than 48 hours

Having said so.. the router is still running however no SSH, web interface and somewhat unreliable during gaming.

Not sure if it's the exact same issue on the AX92U since the fix didn't work for me whilst it did for some as stated on SNB forums.

megosugit commented 1 year ago

Hello,

my routeur is still alive for now... up 2 days, 19:56, I'm dreading the next crash :(

I have another question, just trying my luck here with you. I'll make a proper thread if you have no idea, but I'm not even sure it is related to Asus to be honest. I ought to try another wireless router to confirm this, but I need to borrow one from a friend.

The issue: Very often, when I change a setting (mac reservation for instance) or a wifi band, all my devices reconnect without issue but I have several IoT devices (Xiaomi and Yeelight) which disappear from the cloud. I can still ping them locally and I see them connected to the wifi, but they appear offline on the cloud app (Mi Home and Yeelight). If I switch them off/on, they come back to the network, but they are still offline in the cloud app. I tried to wait days, but it never fixes itself. I tried to block Internet access to the device for several hours, then unblocking it hoping it will reconnect, but no luck. I tried to bind it to the slave or the master, no luck. Only way I found to get them back is to either turn them off for several minutes (at least 5m I would say), then when I turn them back on, they work properly. Other solution, if I change their DHCP reserved IP (from 192.168.1.50 to 51 for instance), then they work fine until it happens again. I have been experiencing this issues for a while, and it is a bit annoying to have to chase them all one by one and see which one is not working properly each time. Would you have any suggestion ? or maybe another forum name where I could try to raise the question ? Kind regards and sorry for the off-topic :-(