search

aparcar / openwrt

Staging tree of Paul Spooren
Other
8 stars 1 forks source link

logread starts before interface is up #19

Open aparcar opened 2 years ago

aparcar commented 2 years ago

Supply the following if possible:

openwrt 21.02.1 on x86_64 (virtualbox) interfaces are all Intel Pro/1000 T Server 82543GC logread is configured to forward messages to a syslog server

It seems as if logread forwarding starts before it is able to actually send messages to the syslog server, resulting in lost messages. This is important as the syslog server generates monitoring information, including status of all interfaces on the router. The loss of early interface messages means that the monitoring does not get a full picture of the status of these devices.

I have attached a log of the console output from the router bootup. I have also attached a log of the messages received by the syslog server.

Interestingly, other routers forward their messages to the same syslog server and yet none of those routers' messages are dropped. These routers have nearly identical configuations too.

Incidentally, a google search reveals that this was previously logged on the old openwrt bug tracker at [[https://dev.archive.openwrt.org/ticket/13861.html|External Link]]. Additionally, a similar bug was logged against logread at [[https://github.com/josefbacik/busybox/issues/4.|External Link]].

aparcar commented 2 years ago

I tried changing the order of the init scripts by moving the "log" service from 12th to 21st in the sequence so that it would come just after "network" service. This does not work either and may actually be losing even more information.

At first it may seem like the question of "which came first, the chicken or the egg?" But I think it is more that the chicken and the egg need to be separated. The "log" service launches both the system log daemon (which saves early boot messages in its ring buffer) and the logread daemon which forwards the messages on to the syslog server.

Instead, if the logd daemon were launched separately, early on in the boot process, and the logread daemon were launched later on after the "network" service has completed its startup, hopefully fewer messages would be dropped or lost. In fact, with a sufficient ring buffer size, lost messages might be virtually eliminated.