search

raspberrypi / piserver

Raspberry Pi Server wizard to serve Raspbian to network booting Pis
310 stars 60 forks source link

Mounting /var/log via nfs mount to piserver #38

Closed rprr closed 5 years ago

rprr commented 5 years ago

I am trying to save the log files as I want to track down instances when the remotely booted pi's become intermittently unresponsive. I thought I could export an nfs export directory on the piserver and then mount the /var/log to this nfs directory. But not having any luck getting it to boot properly. This is what I did:

On piserver machine:

  1. Create directory: cd /; sudo mkdir /rpilogs; sudo chmod go+rwx /rpilogs.
  2. Edit /etc/exports to add the following line: /rpilogs 192.168.1.201(rw,sync,no_subtree_check) where 192.168.1.201 is the IP address of the pi.
  3. Restart nfs exports with: sudo exportfs -ra
  4. I verified that this is exported.

On the pi chroot:

  1. Edit the /etc/fstab to add the following line: piserver:/rpilogs /var/log nfs defaults,nolock 0 0
  2. Earlier I had the logs as tmpfs, I commented out this line. tmpfs /var/log tmpfs defaults,mode=755,size=100m 0 0

But this does not seem to boot the pi properly. The pi is network port is up (can ping) but does not let me ssh into it. This pi is in a remote location with no monitor, keyboard access. I am able to power cycle the pi remotely and can get it back up and running once I remove the offending line in the fstab above. I have tried various options, mode etc but no luck.

Any suggestions on how to keep the logs. Thanks.

PS: I can confirm that regular nfs works. I have been able to mount directories as rw from the command line. For example, mkdir /mnt/rpilogs ; mount -o rw -t nfs piserver:/rpilogs /mnt/rpilogs

maxnet commented 5 years ago

If NFS gives trouble, my first resort would not be to debug NFS on a system you have no local access to, but tell syslogd to sent the log output to a remote server over UDP instead.

Out of the top of my head that involves creating a config file like /etc/rsyslog.d/mylogging.conf in the chroot, and adding *.* @piserver to it. And on the piserver host uncommenting the right lines in /etc/rsyslog.conf to enable listening.

m-gavrilyuk commented 3 years ago

I am trying to save the log files as I want to track down instances when the remotely booted pi's become intermittently unresponsive.

@rprr
Have you find solutions of your problem? I have the same trouble.

m-gavrilyuk commented 3 years ago

I set up sending logs, but the next time clent freeze, there are no errors in them. logs also go further ping goes avalibe but the client itself completely freezes there is no way to connect via ssh and vns and the epoptes client also falls off @maxnet any ideas what is can be? The client are in hard-to-reach places and it is problematic to constantly restart them. Please help .

maxnet commented 3 years ago

I set up sending logs

Through UDP? If logging through network does not work properly, the alternative would be serial console. But that does require another Pi or other device near the problem device to connect it to.

m-gavrilyuk commented 3 years ago

I set up sending logs

Through UDP?

Yes udp. I guess chromimum takes all the RAM. What's happend if all free RAM end ?

maxnet commented 3 years ago

What's happend if all free RAM end ?

If memory becomes low kernel starts killing processes starting with the largest memory consumer.

m-gavrilyuk commented 3 years ago

I see his trying to kill chrome-browser process but but for some reason his cant and freeze. Thanks I'll try to write script to do reload chrome early.

maxnet commented 3 years ago

The clients just run Chromium? In case you auto-login the user and have /home/pi on tmpfs, make sure you disable disk caching like described here: https://github.com/raspberrypi/piserver/issues/42#issuecomment-461565345 Otherwise the "disk" cache is also eating up memory.

m-gavrilyuk commented 3 years ago

I use them for information monitor. Thank for help.