SeattleTestbed / clearinghouse

Donate, share, manage, control, and track resources
MIT License
0 stars 11 forks source link

Clearinghouse logs grow too quickly #169

Open awwad opened 8 years ago

awwad commented 8 years ago

The clearinghouse logs seem to grow too quickly and not be culled or rotated.

These logs started 3 weeks (just over 21 days) before this snapshot, (AFAICT) involve no rapidly recurring errors or other anomalies, and occur on a very low traffic test clearinghouse accessible only on the .poly.edu network:

-rw-r--r--  1 root root 2.9G Feb 19 11:12 transition_twopercent_to_twopercent.log
-rw-r--r--  1 root root 1.4G Feb 19 11:12 transition_canonical_to_twopercent.log
-rw-r--r--  1 root root  37M Feb 19 11:12 backend.log
-rw-r--r--  1 root root 1.1G Feb 19 11:12 transition_donation_to_canonical.log
-rw-r--r--  1 root root  23M Feb 19 11:12 check_active_db_nodes.log
-rw-r--r--  1 root root 247M Feb 19 11:12 lockserver.log
-rw-r--r--  1 root root 2.7G Feb 19 11:12 transition_onepercentmanyevents_to_canonical.log
awwad commented 8 years ago

(Also, why are these logging as root...?)

awwad commented 8 years ago

( Ah...

$SUDO_CMD python $CLEARINGHOUSE_DIR/lockserver/lockserver_daemon.py >>$LOG_DIR/lockserver.log 2>&1 &

While the script is being run as the clearinghouse user, the redirect is still technically as root. I suppose we could move that into the script. I suppose there's probably also some single-line-fu way of making the redirect happen as the clearinghouse user, instead of moving the logging into each of the scripts?

EDIT: Single-line-fu solutions exist in this thread. In short: 'tee' command, or explicitly starting a shell with -c (sudo sh -c 'ls -hal /root/ > /root/test.out') )

awwad commented 8 years ago

As for the original issue:

What ever happened to this issue? I don't see weekly log rotation anywhere (nor size-based).