Expensify / Bedrock

Rock solid distributed database specializing in active/active automatic failover and WAN replication
https://bedrockdb.com
GNU Lesser General Public License v3.0
1.08k stars 82 forks source link

Verbose by default doesn't seem to good #68

Open willfong opened 7 years ago

willfong commented 7 years ago

By default, it seems like verbose logging is enabled:

root@ubuntu2:~# service bedrock start
 * Starting Expensify Bedrock Server bedrock                                                                                                                      [ OK ] 
root@ubuntu2:~# ps aux|grep bedrock
root      3704  0.0  0.0 399492  7604 ?        Ssl  14:17   0:00 /usr/sbin/bedrock -fork -nodeName bedrock -db /var/lib/bedrock/bedrock.db -serverHost 0.0.0.0:8888 -nodeHost 0.0.0.0:8889 -priority 200 -pidfile /var/run/bedrock.pid -quorumCheckpoint 100 -readThreads 4 -plugins status,db,jobs,cache,mysql -v -cache 10001

That doesn't seem like a good default. My 100-query sysbench run added 10M to syslog:

root@ubuntu2:~# logrotate -f /etc/logrotate.conf 
root@ubuntu2:~# ls -la /var/log/syslog
-rw-r----- 1 syslog adm 0 Dec 22 14:21 /var/log/syslog
root@ubuntu2:~# sysbench/sysbench/sysbench --test=sysbench/sysbench/tests/db/oltp.lua --mysql-user=root --mysql-password= --mysql-db=test --mysql-table-engine=innodb --mysql-ignore-duplicates=on --oltp-read-only=off --oltp-dist-type=uniform --oltp-skip-trx=on --oltp-auto-inc=off --init-rng=on --oltp-test-mode=complex --max-requests=0 --report-interval=30 --num-threads=8 --mysql-host=127.0.0.1 --mysql-port=3306 --oltp-table-size=5000000 --oltp-tables-count=4 --max-requests=100 run > bench.log 2>&1
root@ubuntu2:~# ls -la /var/log/syslog
-rw-r----- 1 syslog adm 10327373 Dec 22 14:22 /var/log/syslog
root@ubuntu2:~# grep -v "bedrock" /var/log/syslog | wc -l
0
root@ubuntu2:~# 

Maybe there's a better way to handle this?

Thanks, -will

quinthar commented 7 years ago

Hm, well we're actively developing on Bedrock so those logs are super helpful. However, I agree that if you're not actively improving Bedrock itself, then they're mostly just noise. I think the solution would be to add a -q "quiet mode" option that reduces verbosity (eg, only showing HMMM, WARN, and ALRT).

quinthar commented 7 years ago

To do this should be simple:

  1. This line sets the default log level to LOG_INFO
  2. This line sets it to LOG_DEBUG if you provide command line parameter -v for verbose logging
  3. Add an else if (args.isSet("-q")) check that sets LOG_WARN logging level, meaning it will only log warns and above if -q is provided
  4. Add the parameter after this line documenting the new -q parameter
quinthar commented 7 years ago

FYI, posted an Upwork project here if anybody wants to earn a quick $500: https://www.upwork.com/jobs/~01a73c71734d653add

willfong commented 7 years ago

Is it possible to make the -q option enabled by default in the apt package? That way non-developers won't have to worry about enabling it? Thanks!

quinthar commented 7 years ago

Thanks @basiliscos I really I appreciate the quick work!

@righdforsa I like the idea of making -q the default for the public repo, but running without it internally. How would you recommend doing that?

Also, @righdforsa, what's the ETA for the next public release, and ideally switching our internal use over to the public apt get repo?

quinthar commented 7 years ago

-q option added in HEAD; reassigning to @righdforsa until deployed.