search

splay-project / splay

The main Splay repository.
GNU General Public License v3.0
16 stars 3 forks source link

Error in case of no space left on controller machine #50

Open vschiavoni opened 9 years ago

vschiavoni commented 9 years ago

When the disk space of the controller machine is exhausted and the controller daemon cannot write logs on disk, the following messages are printed on screen:

E, [2015-07-14 11:43:37 #29518] ERROR -- : Errno::ENOSPC: No space left on device - /home/splayd/splay/src/controller/logs/32
/home/splayd/splay/src/controller/lib/logd.rb:196:in `write'
/home/splayd/splay/src/controller/lib/logd.rb:196:in `puts'
/home/splayd/splay/src/controller/lib/logd.rb:196:in `run'
/home/splayd/splay/src/controller/lib/logd.rb:129:in `initialize'
/home/splayd/splay/src/controller/lib/logd.rb:129:in `new'
/home/splayd/splay/src/controller/lib/logd.rb:129:in `run'
/home/splayd/splay/src/controller/lib/logd.rb:74:in `main'
/home/splayd/splay/src/controller/lib/logd.rb:60:in `loop'
/home/splayd/splay/src/controller/lib/logd.rb:60:in `main'
/home/splayd/splay/src/controller/lib/logd.rb:35:in `run'
/home/splayd/splay/src/controller/lib/logd.rb:34:in `initialize'
/home/splayd/splay/src/controller/lib/logd.rb:34:in `new'
/home/splayd/splay/src/controller/lib/logd.rb:34:in `run'
controller_fork.rb:55
controller_fork.rb:53:in `fork'
controller_fork.rb:53
controller_fork.rb:52:in `each'
controller_fork.rb:52

The controller itself does not crash, but we should maybe provide a less scary message to the administrator.

vschiavoni commented 9 years ago

In terms of usability, it might be also good to prevent further job submissions while the controller is in this state of 'reduced operability' mode. Jobs should be either refused or enqueued with some explanatory message sent back to the client.