search

opennetworkinglab / flowvisor

FlowVisor - A network hypervisor
Other
164 stars 67 forks source link

Don't destroy derby.log at startup #229

Closed jbsbbn closed 10 years ago

jbsbbn commented 11 years ago

When FV starts up, it seems to overwrite derby.log, so we lose any log messages there that might help identify the cause of a crash. Could it be appended to, or rotated out of the way, or some such?

alshabib commented 11 years ago

Hmm that file is part of the internal derby db which I don't think has any special capabilities to move or append the file.

For now, could you move the file out of the way in your script and I'll push a change that'll do it at FV startup?

On May 9, 2013, at 8:44 AM, Josh Smift notifications@github.com wrote:

When FV starts up, it seems to overwrite derby.log, so we lose any log messages there that might help identify the cause of a crash. Could it be appended to, or rotated out of the way, or some such?

— Reply to this email directly or view it on GitHub.

gth828r commented 11 years ago

The thing is, the recent I2 outages seemed to point at the db closing, and it was suggested that we look at http://wiki.apache.org/db-derby/DebugPropertiesTmpl and tweak the settings to get more logging info. But we were never looking at the derby log in the first place, and that page includes config changes to the derby logging... when we did look at the derby log (without changing any config settings), we had already started flowvisor again after a crash, and the derby log had been overwritten.

It looks like /etc/flowvisor/derby.properties could just include a line that said: derby.infolog.append=true

and that would give us at least part of what we want. Does that seem like it would help, and does that seem like a reasonable thing to do as part of the installation of the FV package?

alshabib commented 11 years ago

I'd rather not make this change in the package because in the long run we don't want the derby.log file to grow all the time.

The derby.properties file is in /etc/flowvisor, could you simply modify it and add derby.infolog.append=true to it. I am assuming that you don't want this log appended all the time.

On May 9, 2013, at 9:11 AM, Tim Upthegrove notifications@github.com wrote:

The thing is, the recent I2 outages seemed to point at the db closing, and it was suggested that we look at http://wiki.apache.org/db-derby/DebugPropertiesTmpl and tweak the settings to get more logging info. But we were never looking at the derby log in the first place, and that page includes config changes to the derby logging... when we did look at the derby log (without changing any config settings), we had already started flowvisor again after a crash, and the derby log had been overwritten.

It looks like /etc/flowvisor/derby.properties could just include a line that said: derby.infolog.append=true

and that would give us at least part of what we want. Does that seem like it would help, and does that seem like a reasonable thing to do as part of the installation of the FV package?

— Reply to this email directly or view it on GitHub.

alshabib commented 11 years ago

In the long run, we could consider adding derby.log to the logrotate configuration present in flowvisor.

On May 9, 2013, at 9:24 AM, Ali Al-Shabibi alshabibi.ali@gmail.com wrote:

I'd rather not make this change in the package because in the long run we don't want the derby.log file to grow all the time.

The derby.properties file is in /etc/flowvisor, could you simply modify it and add derby.infolog.append=true to it. I am assuming that you don't want this log appended all the time.

On May 9, 2013, at 9:11 AM, Tim Upthegrove notifications@github.com wrote:

The thing is, the recent I2 outages seemed to point at the db closing, and it was suggested that we look at http://wiki.apache.org/db-derby/DebugPropertiesTmpl and tweak the settings to get more logging info. But we were never looking at the derby log in the first place, and that page includes config changes to the derby logging... when we did look at the derby log (without changing any config settings), we had already started flowvisor again after a crash, and the derby log had been overwritten.

It looks like /etc/flowvisor/derby.properties could just include a line that said: derby.infolog.append=true

and that would give us at least part of what we want. Does that seem like it would help, and does that seem like a reasonable thing to do as part of the installation of the FV package?

— Reply to this email directly or view it on GitHub.

gth828r commented 11 years ago

That would be fine with me. Appending to the log (and not doing anything else) is probably not the best path forward.

We tend to want to restore service as quickly as possible and then investigate (which sounds like a pretty normal operational procedure to me). The only real goal of this ticket is to identify that we want the derby log to be preserved when FV is restarted in hopes that it can help us identify some of the more recent crashes while still allowing us to quickly start up FV (and not scramble to make backups of log files first). However you all want to accomplish that is probably fine.