search

rsyslog / rsyslog

a Rocket-fast SYStem for LOG processing
http://www.rsyslog.com
GNU Lesser General Public License v3.0
2.05k stars 655 forks source link

Declare obsolete legacy format as deprecated, begin work to retire it #2383

Open deoren opened 6 years ago

deoren commented 6 years ago

refs #2367

Expected behavior

  1. Use an obsolete legacy format directive in rsyslog.conf
  2. rsyslogd warns that the directive has been deprecated

Example from using the discard operator:

rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]

Actual behavior

Rsyslog does its best to honor both the advanced and obsolete legacy formats, all within the same configuration. This leads to unexpected results because the legacy directives and the modern configuration format parameters do not always mix well.

Assumptions / Observations

It is true that other projects have been criticized for dropping backwards compatibility. As others have said, making this change should not be done lightly (e.g. ,the support burden that this will place on devs and users both). That said, if the change was announced well ahead of the planned retirement and sufficient work was put in place ahead of time to aggressively declare the old format to be deprecated, the results should be manageable. The key would be preparation for this change and sufficient community involvement to get the word out.

Like other projects, putting a distant milestone in place should allow time to begin cleaning up old code that has been kept around and shied away from, likely exposing problems that would have otherwise gone unnoticed.

Recommendations

These points assume that there would not be a desire to maintain two separate stable branches.

  1. Plan to retire the obsolete legacy configuration format.

    • It doesn't have to be soon, not even within the next year or two, but having a future milestone to start working towards (and reference from the docs) would allow for a smooth transition.

  2. Update the docs to add multiple warnings that the obsolete legacy format has been officially deprecated and will cease to function in version X

    • perhaps "rsyslog v10", because 10 is the last version that will ever be needed

  3. Begin work to emit deprecation warnings for any use of the obsolete legacy format

    • Even if the plan is to never entirely drop support for the old format, updating rsyslog to apply this behavior change will make it more obvious that the old format is kept around only for the purpose of gradually migrating to the current format

  4. Create issues and attach to the appropriate milestone for replacing obsolete legacy directives that do not yet have an advanced format equivalent (e.g., $IncludeConfig as noted in #2151).

rgerhards commented 6 years ago

We should probably create a project for this. I have quickly glanced at the source, and there are still some places where we need obsolete format for e.g. some global settings. Also, we possibly need to update some (contributed) modules. More thoughts follow...

davidelang commented 6 years ago

As discussed earlier, there is 'advanced' 'legacy' and 'obsolete legacy'

your example of '~' vs 'stop' is a legacy things, not an 'obsolete legacy' thing (that particular warning should be removed)

Obsolete Legacy things are just about anything where you do a $something on one line and it affects future actions.

Things that are very clearly in the "obsolete legacy" group are things that only affect the next action (queue parameters are a perfect example)

Things that are not as clear are things that affect all future actions (default template and file permissions are good examples)

Anything that is a single line is almost always going to be in the "legacy" group rather than the "obsolete legacy" group.

For simple things, the legacy format is pretty clear, and has decades of documentation and training, there is no reason to complain about a distro doing

*.mail /var/log/mail

David Lang

davidelang commented 6 years ago

global setting probably belong in the "legacy" group rather than the "obsolete legacy" group.

There may be some to declare obsolete, but things that affect everything are not that confusing, it's the things that affect the next action that are extremely confusing

There are some legacy directives that should not be declared obsolete that don't interact well with the advanced directives (file permissions, file ownership, and default templates to use come to mind), but those are so useful with the legacy file outputs that I think they should not be declared "obsolete"

deoren commented 6 years ago

@davidelang: As discussed earlier, there is 'advanced' 'legacy' and 'obsolete legacy'

Sorry, it looks like I may have not been as consistent with my word choices as I meant to be.

I was attempting to use the format names from recent doc updates (that particular bit is still a work in progress):

I was suggesting that the obsolete legacy format should go away, the basic and advanced formats would remain, and rsyslog could be modified to emit warnings for all deprecated statements like it currently does for the discard operator.

your example of '~' vs 'stop' is a legacy things, not an 'obsolete legacy' thing (that particular warning should be removed)

For what it is worth, that warning is how I first learned about the stop statement replacing the discard operator a number of years ago (going from v5 to v7?). I'm not saying that deprecating the discard operator was the right decision (though 'stop' means more to me than the ~ operator), but I am saying that warnings like that are a useful vehicle. We can announce in the forums, mailing lists, the website and elsewhere, but warnings in the logs are at a times a lot closer to the user when you need to make sure that they know when something has been deprecated.

For simple things, the legacy format is pretty clear, and has decades of documentation and training, there is no reason to complain about a distro doing

*.mail /var/log/mail

I agree with all points regarding this format. What may be confusing the issue is the names of the formats. Going by current docs, it was recently renamed as the basic format.

I'm not a huge fan of how the format visually looks, but it is easy to pick up and work with.

rgerhards commented 6 years ago

I'm not a huge fan of how the format visually looks, but it is easy to pick up and work with.

yeah, and it is also omnipresent in tutorials, books, brains, ... thus we really need to support.

Looking at obsolete legacy statements for the globals. I think these should also not be used in the longer term. The problem with them is that you can set them multiple times. Experience has shown that this is done in included config files, and this leads to upredictable results. Most often (but not always!) the last one wins. So adding a new package which drops a config fragment can thrash an otherwise-running rsyslog config, and nobody knows why.

But I must admit that especially replacement of some of these statements needs very close review.

davidelang commented 6 years ago

On Mon, 15 Jan 2018, Deoren Moor wrote:

@davidelang: As discussed earlier, there is 'advanced' 'legacy' and 'obsolete legacy'

sorry for not remembering 'basic' and using the term 'legacy'

your example of '~' vs 'stop' is a legacy things, not an 'obsolete legacy' thing (that particular warning should be removed)

For what it is worth, that warning is how I first learned about the stop statement replacing the discard operator a number of years ago (going from v5 to v7?). I'm saying that deprecating the discard operator was the right decision (though 'stop' means more to me than the ~ operator), but I am saying that warnings like that are a useful vehicle. We can announce in the forums, mailing lists, the website and elsewhere, but warnings in the logs are at a times a lot closer to the user when you need to make sure that they know when something has been deprecated.

the thing is that while 'stop' is clearer than '~', there's no good reason to every depriciate '~' and make it stop working.

That warning was put in place at a time when we were thinking of eliminating all of the 'basic' (aka pre v6) config items. We changed our plans to instead avoid breaking configs and never went back and cleaned up the warnings.

some tool or config check option to rsyslog to flag 'basic' things is a reasonable idea. Warning on perfectly legitimate configs on every startup is not.

For simple things, the legacy format is pretty clear, and has decades of documentation and training, there is no reason to complain about a distro doing

*.mail /var/log/mail

I agree with all points regarding this format. What may be confusing the issue is the names of the formats. Going by current docs, it was recently renamed as the basic format.

yeah, sorry for not remembering that. In the posts I was replying to it looked like there were only two categories, the advanced and the obsolete.

David Lang

rgerhards commented 6 years ago

That warning was put in place at a time when we were thinking of eliminating all of the 'basic' (aka pre v6) config items. We changed our plans to instead avoid breaking configs and never went back and cleaned up the warnings.

nah, ~ is a kind of different beast. That single-char really caused a lot of trouble to make work with flex and bison (for good reasons if you look at compiler theory, btw). So it wouldn't hurt to get rid of some of these constructs. But I do not strongly feel we need it.

There is one other example: * which I think we already have depricated. IIRC, it was impossible to craft a useful grammar that supports it.

jay7x commented 6 years ago

I'd like to have v9 with no legacy directives support :)

rgerhards commented 6 years ago

We should probably begin with a switch that emits warning when such a construct is found. First by default off. Later on. Finally "does not work" ;-)

deoren commented 6 years ago

@rgerhards: We should probably begin with a switch that emits warning when such a construct is found. First by default off. Later on. Finally "does not work" ;-)

Perhaps triggered as well when running rsyslogd -NX (where 'X' is some number higher than the currently available numbers).

davidelang commented 6 years ago

On Tue, 30 Jan 2018, Yury Bushmelev wrote:

I'd like to have v9 with no legacy directives support :)

what is the benefit of breaking existing configs?

jay7x commented 6 years ago

what is the benefit of breaking existing configs?

As I may guess keeping legacy configuration directives may be hard to support from code perspective. That's why I told 'v9' :) But if I'm wrong I'm fine to keep legacy then.

davidelang commented 6 years ago

I think it's more of a support problem (people asking questions about the old stuff), but even that is far less than the support problem we would have if we broke existing configs :-)

deoren commented 6 years ago

@davidelang: I think it's more of a support problem (people asking questions about the old stuff), but even that is far less than the support problem we would have if we broke existing configs :-)

It would definitely had a significant support load for a time, so lots of PR work would need to be done to get the message out.

If a v9 were ever developed with the intention to drop support for the obsolete legacy configuration format, then stable versions prior to that major version change would need to start emitting info level (or whatever non-warning, non-error level is appropriate) messages for the older configuration settings that, while currently valid, would be entirely unsupported in the next major version.

At the early stages of that potential v9 it would also be useful to contribute patches to downstream projects to update the default /etc/rsyslog.conf and conf fragments to use equivalent configuration formats that would still be supported in the next major version.

davidelang commented 6 years ago

remember, it takes YEARS for distros to upgrade, we are still running into people running 7.x, 5+ years old.

so any migration pain will continue for at least 5 years. k

deoren commented 6 years ago

@davidelang: remember, it takes YEARS for distros to upgrade, we are still running into people running 7.x, 5+ years old.

so any migration pain will continue for at least 5 years. k

ACK. I suppose this is one reason (if at all possible/reasonable) to decide whether the obsolete legacy configuration format will be dropped and start emitting info level messages warning of the future end of support of that format. As you note, it will take a long while before the PR push is successful at getting the word out.

To be clear, I'm advocating implementing a "FYI" or warning mode in the current v8.x line long before any support is actually dropped in a future v9 series.

rgerhards commented 6 years ago

I admit that I personally do not see any real need to remove legacy format. It also conflicts with the design goal to make updates always work without surprises except when very important to do so.

How about this as a compromose: let's introduce a new global setting (e.g. supportObsoleteLegacyFormat), which then could be set to always, warning, and disallowed and have rsyslog react accordingly. We would start with a default of always and then move to warning at a much later stage as default and maybe in 5 to 10 years to disallowed. Doing this very fast would probably mean a lot of support effort.

But we would only switch default, so if people need to have the old style, they can switch the default however they like. Also, this means those that do not want to have obsolete legacy format could disable it immediately.

How does that sound?

rgerhards commented 6 years ago

side-note: v9 will never happen, with the new release scheme this makes no sense - everything is done incrementally. So we'll stay at 8.x forever.

davidelang commented 6 years ago

On Thu, 12 Apr 2018, Rainer Gerhards wrote:

I admit that I personally do not see any real need to remove legacy format. It also conflicts with the design goal to make updates always work without surprises except when very important to do so.

How about this as a compromose: let's introduce a new global setting (e.g. supportObsoleteLegacyFormat), which then could be set to always, warning, and disallowed and have rsyslog react accordingly. We would start with a default of always and then move to warning at a much later stage as default and maybe in 5 to 10 years to disallowed. Doing this very fast would probably mean a lot of support effort.

But we would only switch default, so if people need to have the old style, they can switch the default however they like. Also, this means those that do not want to have obsolete legacy format could disable it immediately.

How does that sound?

Sounds reasonable, but not a high priority to implement. Those people who want to drop the old format should be the ones to make the patch.

rgerhards commented 6 years ago

Not as a reply, but as something to consider: many distros provide config snippet dropins to /etc/rsyslog.d - all of those packages also need to be updated, which can be a real pain. If we fully drop support, we need someone who reviews all packages in all distros to see where this happens and work with them getting things changed.