Make logging configuration files easier to use

[2e8d23da-bac4-412b-8dc9-27a278f5f59e]

BPO	6136
Nosy	@vsajip

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/vsajip' closed_at = created_at = labels = ['extension-modules', 'type-feature'] title = 'Make logging configuration files easier to use' updated_at = user = 'https://bugs.python.org/gjb1002' ``` bugs.python.org fields: ```python activity = actor = 'vinay.sajip' assignee = 'vinay.sajip' closed = True closed_date = closer = 'vinay.sajip' components = ['Extension Modules'] creation = creator = 'gjb1002' dependencies = [] files = [] hgrepos = [] issue_num = 6136 keywords = [] message_count = 16.0 messages = ['88467', '88738', '88763', '88772', '88820', '88865', '88959', '88963', '88968', '89045', '89050', '89063', '89161', '89170', '90531', '91087'] nosy_count = 2.0 nosy_names = ['vinay.sajip', 'gjb1002'] pr_nums = [] priority = 'normal' resolution = 'wont fix' stage = None status = 'closed' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue6136' versions = ['Python 2.7', 'Python 3.2'] ```

[2e8d23da-bac4-412b-8dc9-27a278f5f59e]

Recently tried to use the logging configuration file format as understood by logging.config.fileConfig() and found it very unwieldy for normal usage. I feel it needs to "scale down" better. Some thoughts:

a) It creates handlers whether they're needed or not. This means you cannot leave the handler sections present in your configuration file and just disable and enable the logger by changing the level, or your application will open the files/sockets etc. anyway, whether the loggers are enabled or not. This is bordering on being a bug rather than just an annoyance...

b) There is a lot of unnecessary cruft. For example it should be possible to eliminate the [loggers], [handlers] and [formatters] sections. I gather they're there due to a limitation of ConfigParser but I assume this is historical as I can see no good reason for it now. Also, entries should default sensibly, e.g. "qualname" should not be treated as vital but should default to the name in the section header.

c) I'd also suggest providing a newer, non-back-compatible format alongside the existing one that was less wordy for normal use. Get rid of the separate "handlers" and "formatters" and make each logger have a section of its own containing all "handling" and "formatting" information: most users aren't going to want to think about these things as separate entities and in any case there is the hierarchical mechanism to prevent too much copying.

In fact, I'd suggest extending the "basicConfig" idea to be able to call it on individual loggers, and build a new format around that, where each section is read and a logger created with the contents passed as keyword arguments to "basicConfig".

I might have time to do some of this myself but I want to be sure people think it's a good idea before investing time in it.

[vsajip]

Re. point (a): The configuration logic assumes that if you put things in the configuration file, you want them included - that means instantiating handlers, which will generally open their output files (uness the delay parameter is set) and sockets. If you use append semantics for files and configure levels as you want them, the files are opened but not written to (and closed when the handlers are closed or the application exits). This is not a bug, as it was designed to work like this.

Re. point (b): Agreed that the presence of the sections may not seem necessary as you could achieve the same with loggers=, handlers= etc. The sections were originally placed there as a place to hang other settings which could apply across loggers, handlers etc. - which never materialised. In any case, it's not a big problem and not worth breaking backward compatibility for. The qualname is not always the same as the value in the section header - see the documentation for an example of this, search for "compiler.parser". The values in e.g. the section header and the handlers= section are just names to allow the configuration to cross-refer different parts of the configuration - so you could use h1, h2 etc. as handler names.

Re. point (c): there are alternative ways of configuring logging; for one example, see http://www.red-dove.com/python_config.html and look at "logconfig.cfg" and "logconfig.py" in the linked tarball/zip.

[2e8d23da-bac4-412b-8dc9-27a278f5f59e]

a) What is the point of opening files that will never be written to and sockets that will never be used?

A large application might have a great many loggers for debugging which are off by default, and this means you have to either put up with lots of empty files being created all the time, or tell everyone that they need to change the configuration in two places each time they enable or disable a logger.

Logging config files need to be easy to tweak, even for people who aren't coders: it should be quick and obvious how to enable or disable a logger.

b) I don't see why making those sections optional would break backward compatibility. It would be easy to just silently ignore them if they were present (or call today's code that uses them).

I'm aware that "qualname" isn't always the same as the section name. My point is that it should not be *compulsory*, not that it shouldn't exist. It has an obvious default value so it's wrong to fail with a python stack if it isn't present.

c) I know there are other packages out there. I've been using log4py for years, which is now abandoned. But Python now has an official way to do logging and I think it should be more user-friendly for simple usage than it is. I can write my own config file format without too much difficulty but it seems a shame if everyone ends up doing this.

(The one you linked to seemed to have wider ambitions than logging and its format seemed even more unwieldy. Curly braces in Python?!)

[vsajip]

a) Loggers don't have a one-to-one correspondence to files that are opened, so I don't quite understand your point. Perhaps you are conflating loggers with handlers. The common way of using logging is to have a small number of file-based handlers and perhaps a socket handler and console handler, working with a potentially much larger number of loggers.

b) If you want to propose a patch (incl. tests and docs) which maintains backward compatibility, I'll certainly look at it.

the qualname *is* compulsory as it determines exactly which logger is instantiated. So I don't understand your statement.

c) Having used log4py (which I'm not familiar with), it may be that you have to consider that Python's logging package might do some things differently. You are of course free to use your own configuration format

• it's one of those areas where personal taste is more of a factor. I don't know of any alternative formats which have a lot of traction for configuring logging specifically, though the programmatic API for configuring logging is pretty simple and almost any other configuration approach could be made to fit.

It's funny to hear you comment on the inappropriateness of curly braces in Python, how do you manage without dict literals? ;-)

It's probably best not to continue this discussion on the issue tracker

• it's not the best place as it's not its intended purpose, and also the audience here will be much smaller than, say, comp.lang.python. Before rolling your own config format, I would suggest posting on c.l.py with your difficulties with logging configuration and see how others have coped with those or similar problems.

[2e8d23da-bac4-412b-8dc9-27a278f5f59e]

OK, I'll take point (c) further on comp.lang.python. As for the others, it would be useful if you could at least understand my points.

a) I'm aware that there isn't necessarily a one-to-one correspondence between loggers and files, don't see why that's relevant. I have no idea what the "common" way of using logging is, if there is one. Having lots of files in a logging set up doesn't seem to me in any way unusual, even if the number of loggers is potentially even larger.

The main question is the one I posed before: what is the point of opening files that will never be written to and sockets that will never be used? Or to put it another way, if I submitted a patch that only created handlers that were used by at least one logger, would you look at it?

b) "compulsory" as in "compulsory as an entry in the config file". The code would be

if "qualname" in opts:
    qn = cp.get(sectname, "qualname")
else:
    qn = log

To make life easier for those of us who don't see the point in naming loggers differently in the config file and in the code...

[vsajip]

Re point a) and opening files - why not use the "delay" parameter on FileHandlers, which delays opening until a message is actually logged? I can understand why one wouldn't want empty log files cluttering up the place, but the same argument doesn't apply to sockets. If you don't want an always-open connection, you could also use UDP. While the delay parameter takes care of files, why does it matter so much about sockets? Is it actually causing a practical problem, or does it just offend your aesthetic sensibilities?

If you submit a patch for fileConfig that only created handlers that were used by at least one logger, I would certainly look at it - I have both accepted and rejected numerous patches over the years. Would your patch also consider unused formatters? As the new behaviour could conceivably create a backward-compatibility problem (some users may be relying on the current behaviour, no matter how broken it seems to you), the new behaviour should be accessible through another config setting (e.g. ignore_unused) which defaults to False (the existing behaviour) if not specified in the config file.

Re point b), it only really covers the case where you have single-level loggers (qualname = word), and not actually a multi-level hierarchy (qualname = words-separated-by-dots). So I don't feel it's worth doing this, especially as there seems to be no widespread demand for this functionality.

I'll mark the issue as pending for now.

[2e8d23da-bac4-412b-8dc9-27a278f5f59e]

OK, I hadn't seen the "delay" parameter until now. I guess this is new in Python 2.6? Good that there is already a way to avoid lots of empty files, though it'll be a while before I can assume Python 2.6 unfortunately... that probably renders point (a) moot.

As for (b), do you not think a large number of users will not bother with the hierarchical aspect of the logging framework? I'd say you need to be pretty advanced/large scale before that becomes interesting.

I don't really understand why accepting such a patch would be a problem, as it's a simple change that wouldn't break backwards compatibility. It's surely got to be better than exiting with a python stack, which is what happens today.

(To give an idea of the bloat-factor, since migrating to the logging framework a typical configuration file for my system is now roughly 3 times the size it used to be for the same functionality)

[vsajip]

"As for (b), do you not think a large number of users will not bother with the hierarchical aspect of the logging framework? I'd say you need to be pretty advanced/large scale before that becomes interesting."

I disagree with this. The hierarchical set up is one of the key points of the approach. Casual users may not use it, but there are a lot of users who do - and not only on advanced or large-scale systems. Users wh o don't need hierarchies don't have to use them, but the system has to support those who find them useful.

"I don't really understand why accepting such a patch would be a problem, as it's a simple change that wouldn't break backwards compatibility. It's surely got to be better than exiting with a python stack, which is what happens today."

Any software might result in an exception if you don't interface to it in the expected and documented way. And, in my previous comment, I merely set some conditions which a patch would have to meet. Also, as it is definitely the case that many users use the hierarchical feature, even if you don't, I expect that the qualname setting will stay.

"(To give an idea of the bloat-factor, since migrating to the logging framework a typical configuration file for my system is now roughly 3 times the size it used to be for the same functionality)"

How big is that in KB? Disk space is pretty cheap these days. If it is a big problem for you, you can always try using your existing configuration format, reading it yourself and using the programmatic API to configure logging yourself. It should be a small bit of up-front work which can then be used on all your future projects.

[2e8d23da-bac4-412b-8dc9-27a278f5f59e]

Who said anything about not supporting users who want the hierarchy? I'm talking about making "qualname" optional, not removing it entirely! I even supplied the entirety of the code (all 4 lines of it) to be clear what I meant a few comments ago.

The files have gone from about 5kb to about 15kb. Of course diskspace is not a problem in itself, but these files need to be read and edited by non-coders and they're a lot scarier (and harder to tweak) than the old ones were. Basically they're full of abstract technical concepts ("qualname", "handler") and bits of python code to be eval'ed.

Yes, I can write my own format. But I can't see anything about my case which is unusual and I'm sure there must be a demand for something simpler, which is why I bothered to report this issue at all.

It's not particularly hard to find people out there raising this if you google a bit. But I shall raise this on comp.lang.python as you suggest.

[vsajip]

"Who said anything about not supporting users who want the hierarchy? I'm talking about making "qualname" optional, not removing it entirely!"

Ok, I see - sorry for the misunderstanding on my part.

"... these files need to be read and edited by non-coders and they're a lot scarier (and harder to tweak) than the old ones were. Basically they're full of abstract technical concepts ("qualname", "handler") and bits of python code to be eval'ed."

The configuration format is not (and was never) intended for non-technical end users. Because of the way it works, typos in the elements which are eval'd can throw exceptions, which is obviously undesirable. If you want such users to be able to change the logging configuration, I would advise you implement your own layer which does not use any technical terminology such as "handler" or "qualname", and from that layer either update the logging configuration via the logging API, or write a standard logging config file.

"It's not particularly hard to find people out there raising this if you google a bit."

Well, I did do a search for "+python +logging +config +problems" which didn't throw up much (other than this issue). I'd be grateful for some specific links to recent items which you have found.

[2e8d23da-bac4-412b-8dc9-27a278f5f59e]

If it was never intended for end users, then perhaps you could rename this request as "provide an end-user-friendly log configuration format" (that would live alongside the current one). It's hardly unusual that users should be able to troubleshoot systems themselves by requesting more (or different) logging, is it?

For example, log4py's format looked like this: [my_logger] LogLevel: Normal Target: my_filename.log

Pretty much anyone can edit a bunch of sections that look like that (with optional extras such as formatting where appropriate).

------------------------

As for mentions of this issue online, I linked to one such on my comp.lang.python posting. It isn't recent, but the points about the config file still apply as it hasn't substantially changed since then as far as I can see. 3 different users raise essentially the same point (and you also contributed to this thread)

http://www.mechanicalcat.net/richard/log/Python/Simple_usage_of_Python_s_logging_module

Here's another (only some of the discussion is relevant, but this exact point is raised by the original poster and agreed with by the responder):

http://markmail.org/message/amxocg2nskd72qaf

[vsajip]

"If it was never intended for end users, then perhaps you could rename this request as "provide an end-user-friendly log configuration format" (that would live alongside the current one). It's hardly unusual that users should be able to troubleshoot systems themselves by requesting more (or different) logging, is it?"

It's unlikely that this would be provided as part of the Python stdlib: requirements which are truly end-user-friendly are also likely to be application-specific. When the configuration format was devised I did seek feedback - but people with strong opinions on the issue told me they'd prefer to roll their own. I'm fine with that; you can't please all of the people all of the time and I sadly can't devote much time to this at the moment.

"For example, log4py's format looked like this: [my_logger] LogLevel: Normal Target: my_filename.log"

Looks fine for logging to files only. Does log4py support the wide range of output sinks that Python logging does? If not, then the more complicated logging configuration would seem reasonable as there are more options in Python logging.

"As for mentions of this issue online, I linked to one such on my comp.lang.python posting."

I'm not sure which posting you mean - I didn't see any recent postings by you on c.l.py (I look at it via the corresponding Google group).

Re. the links you posted - Richard Jones' thread came out shortly after logging was released and was largely about documentation and the need for casual-use functionality which was subsequently provided by basicConfig(). The MarkMail thread which you linked to doesn't appear to raise the same points you did - Steve Hindle's original post of 17 May 2006 mentions only in point 4 about "crufty config file syntax", with no specifics, and I can only find 3 messages in that thread - 2 by Steve Hindle and one by Aahz which simply suggests moving the discussion to c.l.py. I checked the Baypiggies archive on mail.python.org - same thing.

My suggestion would be - if you want to promote a simpler config file syntax for logging, implement one and upload it on PyPI, and post announcements on c.l.py and c.l.py.announce. If there is demand for this functionality, people can easily download your package from PyPI and use it. After all, logging.config is a separate sub-module and need not be loaded at all.

[2e8d23da-bac4-412b-8dc9-27a278f5f59e]

My comp.lang.python thread is here:

http://groups.google.com/group/comp.lang.python/browse_thread/thread/a0c35c3c9ad210a4

It was met by deafening silence though.

I don't see why a simple format would need to be customer-specific. log4py's isn't/wasn't and that worked just fine. It did support logging to more than files but its configuration file scaled down much more gracefully for simple usage, mostly because it didn't expose its internal design like the logging one does. It had only loggers instead of loggers/handlers/formatters.

But log4py is discontinued now as a project and I can't face maintaining my own copy of it any more.

I'm getting the feeling you're just trying to fob me off here. You dismiss the threads I found as being "mostly about other things" or "not mentioning specifics". That may be so, but the fact is, in those threads you have five other people expressing in one way or another that the configuration file is too complex - and I'm sure I could find more if you really want. If you prefer to ignore them and me there's not much point in discussing further.

I'm not demanding that you do this work. I'm simply trying to raise the issue and asking you to consider accepting such a patch if I or somebody else produce it.

[vsajip]

"It was met by deafening silence though."

Give it time - it's only been a few days. For some reason, Google Groups doesn't show your post in the first page of results when I search for logging configuration by date (i.e. most recent on top):

http://groups.google.com/group/comp.lang.python/search?q=logging+configuration&start=0&scoring=d

The same problem if I search for "logging configuration" the phrase - again, it doesn't show up.

However, if after a while there's still not much response, it would indicate that this is not perhaps such an important issue for the community as you feel it is. This doesn't stop you from rolling your own format, but gives less justification for adding a patch to the core stdlib.

"[log4py's] configuration file scaled down much more gracefully for simple usage, mostly because it didn't expose its internal design like the logging one does. It had only loggers instead of loggers/handlers/formatters."

Yes - Python logging is more complex because that's what's useful for developers. It's not really intended for end-users to change - in fact once something is in production, typically only levels need to change. This is surely editable by end users even with the existing config file format, as long as they're not too fazed by the other stuff which they don't need to touch. If they are - then a much simpler, application-specific, end-user friendly format seems more in order.

"But log4py is discontinued now as a project and I can't face maintaining my own copy of it any more."

What's to maintain? Python logging has been pretty stable now for a long time, and log4py being simpler shouldn't need any particular maintenance (since it has worked for you in the past).

"I'm getting the feeling you're just trying to fob me off here. You dismiss the threads I found as being 'mostly about other things' or 'not mentioning specifics'. That may be so, but the fact is, in those threads you have five other people expressing in one way or another that the configuration file is too complex - and I'm sure I could find more if you really want. If you prefer to ignore them and me there's not much point in discussing further."

I'm not trying to fob you off - I just stated what I found about those posts. The complaints you refer to were not specific enough to suggest improvements, and anyone can write comments about how crufty they think something is - it doesn't exactly tell the maintainer which direction they would like to go in. I'm not saying that applies to anything you personally have said - I'm referring to the comments in those posts you referred to. All of us in open source development have to balance a number of different issues and we all have different agendas and priorities. My position is that the logging configuration system, while not perfect, works and is used by quite a lot of people without problems. It's just not high on my list of priorities to tinker with the format, because the feedback I've had in the past is that those people who care a lot about configuration will roll their own anyway. I'm never going to be able to please them all, so why not focus my energies elsewhere?

"I'm not demanding that you do this work. I'm simply trying to raise the issue and asking you to consider accepting such a patch if I or somebody else produce it."

As I've said before, I've accepted numerous patches from numerous people in the past. You can confirm this from SVN where commit messages generally refer to issue numbers on this tracker. Clearly I can't make promises in advance to accept any future patch, but I've indicated where I'd set the bar (backward compatibility, doc changes, test changes) for a patch to be considered.

[vsajip]

There hasn't been any activity for around a month now - marking as pending, will close shortly unless there's some reason not to.

[vsajip]

No feedback, closing.