Configparser fails when the .cfg file contains inline 'comments'

[ed635b22-fb28-46fc-90a6-7b82f1ca2166]

BPO	44892
Nosy	@terryjreedy, @ambv, @zware, @serhiy-storchaka, @uranusjr, @DiddiLeija

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['type-bug', '3.9', '3.10', '3.11', 'invalid', 'library'] title = "Configparser fails when the .cfg file contains inline 'comments'" updated_at = user = 'https://github.com/DiddiLeija' ``` bugs.python.org fields: ```python activity = actor = 'DiddiLeija' assignee = 'none' closed = True closed_date = closer = 'terry.reedy' components = ['Library (Lib)'] creation = creator = 'DiddiLeija' dependencies = [] files = [] hgrepos = [] issue_num = 44892 keywords = [] message_count = 10.0 messages = ['399415', '399562', '399580', '401081', '401207', '401209', '401211', '402621', '402625', '402635'] nosy_count = 6.0 nosy_names = ['terry.reedy', 'lukasz.langa', 'zach.ware', 'serhiy.storchaka', 'uranusjr', 'DiddiLeija'] pr_nums = [] priority = 'normal' resolution = 'not a bug' stage = 'resolved' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue44892' versions = ['Python 3.9', 'Python 3.10', 'Python 3.11'] ```

[ed635b22-fb28-46fc-90a6-7b82f1ca2166]

On the Pip GitHub issue tracker (https://github.com/pypa/pip/issues/10348), a user reported a strange behaviour when using a config file (setup.cfg) on its project.

The config file had a percentage character ("%") inside a commentary. But the module "configparser" failed with traceback:

configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(', found: "%' in string formatting

We consider that the character was badly recognized as a part of the file, when it was just a part of an inline comment.

Is there any way to fix this bug?

[ed635b22-fb28-46fc-90a6-7b82f1ca2166]

Lukasz Langa, I would like to know your opinion, as you are recognized as the "configparser" developer.

[ed635b22-fb28-46fc-90a6-7b82f1ca2166]

I would like to give a better explanation of the issue (maybe the previous explanation was unclear). A user reported to Pip that, when he used a "setup.cfg" file, the configparser module crashed (as I said above).

The file contained a percentage character (%) inside a commentary (which is expected to be ignored):

[flake8]
ignore =
        WPS323      #  percentage sign '%'

And, because pip uses configparser on those cases, any operation with pip fails.

So, I would like to know if we can do something to fix this bug (I noticed the place where it fails at https://github.com/python/cpython/blob/main/Lib/configparser.py#L442). If I can fill a PR to help, just tell me, OK?

[ed635b22-fb28-46fc-90a6-7b82f1ca2166]

Any commentaries about this issue?

[terryjreedy]

Diego, I repaired the defacement, which included removing you as nosy.

Relevant doc: https://docs.python.org/3/library/configparser.html#configparser.BasicInterpolation

To me, the following line implies that % may be used freely in comments. gain: 80%% # use a %% to escape the % sign (% is the only character ... However, IDLE uses .cfg files and configparser, and when I added #% to my C:/Users/Terry/.idlerc/config-main.cfg, IDLE startup fails with the following, confirming the behavior.

Traceback (most recent call last):
  File "f:\dev\3x\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\__main__.py", line 7, in <module>
    idlelib.pyshell.main()
    ^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\pyshell.py", line 1648, in main
    shell = flist.open_shell()
            ^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\pyshell.py", line 335, in open_shell
    self.pyshell = PyShell(self)
                   ^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\pyshell.py", line 898, in __init__
    OutputWindow.__init__(self, flist, None, None)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\outwin.py", line 79, in __init__
    EditorWindow.__init__(self, *args)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\editor.py", line 215, in __init__
    text['font'] = idleConf.GetFont(self.root, 'main', 'EditorWindow')
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\config.py", line 742, in GetFont
    bold = self.GetOption(configType, section, 'font-bold', default=0,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\config.py", line 229, in GetOption
    return self.userCfg[configType].Get(section, option,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\config.py", line 61, in Get
    return self.getboolean(section, option)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\configparser.py", line 829, in getboolean
    return self._get_conv(section, option, self._convert_to_boolean,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\configparser.py", line 809, in _get_conv
    return self._get(section, conv, option, raw=raw, vars=vars,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\configparser.py", line 804, in _get
    return conv(self.get(section, option, **kwargs))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\configparser.py", line 800, in get
    return self._interpolation.before_get(self, section, option, value,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\configparser.py", line 395, in before_get
    self._interpolate_some(parser, option, L, value, section, defaults, 1)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\configparser.py", line 442, in _interpolate_some
    raise InterpolationSyntaxError(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(', found: '%'

When changed one of the lines in my file see what a valid interpolation would do, font-size = 12 # for %(font)s I got

"Warning: config.py - IdleConf.GetOption - invalid 'int' value for configuration option 'font-size' from section 'EditorWindow': '12 # for source code pro'"

configparser interpreted the comment as part of the value, and started IDLE with the default size 10. The same happened when I used %%: the bad value was '12 # for %' And again with just '#'. Values read as string are also disabled by '#'. "name = Custom Dark # %(parenstyle)s" resulted in the default light theme. Conclusion: 1. Comments are currently worse than useless. 2. Test suite is deficient.

Diego, good PRs are welcome. Please sign CLA first. For this, the problem is comments not being recognized and stripped. Fix that and % in comments will not be an issue.

[terryjreedy]

Make PR branch from main, which I used for testing. I verified issue on 3.10 and 3.9, so we will backport.

[serhiy-storchaka]

By default configparser does not support inline comments. "# percentage sign '%'" is a part of value. If you want to support inline comments you should pass the inline_comment_prefixes argument.

But note that it can break parsing existing valid config files containing such prefixes. For example "background=#ffffff" or "path=C:\Python\bin;D:\Users\Me\bin" if use standard comment prefixes "#" and ";". This is why it is disabled by default.

[ed635b22-fb28-46fc-90a6-7b82f1ca2166]

Hi Terry, I didn't see your response. I think this won't be possible, taking in count the comment that Serhiy posted:

By default configparser does not support inline comments. "# percentage sign '%'" is a part of value. If you want to support inline comments you should pass the inline_comment_prefixes argument.

Maybe this is a reasonable behavior. What do you think about it?

[terryjreedy]

Please report back on the pip issue to read more of the doc, about inline comments in https://docs.python.org/3/library/configparser.html#customizing-parser-behaviour like I should have before writing what I did.

[ed635b22-fb28-46fc-90a6-7b82f1ca2166]

Sure, thanks!