Feature request: Python string values

[cortesi]

I'm just about to release the first version of mitmproxy with config parsing based on ConfigArgParse. Thanks a bunch for your solid work.

Something we'd find immensely useful is the ability to specify values in key/value pairs as Python strings and triple-quoted multi-line strings, with Python-style escape sequences. This would make the config files for our tools hugely more powerful.

This would require converting the current parser to something more sophisticated. PyParsing has built-in definitions for quoted strings, and implementing the parser in it would not be very much work.

I don't have the bandwidth for this myself, but please let me know if a patch like this would be pulled, and I may get to it some time down the track.

[ghost]

That's great! Good luck with the release.

It would definitely be nice to add multiline value support. Regarding PyParsing, my hunch is it's better to keep heavy-duty parsing logic in user code both to keep configargparse simple and avoid 3rd party dependencies when possible.

Also, configargparse currently supports config values like:

x = this is a format string %s
y = this is a new-style format string: {players[0]}
z = testing \t escape \r\n chars    .

(which gets parsed as Namespace(x='this is a format string %s', y='this is a new-style format string: {players[0]}', z='testing \\t escape \\r\\n chars\t.'))

Are there other escape char behaviors you are looking for? What do you think about multilines implemented like the following pseudocode?

    if config value starts with """ or ''':
         append lines to value until you find """ or '''
    elif config value line ends with "\":
         next line is part of current line

[cortesi]

Hi there,

If you prefer to keep the package lean and dependencies minimal, pyparsing is not needed. I don't think we can get away with doing the parsing without keeping state. Consider, for instance, a string like this:

"""    \"""
string
"""

Here, the escaped quote means the string has not ended, but it would be detected as a string terminator by code like you suggested. You can come up with more complicated examples needing more and more state, until you get to the point where just using a simple recursive descent parser is the way to go. If you don't get to it, I'd be happy to write this - it shouldn't be much code.

Regarding Python-style escape sequences - to make this elegant, we'd need the un-escaping to happen within ConfigArgParse itself. The reason for this is that otherwise we would have to detect whether an argument comes from the command-line or from the config file, and treat these differently, because we need to expand escape sequences in one case but not the other. Ideally, the full set of Python string escape sequences would be available, including \xYY for hexadecimal specifcation. This is easily done once you have the string with string.decode('string_escape').

The gold standard for us would be if we could take an arbitrary Python string, repr it, and write it to a config file, and know we'd get the same string back. This opens up all sorts of possibilities for mitmproxy to modify its own config programmatically.

By the way, I've given this project a shout-out in my mitmproxy release announcement (http://corte.si/posts/code/mitmproxy/announce_0_11_2/index.html). More people should be using it!

[ghost]

Great points. I didn't know you could escape triple-quotes! What you're saying is reminding me of this paragraph on microsoft CSV format, so I agree 100% - a real solution is needed. If you know an easy way to replace the current config-file parser with a recursive-descent parser, especially using built-ins, I think that would be awesome. Fixing string decoding also sounds great.

Regarding dynamically generating config files, is there an API that would help with this, and also take care of #9 ? For example, would having methods like these be useful?

• get_default_config_file_paths() # returns list of file paths given to the ArgumentParser constructor
• get_user_config_file_paths() # can be called only after parse_args(). Returns dict that maps config file key to the user-specified path
• compute_config_file_contents() # can be called only after parse_args(). Returns dict of config keys mapped to values parsed from the command line, config file, env vars, defaults.

[bw2]

coalesced into #30