yaml.parser.ParserError: while parsing a flow mapping

[MountainX]

I am just switching to configargparse as a replacement for configparser.ConfigParser. Thanks for providing this.

I'm following the instructions at https://pypi.org/project/ConfigArgParse/. In particular, I am attempting this:

dictionaries can be read in and converted to valid python dictionaries with PyYAML’s safe_load

I have regex expressions in my config file, similar to:

user_regex: {
  regex1 = '^\d{2}-[a-zA-Z0-9_]+',
  regex2 = '^\d{2}-[a-zA-Z0-9А-Яа-яёЁ_]+',
  regex3 = ...
  }

According to this answer, I guess yaml should handle regex: Python interpreting a Regex from a yaml config file - Stack Overflow

Given the error, which I will paste below, I also tried a few variations such as:

regex1 = "^\d{2}-[a-zA-Z0-9_]+",
regex1 = "^\\d{2}-[a-zA-Z0-9_]+",
regex1: "^\d{2}-[a-zA-Z0-9_]+",

etc.

My code is similar to what is provided in the example at https://pypi.org/project/ConfigArgParse/

import configargparse
import yaml
import re
from pathlib import Path

def define_args(parser):
      parser.add_argument('--user_regex', type=yaml.safe_load, required=False, help='some help')
      # a few other args added

p = Path('~/config/myscript.ini')
config_path = p.expanduser()
config_path.parent.mkdir(mode=0o644, parents=True, exist_ok=True)

parser = configargparse.ArgParser(
        default_config_files=[config_path],
        config_file_parser_class=configargparse.ConfigparserConfigFileParser)

define_args(parser)
parser.add('-c', '--config', required=False, is_config_file=True, help='config file path')
args = parser.parse_args()
print(args)
print(f'{args.path = }')
print("----------")
print(parser.format_help())
print("----------")
print(parser.format_values())

Below is the error I am seeing. I tried a few things, such as

Traceback (most recent call last):
  File "/workspace/myscsript.py", line 303, in <module>
    args = parser.parse_args()
  File "/usr/lib/python3.9/site-packages/configargparse.py", line 457, in parse_args
    args, argv = self.parse_known_args(
  File "/usr/lib/python3.9/site-packages/configargparse.py", line 632, in parse_known_args
    namespace, unknown_args = argparse.ArgumentParser.parse_known_args(
  File "/usr/lib/python3.9/argparse.py", line 1853, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python3.9/argparse.py", line 2062, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/usr/lib/python3.9/argparse.py", line 2002, in consume_optional
    take_action(action, args, option_string)
  File "/usr/lib/python3.9/argparse.py", line 1914, in take_action
    argument_values = self._get_values(action, argument_strings)
  File "/usr/lib/python3.9/argparse.py", line 2445, in _get_values
    value = self._get_value(action, arg_string)
  File "/usr/lib/python3.9/argparse.py", line 2478, in _get_value
    result = type_func(arg_string)
  File "/usr/lib/python3.9/site-packages/yaml/__init__.py", line 162, in safe_load
    return load(stream, SafeLoader)
  File "/usr/lib/python3.9/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/usr/lib/python3.9/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/usr/lib/python3.9/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/lib/python3.9/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/usr/lib/python3.9/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/lib/python3.9/site-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
  File "/usr/lib/python3.9/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/lib/python3.9/site-packages/yaml/parser.py", line 549, in parse_flow_mapping_key
    raise ParserError("while parsing a flow mapping", self.marks[-1],
yaml.parser.ParserError: while parsing a flow mapping
  in "<unicode string>", line 1, column 1:
    { regex1 = r'^\d{2}-[a- ...
    ^
expected ',' or '}', but got '{'
  in "<unicode string>", line 1, column 26:
    { regex1 = r'^\d{2}-[a-zA-Z0-9_]+', regex2 ...

Not sure if this issue is related to yaml, my config file formatting or configargparse. With configparser.ConfigParser, I did not need to use yaml. I was using a different approach with ExtendedInterpolation. Since I did not see you mention ExtendedInterpolation, but you did give a yaml example, I followed your lead.

[MountainX]

From what I have worked out so far, the solution seems to be:

• escape backslashes (\\)
• use colon instead of equal to separate keys and values (key: value)
• use double quotes instead of single quotes ("value")
• use the braces and commas as shown in the example on the configargparse help page.

This is working for me now:

user_regex: {regex1: "^\\d{2}-[a-zA-Z0-9_]+",
  regex2: "^\\d{2}-[a-zA-Z0-9А-Яа-яёЁ_]+",
  regex3: ...}

However, it is very sensitive and it throws exceptions if everything is not just right.

Is there any alternative for reading a set of user-defined keys from the config file? When yaml is not used, everything is a lot easier and more robust it seems.

[bw2]

I'm guessing the underlying issue is that configargparse works by converting all config file values to command line args before relying on argparse to parse them - and all the special chars are causing problems in argparse rather than in the parse-yaml step. I'm surprised your use-case is working at all, and I'm afraid a true fix would require modifying configargparse to bypass command line args, and have options that are only settable via config file. My initial assumption was that these types of use-cases - where values are complex enough to make it difficult to specify them on the command line - would be better handled by directly reading a config file, without using configargparse (at least for those values).

[MountainX]

I'm guessing the underlying issue is that configargparse works by converting all config file values to command line args before relying on argparse to parse them - and all the special chars are causing problems in argparse rather than in the parse-yaml step. I'm surprised your use-case is working at all, and I'm afraid a true fix would require modifying configargparse to bypass command line args, and have options that are only settable via config file. My initial assumption was that these types of use-cases - where values are complex enough to make it difficult to specify them on the command line - would be better handled by directly reading a config file, without using configargparse (at least for those values).

Thanks for sharing your recommendations. I appreciate your work on this package. It's very useful.