better support for YAML

[AvdN]

ruamel.yaml is a super-set of PyYAML(both in that it supports 1.2 and 1.1, and that it has features to round-trip (load, modify, dump) data. anyconfig is currently not exploiting those features resulting in e.g. the restriction that output results are not ordered as indicated for the YAML backend.

There is a PyYAML backward compatible interface to ruamel.yaml, which you can get by using the RoundTripLoader and RoundTripDumper as Loader and Dumper:

    from ruamel.yaml import RoundTripLoader as Loader
    from ruamel.yaml import RoundTripDumper as Dumper

These are subclasses of the safe versions not of the (PyYAML default) unsafe versions.

In particular YAML mappings are loaded as a subclass of (ordered)dict and YAML sequences as a subclass of list. Therefor the key ordering of the original mappings is preserved and not sorted (as with PyYAML). These subclasses are CommentedMap and CommentedSeq. The subclassing also allows:

• preservation of comments in the original file
• preservation of style of each mapping or sequence (flow (JSON) style or block style) instead of PyYAML's all flow/all block/"block with leaf node flow" choice using the default_flow_style parameter
• preservation of anchor/alias names (probably not so important for configurations)
• preservation of scalar string style (single quotes, double quotes, literal block style, folded block style
• dumps to JSON and still be cast to dict resp. list
• a slightly enhanced indentation setting allow to indent the dash in lists/sequences

There is a newer API used as:

from ruamel.yaml import YAML
yaml = YAML()
yaml.preserve_quotes = True
yaml.indent(mapping=4, sequence=6, offset=3)
data = yaml.load(string_stream_Path)
# modify data
yaml.dump(data, stream_Path)

There are also other, less important features, only available through the new API. The major restriction in the new API is that print(yaml.dump(data)) is no longer possible. This was much abused by people not understanding the streaming nature of the YAML dumper (which is in exact contrast to the standard library JSON dumper, where dump() calls dumps() anyway)

Supporting roundtriploading certainly requires changes as at least CommentedSeq gets dumped with all its metadata by anyconfig. It is probably best to first split the functionality between PyYAML and ruamel.yaml based on whether the latter can be imported and then adapt the functionality from there on. As indicated the loaded data can be dumped to JSON, but I have not tested doing so with anyconfig, nor how this influence the other front-backend combinations in the matrix of supported formats.

[ssato]

Great thanks for your details!

• I think that to support a newer API of ruamel.yaml should not be difficult. I'll try to implement that first asap.
• To keep all metadata of CommentedMap and CommentedSeq after load should not be difficult also, I guess.
• Dump all metadata of CommentedMap and CommentedSeq might be hard. Like XML backend, it has to dump metadata in distinction from data.

[ssato]

Sorry to late.

I add and enable basic support of ruamel.yaml with some commits such as 90abb01 in the next branch.

ssato@x1-carbon-gen6% ls                                                                          ~/repos/public/github.com/ssato/python-anyconfig.git
AUTHORS.txt  LICENSE.MIT  MANIFEST.in  NEWS  README.rst  anyconfig  anyconfig.egg-info  build  dist  docs  pkg  setup.cfg  setup.py  tests  tox.ini
ssato@x1-carbon-gen6% bpython3
bpython version 0.17.1 on top of Python 3.7.1 /usr/bin/python3
>>> import sys, os, pprint
>>> sys.path.insert(0, os.curdir)
>>> import anyconfig
>>> anyconfig.backend.yaml.ryaml
<module 'anyconfig.backend.yaml.ruamel_yaml' from './anyconfig/backend/yaml/ruamel_yaml.py'>
>>> d = anyconfig.load("tests/res/00-scm.yml")
>>> d
CommentedMap([('type', 'object'), ('properties', CommentedMap([('name', CommentedMap([('type', 'string')])), ('a', CommentedMap([('type', 'integer')]))
, ('b', CommentedMap([('type', 'object'), ('properties', CommentedMap([('b', CommentedMap([('type', 'array'), ('items', CommentedMap([('type', 'integer
')]))]))]))]))]))])
>>> s = anyconfig.dumps(d, default_flow_style=False, ac_parser="yaml")
>>> pprint.pprint(s)
('type: object\n'

 'properties:\n'

 '  name:\n'

 '    type: string\n'

 '  a:\n'

 '    type: integer\n'

 '  b:\n'

 '    type: object\n'

 '    properties:\n'

 '      b:\n'

 '        type: array\n'

 '        items:\n'

 '          type: integer\n')
>>>

[ssato]

Updated. Now users can specify the backend parser explicitly with the commits such as 67b8600. If it's OK, I'll clean up the code and merge changes into the master branch.

ssato@x1-carbon-gen6% bpython3                                                                    ~/repos/public/github.com/ssato/python-anyconfig.git
bpython version 0.17.1 on top of Python 3.7.1 /usr/bin/python3
>>> import sys,os
>>> sys.path.insert(0, os.curdir)
>>> import anyconfig
>>> print(open("tests/res/00-scm.yml").read())
# comment-0
type: object
properties:
  # comment-1
  name:
    # comment-2
    type: string
  a:
    type: integer
  b:
    type: object
    properties:
      b:
        type: array
        items:
          type: integer
>>> d = anyconfig.load("tests/res/00-scm.yml", ac_parser="ruamel.yaml", typ="rt")
>>> d
CommentedMap([('type', 'object'), ('properties', CommentedMap([('name', CommentedMap([('type', 'string')])), ('a', CommentedMap([('type', 'integer')]))
, ('b', CommentedMap([('type', 'object'), ('properties', CommentedMap([('b', CommentedMap([('type', 'array'), ('items', CommentedMap([('type', 'integer
')]))]))]))]))]))])
>>> print(anyconfig.dumps(d, ac_parser="ruamel.yaml", explicit_start=True, indent=4))
---
type: object
properties:
    name:
        type: string
    a:
        type: integer
    b:
        type: object
        properties:
            b:
                type: array
                items:
                    type: integer
>>>

[ssato]

I've just released 0.9.8 contains this changes and fixes, and let this issue closed. Please let me know if any further issues found related to this.

Thanks again for your suggestion!

[gsemet]

Hi. I have configuration where is does not work (ruamel.yaml without pyyaml package)

Python 3.5.2 (default, Nov 12 2018, 13:43:14) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import anyconfig
>>> anyconfig.backend.yaml.ryaml
False
>>> dir(anyconfig.backend.yaml)
['PARSERS', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'absolute_import', 'ryaml']
>>> anyconfig.backend.yaml.PARSERS
[]
>>> import ruamel.yaml
>>> ruamel.yaml
<module 'ruamel.yaml' from '/home/username/projects/myproject/.venv/lib/python3.5/site-packages/ruamel/yaml/__init__.py'>

pip freeze shows ruamel.yaml==0.15.87 and anyconfig==0.9.8.

Installing pyyaml seem to enable the ruamel backend:

$ pip install pyyaml
$ python
>>> import anyconfig.backend.yaml
>>> anyconfig.backend.yaml.PARSERS
[<class 'anyconfig.backend.yaml.pyyaml.Parser'>, <class 'anyconfig.backend.yaml.ruamel_yaml.Parser'>]

[ssato]

@gsemet Thanks a lot for your (possible bug) report! I opened another issue for this and will look into it.

[ssato]

@gsemet FYI. https://github.com/ssato/python-anyconfig/issues/99#issuecomment-461838387

[gsemet]

look great thanks!