Implement ignore_errors for DictParser.Reader

mpsonntag commented 4 years ago

The XMLParser.Reader comes with an option to ignore errors when opening a file while notifying the user with all encountered odml format problems, enabling users to fix problematic odml files using the odml library instead of having to directly manipulate the file content.

This option has now been added to the DictParser.Reader as well, also allowing to open and fix problematic odml JSON and YAML files.

The PR

adds all necessary changes in the DictParser.Reader to enable ignoring errors, collecting warnings and printing corresponding notifications. These changes close issue #367.
changes the ODMLParser.Reader for JSON and YAML to now ignore_errors by default e.g. when using the odml.load function.
fixes a bug in format.revmap where the reverse mapping of an odml attribute would always return the case that the attribute is part of the format, even if it was not.
adds tests for parsing problematic JSON and YAML odml files.

Don't be afraid, most of the changes in this PR again increase the test coverage.

coveralls commented 4 years ago

Coverage increased (+0.2%) to 76.945% when pulling 90a5810be6534f81630189a4237001bae4bddecd on mpsonntag:dictParseIgnore into 206cbba5ad18d55049e7f06080741fe63503f53e on G-Node:master.

mpsonntag commented 4 years ago

I also don't particularly enjoy the way the file creation strings are formatted in these tests, but in most cases it makes it easier to see at one glance what is being tested and how. I am not dogmatic about it though, if there are any suggestions for a different setup, let me know! :)

achilleas-k commented 4 years ago

I also don't particularly enjoy the way the file creation strings are formatted in these tests, but in most cases it makes it easier to see at one glance what is being tested and how. I am not dogmatic about it though, if there are any suggestions for a different setup, let me know! :)

Could put them in a global fake constant in the file, right at the top, at indent level 0. Maybe something like:

file_content = """
Document:
  id: 82408bdb-1d9d-4fa9-b4dd-ad78831c797c
  sections:
  - id: d4f3120a-c02f-4102-a9fe-2e8b77d1d0d2
    name: sec

odml-version: '1.1'
""".strip()

That way you preserve the locality somewhat (it's in the same file), the content is indented as it will appear in the file without breaking flow, and the strip() takes care of removing leading and trailing newlines.

jgrewe commented 4 years ago

I would like that and it also gives you the option to call the variable e.g. something like: invalid_file_missing_id = "" or something similar

G-Node / python-odml

Implement ignore_errors for DictParser.Reader #386