map/list with all optional fields

[drmull]

Hi,

When introducing 2.0 I changed the behavior of maps/lists with all optional keys as discussed in https://github.com/23andMe/Yamale/pull/60#issuecomment-521364154 . Now I stumbled on a project having a problem with this new behavior: https://github.com/canada-ca/ore-ero/issues/702

This gave me a bit of a bad feeling so I took another look at the issue and now I think I can fix it so it works like in pre 2.0 without too much trouble.

Its a bit of a mess since if we do this we make another backwards incompatible change in regards with 2.0 but since 2.0 haven't been out long maybe it could be viewed as an regression. Otherwise I suppose we have to bump the version again...

I don't know, what do you think @mildebrandt? Do you want a PR for this?

[mildebrandt]

I'm assuming this is the schema they're referring to:

---
schemaVersion: str()
adminCode: str()
projects:
  - contact:
      email: str()
      name: str(required=False)
      phone: str(required=False)
    category: str()
    date:
      closed: str()
      started: str()
      metadataLastUpdated: str()
    description:
      whatItDoes:
        en: str()
        fr: str()
      howItWorks:
        en: str(required=False)
        fr: str(required=False)
    name:
      en: str()
      fr: str()
    partners:
      - adminCode: str(required=False)
        email: str(required=False)
        name: str(required=False)
    tags:
      en:
        - str(required=False)
      fr:
        - str(required=False)
    team:
      en: str(required=False)
      fr: str(required=False)

I tried using this yaml document to test, and it failed with a list index out of range:

schemaVersion: v5
adminCode: c9
projects:
  - contact:
      email: f@n.com
    category: "asd"
    date:
      closed: "yesterday"
      started: "last week"
      metadataLastUpdated: "a year ago"
    description:
      whatItDoes:
        en: "en"
        fr: "fr"
      howItWorks: {}
    name:
      en: "sd"
      fr: "sd"
    partners: []
    tags:
      en: []
      fr: []
    team: {}

Error:

Validating /Users/chrism/workspace/Yamale/partnership1.yaml...

Error!
Schema: schemaPartnership.yaml
Data file: /Users/chrism/workspace/Yamale/partnership1.yaml
Traceback (most recent call last):
  File "/Users/chrism/workspace/Yamale/yamale/command_line.py", line 29, in _validate
    yamale.validate(schema, data, strict)
  File "/Users/chrism/workspace/Yamale/yamale/yamale.py", line 38, in validate
    schema.validate(d, path, strict)
  File "/Users/chrism/workspace/Yamale/yamale/schema/schema.py", line 57, in validate
    errors = self._validate(self._schema, data, path, strict)
  File "/Users/chrism/workspace/Yamale/yamale/schema/schema.py", line 99, in _validate
    strict)
  File "/Users/chrism/workspace/Yamale/yamale/schema/schema.py", line 145, in _validate_static_map_list
    key)
  File "/Users/chrism/workspace/Yamale/yamale/schema/schema.py", line 85, in _validate_item
    return self._validate(validator, data_item, path, strict)
  File "/Users/chrism/workspace/Yamale/yamale/schema/schema.py", line 99, in _validate
    strict)
  File "/Users/chrism/workspace/Yamale/yamale/schema/schema.py", line 145, in _validate_static_map_list
    key)
  File "/Users/chrism/workspace/Yamale/yamale/schema/schema.py", line 85, in _validate_item
    return self._validate(validator, data_item, path, strict)
  File "/Users/chrism/workspace/Yamale/yamale/schema/schema.py", line 99, in _validate
    strict)
  File "/Users/chrism/workspace/Yamale/yamale/schema/schema.py", line 145, in _validate_static_map_list
    key)
  File "/Users/chrism/workspace/Yamale/yamale/schema/schema.py", line 85, in _validate_item
    return self._validate(validator, data_item, path, strict)
  File "/Users/chrism/workspace/Yamale/yamale/schema/schema.py", line 99, in _validate
    strict)
  File "/Users/chrism/workspace/Yamale/yamale/schema/schema.py", line 145, in _validate_static_map_list
    key)
  File "/Users/chrism/workspace/Yamale/yamale/schema/schema.py", line 76, in _validate_item
    data_item = data[key]
IndexError: list index out of range

I'd like a PR for this IndexError first. And if the above schema and yaml validate properly, I'm fine with that functionality. What I don't want, for example, is projects.team or projects.partners to be optional. They are marked as required, so they should be required.

[mildebrandt]

Oh, and thanks for following up with other bugs in other projects. I really appreciate it! :)

[drmull]

Good find!

The index error is an easy fix. The yaml still won't validate because projects.partners is expected to be a single element list according to the schema. The schema should use the list() validator to get the intended behavior.

The reason it passed before 2.0 is because the non-strict behavior and the optional record behavior. No element passed the first would be validated.

I will submit a PR to fix the index error after the weekend.

[mildebrandt]

Ah, yes indeed! Thanks!

[drmull]

I created a new issue for the IndexError #66