Validator Experiment errors improvements

[juettemann]

--- experiment-no_EXPERIMENT_ONTOLOGY_CURIE.xml Align to other error messages by removing 'missing' array

{
  "Experiment-no_EXPERIMENT_ONTOLOGY_CURIE": {
    "error_type": "__prevalidation__",
      "errors": [
        "missing",
      [
        "experiment_ontology_curie"
      ]
      ],
      "ok": false,
      "version": "1.1"
  }
},

New (alternatively " is a required property" can be attached to be aligned to the json schema output):

{
  "Experiment-no_EXPERIMENT_ONTOLOGY_CURIE": {
    "error_type": "__prevalidation__",
    "errors": [
      "missing: experiment_ontology_curie"
    ],
    "ok": false,
    "version": "1.1"
  }
},

-- experiment-no_EXPERIMENT_TYPE__EXPERIMENT_ONTOLOGY_URI.xml Return array for 1.0. If possible, remove underscores.

{
  "Experiment-no_EXPERIMENT_TYPE__EXPERIMENT_ONTOLOGY_URI.xml": {
    "error_type": "__prevalidation__",
    "errors": "__mising_both__:__experiment_ontology_uri+experiment_type__",
    "ok": false,
    "version": "1.0"
  }
}

{
  "Experiment-no_EXPERIMENT_TYPE__EXPERIMENT_ONTOLOGY_URI.xml": {
    "error_type": "__prevalidation__",
    "errors": [
      "mising both: experiment_ontology_uri+experiment_type",
    ]
      "ok": false,
    "version": "1.0"
  }
}

-- experiment-no_LIBRARY_STRATEGY.xml Assertion error. Can this be moved to prevalidation?

-- experiment-no_MOLECULE__MOLECULE_ONTOLOGY_URI.xml Align to other error messages by removing 'missing' array. See above

{
  "Experiment-no_MOLECULE__MOLECULE_ONTOLOGY_URI": {
    "error_type": "__prevalidation__",
      "errors": [
        "missing",
      [
        "molecule"
      ]
      ],
      "ok": false,
      "version": "1.1"
  }
},

-- experiment-no_MOLECULEMOLECULE_ONTOLOGY_URIMOLECULE_ONTOLOGY_CURIE.xml Align to other error messages by removing 'missing' array. see above

  {
    "Experiment-no_MOLECULE__MOLECULE_ONTOLOGY_URI__MOLECULE_ONTOLOGY_CURIE": {
      "error_type": "__prevalidation__",
      "errors": [
        "missing",
      [
        "molecule",
      "molecule_ontology_curie"
      ]
      ],
      "ok": false,
      "version": "1.1"
    }
  },

-- experiment-only_LIBRARY_STRATEGY.xml Return array. If possible, remove underscores

{
  "Experiment-only_LIBRARY_STRATEGY": {
    "error_type": "__prevalidation__",
    "errors": "__mising_both__:__experiment_ontology_curie+experiment_type__",
    "ok": false,
    "version": "1.1"
  }
},
{
  "Experiment-only_LIBRARY_STRATEGY": {
    "error_type": "__prevalidation__",
    "errors": "__mising_both__:__experiment_ontology_uri+experiment_type__",
    "ok": false,
    "version": "1.0"
  }
}

New

{
  "Experiment-only_LIBRARY_STRATEGY": {
    "error_type": "__prevalidation__",
    "errors": [
      "mising both: experiment_ontology_uri+experiment_type"
    ],
    "ok": false,
    "version": "1.1"
  }
},
{
  "Experiment-only_LIBRARY_STRATEGY": {
    "error_type": "__prevalidation__",
    "errors":[
      "mising both: experiment_ontology_uri+experiment_type"
    ],
    "ok": false,
    "version": "1.0"
  }
}

experiment-empty_EXPERIMENT_ATTRIBUTES.xml Assertion error. Can this be moved to prevalidation?

-- experiment-no_EXPERIMENT_TYPEEXPERIMENT_ONTOLOGY_URIEXPERIMENT_ONTOLOGY_CURIE.xml Return array. If possible, remove underscores

{
  "Experiment-no_EXPERIMENT_TYPE__EXPERIMENT_ONTOLOGY_URI__EXPERIMENT_ONTOLOGY_CURIE": {
    "error_type": "__prevalidation__",
    "errors": "__mising_both__:__experiment_ontology_curie+experiment_type__",
    "ok": false,
    "version": "1.1"
  }
},
{
  "Experiment-no_EXPERIMENT_TYPE__EXPERIMENT_ONTOLOGY_URI__EXPERIMENT_ONTOLOGY_CURIE": {
    "error_type": "__prevalidation__",
    "errors": "__mising_both__:__experiment_ontology_uri+experiment_type__",
    "ok": false,
    "version": "1.0"
  }
}

experiment-version_2.0_complete.xml Should this not pass?

"IHECVersion2.0_test": {
  "error_type": "__prevalidation__",
  "errors": [
    "missing",
    [
      "chip_antibody",
      "chip_antibody_catalog",
      "chip_antibody_lot",
      "chip_antibody_provider",
      "chip_protocol",
      "chip_protocol_antibody_amount",
      "chip_protocol_bead_amount",
      "chip_protocol_bead_type",
      "chip_protocol_chromatin_amount",
      "chip_protocol_crosslink_time",
      "experiment_ontology_curie",
      "experiment_target_tf_modification",
      "extraction_protocol",
      "extraction_protocol_sonication_cycles",
      "extraction_protocol_type_of_sonicator",
      "library_generation_fragment_size_range",
      "molecule_ontology_curie"
    ]
  ],
  "ok": false,
  "version": "1.1"
}

[sitag]

@juettemann this is addressed by the same function as the other #92 . are you not using the json output and instead parsing this in shell with grep/awk?

[sitag]

@juettemann That doesn't pass because the prevalidation demanded everything (it was written before we required only a few). What all do you want to check in prevalidation (as in where do you want to pull those from the experiment schema) ?

[juettemann]

@sitag I am using the json output in the script, but the testing was done manually. With regards what should go into the pre-validation, I leave that ultimately up to you, see my comment in #92 . Consistent output and avoidance of assertion errors are main main concerns.

[sitag]

@juettemann @dzerbino I am trying to figure out what the new required fields for 1.1 experiment, but https://github.com/IHEC/ihec-ecosystems/blob/master/schemas/json/1.1/experiment.json still has only 3 required fields and https://github.com/IHEC/ihec-ecosystems/blob/master/docs/metadata/1.1/Ihec_metadata_specification.md hasn't been updated in a long time. Not sure where the 1.1 spec is.

Can't fix the prevalidation till it's clear.

[juettemann]

@dzerbino As far as I can see, these are the requirements for 1.1 (aka 2.0):

LIBRARY_STRATEGY EXPERIMENT_TYPE EXPERIMENT_ONTOLOGY_CURIE MOLECULE MOLECULE_ONTOLOGY_CURIE

ChIP-Seq: EXPERIMENT_TARGET_TF EXPERIMENT_TARGET_HISTONE

Removed: EXPERIMENT_ONTOLOGY_URI MOLECULE_ONTOLOGY_URI

Could you confirm / deny?

[dzerbino]

Hello everyone,

I don't think we're all on the same page. The required attributes for 1.1 (soon to be 2.0) are still lingering on a pull request: https://github.com/IHEC/ihec-ecosystems/pull/97

However, Sita did rightly point out that I had not updated the Markdown spec on my PR, which I have just corrected.

Cheers,

Daniel

[juettemann]

Closing this issue. Error messages look good now, and discussions regarding the version can be done at the PRs.

[sitag]

@juettemann You can't close this yet as prevalidation for experiment-version_2.0_complete.xml is still open

[juettemann]

@sitag Good point & catch. Do you want to keep it here, or shall I open a dedicated issue?

[sitag]

fixed for the xml in question. prevalidation will only check 5 core attributes required for all experiments, the two chip will be delegated to json schema

python -m version_metadata -overwrite-outfile -out:./delete_me -experiment version_metadata/examples/experiment-version_2.0_complete.xml

{
    "version_metadata/examples/experiment-version_2.0_complete.xml": [
        {
            "IHECVersion2.0_test": {
                "error_type": "__prevalidation__",
                "errors": [
                    "missing at prevalidate : experiment_ontology_curie",
                    "missing at prevalidate : molecule_ontology_curie"
                ],
                "ok": false,
                "version": "1.1"
            }
        },
        {
            "IHECVersion2.0_test": {
                "errors": [],
                "ok": true,
                "version": "1.0"
            }
        }
    ]
}

[juettemann]

Perfect. Unless there is anything else, can you close this issue @sitag?