BCO validation fails - Githubissues

osipovaak commented 5 years ago

Submitting this BCO:

I am using the python script provided with the BCO, and and get the following error: 'namespace' is a required property But the namespace is provided in the JSON above.

corburn commented 5 years ago

The json appears malformed:

python -m json.tool bco.json
Expecting ',' delimiter: line 1 column 2600 (char 2599)

The issue appears to be with how the value strings are being encoded:

{
  "param": "formatHIVE",
  "value": "{"23":61321,"31":60992} ",
  "step":"2"
}

osipovaak commented 5 years ago

still produces the same error

corburn commented 5 years ago

There is still a nested quote issue with the value properties:

{
  "param":"thrSNP",
  "value":"{"minFreqIgnoreSNP":0,"minFreqPercent":0} ",
  "step":"1"
},

The only change between the two documents is a namespace property in xref and a name property in pipeline_steps are now empty strings:

Before:

"xref":[{"namespace":"dfgdfg","name":"hgj","ids":[" "]
"pipeline_steps":[
  {"step_number":"1","name":"sdff",

After:

"xref":[{"namespace":" ","name":"hgj","ids":[" "],
"pipeline_steps":[
  {"step_number":"1","name":" ","description":"Engine to perform coverage analysis ,base-calling, SNP calling ",

osipovaak commented 5 years ago

The quoted issues appears when I post the text into github. All of the escaped quotes turn into regular quotes (\" -> "). All of the produced JSONs are proper JSONs

The issue is with the validation against the schema. A space and a string should pass the check, they are both of type string

corburn commented 5 years ago

You are getting the namespace error because it is present in only 1 of the 3 xref objects.

```json Text surrounded with triple backticks will be displayed as formatted. Optionally including the language will enable syntax highlighting. ```

'namespace' is a required property in the schema path None#properties/description_domain/properties/xref/items/required
'ids' is a required property in the schema path None#properties/description_domain/properties/xref/items/required
'access_time' is a required property in the schema path None#properties/description_domain/properties/xref/items/required
['Sequence Profiling Engine', ' '] is not of type 'string' in the schema path None#properties/description_domain/properties/xref/items/properties/name/type
'namespace' is a required property in the schema path None#properties/description_domain/properties/xref/items/required
'ids' is a required property in the schema path None#properties/description_domain/properties/xref/items/required
'access_time' is a required property in the schema path None#properties/description_domain/properties/xref/items/required
'1' is not of type 'integer' in the schema path None#properties/description_domain/properties/pipeline_steps/items/properties/step_number/type
Additional properties are not allowed ('sha1_chksum' was unexpected) in the schema path None#properties/description_domain/properties/pipeline_steps/items/properties/output_list/items/additionalProperties
'name' is a required property in the schema path None#properties/description_domain/properties/pipeline_steps/items/required
'2' is not of type 'integer' in the schema path None#properties/description_domain/properties/pipeline_steps/items/properties/step_number/type
Additional properties are not allowed ('sha1_chksum' was unexpected) in the schema path None#properties/execution_domain/properties/software_prerequisites/items/properties/uri/additionalProperties
Additional properties are not allowed ('sha1_chksum' was unexpected) in the schema path None#properties/io_domain/properties/output_subdomain/items/properties/uri/additionalProperties
{
  "bco_id": "https://comb.fda.gov/usr/ekaterina.minina/dna.cgi?cmdr=propget&ids=64330&mode=json",
  "bco_spec_version": "1.3",
  "checksum": "6be38c0abac80f6b99a205556f3ebb876ff919f3",
  "provenance_domain": {
    "name": " ",
    "version": " ",
    "review": [],
    "derived_from": "",
    "obsolete_after": "",
    "embargo": {
      "start_time": "",
      "end_time": ""
    },
    "created": "",
    "modified": "",
    "contributors": [],
    "license": " "
  },
  "usability_domain": [
    " "
  ],
  "description_domain": {
    "keywords": [
      " "
    ],
    "xref": [
      {
        "namespace": " ",
        "name": "hgj",
        "ids": [
          " "
        ],
        "access_time": "2001-12-12T05:00:00.000"
      },
      {
        "name": [
          "Sequence Profiling Engine",
          " "
        ]
      },
      {
        "name": " "
      }
    ],
    "platform": [
      "HIVE"
    ],
    "pipeline_steps": [
      {
        "step_number": "1",
        "name": " ",
        "description": "Engine to perform coverage analysis ,base-calling, SNP calling ",
        "version": "",
        "prerequisite": [],
        "input_list": [],
        "output_list": [
          {
            "uri": "https://comb.fda.gov/usr/ekaterina.minina/dna.cgi?cmd=objFile&ids=61321&filename=all_indels.dict",
            "access_time": "2019-06-03T15:44:18.000",
            "sha1_chksum": "",
            "filename": "all_indels.dict"
          },
          {
            "uri": "https://comb.fda.gov/usr/ekaterina.minina/dna.cgi?cmd=objFile&ids=61321&filename=FreqProfile.csv",
            "access_time": "2019-06-03T15:44:18.000",
            "filename": "FreqProfile.csv"
          },
          {
            "uri": "https://comb.fda.gov/usr/ekaterina.minina/dna.cgi?cmd=objFile&ids=61321&filename=SNPprofile.csv",
            "access_time": "2019-06-03T15:44:18.000",
            "filename": "SNPprofile.csv"
          }
        ]
      },
      {
        "step_number": "2",
        "description": "Regulatory Viral Mutation Comparator",
        "prerequisite": [],
        "input_list": [],
        "output_list": []
      }
    ]
  },
  "execution_domain": {
    "script": [],
    "script_driver": "HIVE",
    "software_prerequisites": [
      {
        "name": "HIVE-heptagon Profiler",
        "version": "1",
        "uri": {
          "uri": "https://comb.fda.gov/usr/ekaterina.minina/dna.cgi?cmd=dna-heptagon-new",
          "access_time": "2019-08-02T16:37:05.096",
          "sha1_chksum": "",
          "filename": ""
        }
      },
      {
        "name": "Viral Mutation Comparator",
        "version": " 1.1",
        "uri": {
          "uri": "https://comb.fda.gov/usr/ekaterina.minina/dna.cgi?cmd=reg_virMut",
          "access_time": "2019-08-02T16:37:07.759"
        }
      }
    ],
    "external_data_endpoints": [],
    "environment_variables": {}
  },
  "parametric_domain": [
    {
      "param": "countAAs",
      "value": "false ",
      "step": "1"
    },
    {
      "param": "cutEnds",
      "value": "do not cut ",
      "step": "1"
    },
    {
      "param": "disbalanceFR",
      "value": "0 ",
      "step": "1"
    },
    {
      "param": "entrCutoff",
      "value": "0 ",
      "step": "1"
    },
    {
      "param": "filterZeros",
      "value": "1 ",
      "step": "1"
    },
    {
      "param": "histograms",
      "value": "0 ",
      "step": "1"
    },
    {
      "param": "lenPercCutoff",
      "value": "no cutoff ",
      "step": "1"
    },
    {
      "param": "maxLowQua",
      "value": "no cutoff ",
      "step": "1"
    },
    {
      "param": "maxMissmatchPercCutoff",
      "value": "0 ",
      "step": "1"
    },
    {
      "param": "maxRptIns",
      "value": "3 ",
      "step": "1"
    },
    {
      "param": "minCover",
      "value": "10 ",
      "step": "1"
    },
    {
      "param": "thrSNP",
      "value": "{\"minFreqIgnoreSNP\":0,\"minFreqPercent\":0} ",
      "step": "1"
    },
    {
      "param": "minImportantEntropy",
      "value": "0 ",
      "step": "1"
    },
    {
      "param": "noise",
      "value": "{\"noiseFilterParams\":\"none\",\"noiseProfileMax\":0.01,\"noiseProfileResolution\":0.0001} ",
      "step": "1"
    },
    {
      "param": "SEARCHFORREPEATS",
      "value": "false ",
      "step": "1"
    },
    {
      "param": "algo",
      "value": "1547 ",
      "step": "2"
    },
    {
      "param": "service",
      "value": "viral-mutation-comp ",
      "step": "2"
    },
    {
      "param": "resubmitMode",
      "value": "false ",
      "step": "2"
    },
    {
      "param": "formatAlternatives",
      "value": "{\"21\":64037} ",
      "step": "2"
    },
    {
      "param": "baseline_keywords",
      "value": "{\"22\":\"baseline;screening\"} ",
      "step": "2"
    },
    {
      "param": "formatHIVE",
      "value": "{\"23\":61321,\"31\":60992} ",
      "step": "2"
    },
    {
      "param": "frequencyThreshold",
      "value": "0.1 ",
      "step": "2"
    },
    {
      "param": "AAconsensus",
      "value": "true ",
      "step": "2"
    },
    {
      "param": "algosToReport",
      "value": "1 ",
      "step": "2"
    },
    {
      "param": "enrichment",
      "value": "1 ",
      "step": "2"
    },
    {
      "param": "skipSynonymous",
      "value": "true ",
      "step": "2"
    },
    {
      "param": "collapseIdenticalAAMutations",
      "value": "false ",
      "step": "2"
    }
  ],
  "io_domain": {
    "input_subdomain": [],
    "output_subdomain": [
      {
        "mediatype": "text",
        "uri": {
          "uri": "https://comb.fda.gov/usr/ekaterina.minina/dna.cgi?cmd=objFile&ids=61321&filename=all_indels.dict",
          "access_time": "",
          "sha1_chksum": "",
          "filename": ""
        }
      },
      {
        "mediatype": "text",
        "uri": {
          "uri": "https://comb.fda.gov/usr/ekaterina.minina/dna.cgi?cmd=objFile&ids=61321&filename=FreqProfile.csv"
        }
      },
      {
        "mediatype": "text",
        "uri": {
          "uri": "https://comb.fda.gov/usr/ekaterina.minina/dna.cgi?cmd=objFile&ids=61321&filename=SNPprofile.csv"
        }
      },
      {
        "mediatype": "text",
        "uri": {
          "uri": "https://comb.fda.gov/usr/ekaterina.minina/dna.cgi?cmd=objFile&ids=64290&filename=input.csv"
        }
      }
    ]
  },
  "error_domain": {
    "empirical_error": {},
    "algorithmic_error": {}
  }
}

osipovaak commented 5 years ago

Throws error: Additional properties are not allowed ('sha1_checksum', 'filename', 'access_time' were unexpected). Failed validating 'additionalProperties' in schema['properties']['io_domain']['properties']['input_subdomain']['items']

But the io_domain spec points to definition of uri, which includes sha1_checksum', 'filename', and 'access_time' .

corburn commented 5 years ago

To match the schema, your input_subdomain should follow the same structure as your output_subdomain:

{ "uri": { "uri": "", "access_time": "", "sha1_checksum": "", "filename": "" } }

  "io_domain": {
    "input_subdomain": [
      {
        "uri": "https://comb.fda.gov/usr/ekaterina.minina/dna.cgi?cmd=objFile&ids=10512",
        "access_time": "",
        "sha1_checksum": "",
        "filename": ""
      },
      {
        "uri": "https://comb.fda.gov/usr/ekaterina.minina/dna.cgi?cmd=objFile&ids=9567"
      },
      {
        "uri": "https://comb.fda.gov/usr/ekaterina.minina/dna.cgi?cmd=objFile&ids=61320"
      }
    ],
    "output_subdomain": [
      {
        "mediatype": "text",
        "uri": {
          "uri": "https://comb.fda.gov/usr/ekaterina.minina/dna.cgi?cmd=objFile&ids=61320&filename=coverage_dict",
          "access_time": "",
          "sha1_checksum": "",
          "filename": ""
        }
      },
      {
        "mediatype": "text",
        "uri": {
          "uri": "https://comb.fda.gov/usr/ekaterina.minina/dna.cgi?cmd=objFile&ids=61320&filename=alignment.hiveal"
        }
      },
      {
        "mediatype": "text",
        "uri": {
          "uri": "https://comb.fda.gov/usr/ekaterina.minina/dna.cgi?cmd=objFile&ids=61321&filename=SNPprofile.csv"
        }
      }
    ]
  },

biocompute-objects / BCO_Documentation

BCO validation fails #68