search

NASA-PDS / validate

Validates PDS4 product labels, data and PDS3 Volumes
https://nasa-pds.github.io/validate/
Apache License 2.0
16 stars 11 forks source link

VTool generates invalid JSON report for M20 mesh files #400

Open stirlingalgermissen opened 3 years ago

stirlingalgermissen commented 3 years ago

🐛 Describe the bug

VTool creates invalid JSON when processing M20 mesh files.

📜 To Reproduce

Steps to reproduce the behavior: Run VTool on the attached example dataset. Command is:

/usr/local/bin/validate/bin/validate /home/jpluser/workspace/bundles/old3/old/mars2020_ecam/data/sol/00009/ids/rdr/mesh/NLF_0009_0667756282M606RAS_N0030000NCAM05000_0A0295J03.oj.xml -c /home/jpluser/data_dictionaries/validate_config.txt --report-style json --add-context-product /home/jpluser/validation_context/mars2020/registered_context_products.json

Config is:

validate.schema = /home/jpluser/data_dictionaries/PDS4_CART_1G00_1950.xsd, /home/jpluser/data_dictionaries/PDS4_DISP_1G00_1500.xsd, /home/jpluser/data_dictionaries/PDS4_GEOM_1G00_1930.xsd, /home/jpluser/data_dictionaries/PDS4_IMG_1G00_1860.xsd, /home/jpluser/data_dictionaries/PDS4_IMG_SURFACE_1G00_1260.xsd, /home/jpluser/data_dictionaries/PDS4_MARS2020_1G00_1000.xsd, /home/jpluser/data_dictionaries/PDS4_MSN_1G00_1300.xsd, /home/jpluser/data_dictionaries/PDS4_MSN_SURFACE_1G00_1220.xsd, /home/jpluser/data_dictionaries/PDS4_MSSS_CAM_MH_1G00_1020.xsd, /home/jpluser/data_dictionaries/PDS4_MSSS_CAM_MH_1G00_1030.xsd, /home/jpluser/data_dictionaries/PDS4_PROC_1G00_1210.xsd
validate.schematron = /home/jpluser/data_dictionaries/PDS4_CART_1G00_1950.sch, /home/jpluser/data_dictionaries/PDS4_DISP_1G00_1500.sch, /home/jpluser/data_dictionaries/PDS4_GEOM_1G00_1930.sch, /home/jpluser/data_dictionaries/PDS4_IMG_1G00_1860.sch, /home/jpluser/data_dictionaries/PDS4_IMG_SURFACE_1G00_1260.sch, /home/jpluser/data_dictionaries/PDS4_MARS2020_1G00_1000.sch, /home/jpluser/data_dictionaries/PDS4_MSN_1G00_1300.sch, /home/jpluser/data_dictionaries/PDS4_MSN_SURFACE_1G00_1220.sch, /home/jpluser/data_dictionaries/PDS4_MSSS_CAM_MH_1G00_1020.sch, /home/jpluser/data_dictionaries/PDS4_MSSS_CAM_MH_1G00_1030.sch, /home/jpluser/data_dictionaries/PDS4_PROC_1G00_1210.sch

Output is:

/usr/local/bin/validate/bin/validate /home/jpluser/workspace/bundles/old3/old/mars2020_ecam/data/sol/00009/ids/rdr/mesh/NLF_0009_0667756282M606RAS_N0030000NCAM05000_0A0295J03.obj.xml -c /home/jpluser/data_dictionaries/validate_config.txt --report-style json --add-context-product /home/jpluser/validation_context/mars2020/registered_context_products.json
{
  "title": "PDS Validation Tool Report",
  "configuration": {
    "version": "2.0.6",
    "date": "2021-09-09T16:53:26Z"
  },
  "parameters": {
    "targets": "[file:/home/jpluser/workspace/bundles/old3/old/mars2020_ecam/data/sol/00009/ids/rdr/mesh/NLF_0009_0667756282M606RAS_N0030000NCAM05000_0A0295J03.obj.xml]",
    "userSpecifiedSchemas": "[file:/home/jpluser/data_dictionaries/PDS4_CART_1G00_1950.xsd, file:/home/jpluser/data_dictionaries/PDS4_DISP_1G00_1500.xsd, file:/home/jpluser/data_dictionaries/PDS4_GEOM_1G00_1930.xsd, file:/home/jpluser/data_dictionaries/PDS4_IMG_1G00_1860.xsd, file:/home/jpluser/data_dictionaries/PDS4_IMG_SURFACE_1G00_1260.xsd, file:/home/jpluser/data_dictionaries/PDS4_MARS2020_1G00_1000.xsd, file:/home/jpluser/data_dictionaries/PDS4_MSN_1G00_1300.xsd, file:/home/jpluser/data_dictionaries/PDS4_MSN_SURFACE_1G00_1220.xsd, file:/home/jpluser/data_dictionaries/PDS4_MSSS_CAM_MH_1G00_1020.xsd, file:/home/jpluser/data_dictionaries/PDS4_MSSS_CAM_MH_1G00_1030.xsd, file:/home/jpluser/data_dictionaries/PDS4_PROC_1G00_1210.xsd]",
    "userSpecifiedSchematrons": "[file:/home/jpluser/data_dictionaries/PDS4_CART_1G00_1950.sch, file:/home/jpluser/data_dictionaries/PDS4_DISP_1G00_1500.sch, file:/home/jpluser/data_dictionaries/PDS4_GEOM_1G00_1930.sch, file:/home/jpluser/data_dictionaries/PDS4_IMG_1G00_1860.sch, file:/home/jpluser/data_dictionaries/PDS4_IMG_SURFACE_1G00_1260.sch, file:/home/jpluser/data_dictionaries/PDS4_MARS2020_1G00_1000.sch, file:/home/jpluser/data_dictionaries/PDS4_MSN_1G00_1300.sch, file:/home/jpluser/data_dictionaries/PDS4_MSN_SURFACE_1G00_1220.sch, file:/home/jpluser/data_dictionaries/PDS4_MSSS_CAM_MH_1G00_1020.sch, file:/home/jpluser/data_dictionaries/PDS4_MSSS_CAM_MH_1G00_1030.sch, file:/home/jpluser/data_dictionaries/PDS4_PROC_1G00_1210.sch]",
    "severityLevel": "WARNING",
    "recurseDirectories": "true",
    "fileFiltersUsed": "[*.xml, *.XML]",
    "dataContentValidation": "on",
    "productLevelValidation": "on",
    "maxErrors": "100000",
    "registeredContextsFile": "/usr/local/bin/validate/resources/registered_context_products.json",
    "nonRegisteredContextsFile": "/home/jpluser/validation_context/mars2020/registered_context_products.json"
  },
  "productLevelValidationResults": [
    {
      "status": "PASS",
      "label": "gov.nasa.pds.validate.ValidateLauncher",
      "messages": [
        {
          "severity": "WARNING",
          "type": "warning.product_not_registered",
          "message": "Non-registered context products should only be used during archive development. All context products must be registered for a valid, released archive bundle. "
        }
      ],
      "fragments": [],
      "dataContents": []
    }.............................,
    {
      "status": "PASS",
      "label": "file:/home/jpluser/workspace/bundles/old3/old/mars2020_ecam/data/sol/00009/ids/rdr/mesh/NLF_0009_0667756282M606RAS_N0030000NCAM05000_0A0295J03.obj.xml",
      "messages": [],
      "fragments": [],
      "dataContents": []
    }
  ],
  "summary": {
    "totalErrors": 0,
    "totalWarnings": 1,
    "messageTypes": [
      {
        "messageType": "warning.product_not_registered",
        "total": 1
      }
    ]
  }
}

Note the periods in the JSON output

🕵️ Expected behavior

VTool returns a report with valid JSON

📚 Version of Software Used

/usr/local/bin/validate/bin/validate -V

gov.nasa.pds:validate
Version 2.0.6
Release Date: 2021-05-25 12:08:21

🩺 Test Data / Additional context

NLF_0009_0667756282M606RAS_N0030000NCAM05000_0A0295J03.zip

🖥 System Info

🦄 Related requirements

⚙️ Engineering Details

jordanpadams commented 3 years ago

@stirlingalgermissen are you piping this output to a file? or running with the output file flag? If the former, I think the latter make fix this? If not, this is a bug we will look into.

 -r,--report-file <file name>            Specify the report file name. Default
                                         is standard out.
stirlingalgermissen commented 3 years ago

Yes writing the report to a file works. This issue is specific to when the report is written to stdout

jordanpadams commented 3 years ago

@stirlingalgermissen copy that. i'm not sure this is something we are going to be able to fix, unfortunately. those dots to stdout are actually intentional for long-running validations to show a status to the user that validate hasn't frozen on them. i will keep this on the backburner in case we (or someone) comes up with a better idea on how to do this.

stirlingalgermissen commented 3 years ago

This breaks APPS processing for M20. Is it not possible to create valid json with stdout?

jordanpadams commented 3 years ago

@stirlingalgermissen sorry for the delay here. we will update the code to not output the status information when the report output == JSON, however, we recommend using the JSON report output as a workaround for the foreseeable future.

stirlingalgermissen commented 3 years ago

Thanks - yeah that's our plan for resolving this on our end