validate 2.1.0 indicates the table offset is not correct when validating a binary table inside a FITS

fran-vallejo commented 3 years ago

🐛 Describe the bug

validate raises and error message indicating the offset start for a binary table inside FITS file is incorrect. In principle the error looks like a false positive as the table starts in the byte position indicated by the label and the location in which the table should start according to validate is in the middle of the data.

📜 To Reproduce

Run the product attached with validate 2.1.0.

🕵️ Expected behavior

validate validates the attached test product and it does not raise the following error message: ERROR [error.table.fields_mismatch] table 1: The table offset 5760 for object 'METADATA' is invalid. The previously defined object ends at byte 17280

📚 Version of Software Used

validate 2.1.0

🩺 Test Data / Additional context

testdata.zip

fvallejo@BepiVirtualBox:~/Software$ validate-2.1.0/bin/validate --add-context-products /home/fvallejo/testdata/local_context_products.json --schema /home/fvallejo/testdata/PDS4_BC_1F00_1100.xsd /home/fvallejo/testdata/PDS4_GEOM_1F00_1910.xsd /home/fvallejo/testdata/PDS4_PSA_1F00_1300.xsd /home/fvallejo/testdata/PDS4_DISP_1F00_1500.xsd /home/fvallejo/testdata/PDS4_PDS_1F00.xsd --schematron /home/fvallejo/testdata/PDS4_PSA_1F00_1300.sch /home/fvallejo/testdata/PDS4_GEOM_1F00_1910.sch /home/fvallejo/testdata/PDS4_DISP_1F00_1500.sch /home/fvallejo/testdata/PDS4_BC_1F00_1100.sch /home/fvallejo/testdata/PDS4_PDS_1F00.sch --target /home/fvallejo/testdata/mix_raw_calib_mixs-c_sw_offset_table_20160301.xml 

PDS Validate Tool Report

Configuration:
   Version                       2.1.0
   Date                          2021-10-06T15:13:54Z

Parameters:
   Targets                       [file:/home/fvallejo/testdata/mix_raw_calib_mixs-c_sw_offset_table_20160301.xml]
   User Specified Schemas        [file:/home/fvallejo/testdata/PDS4_BC_1F00_1100.xsd, file:/home/fvallejo/testdata/PDS4_GEOM_1F00_1910.xsd, file:/home/fvallejo/testdata/PDS4_PSA_1F00_1300.xsd, file:/home/fvallejo/testdata/PDS4_DISP_1F00_1500.xsd, file:/home/fvallejo/testdata/PDS4_PDS_1F00.xsd]
   User Specified Schematrons    [file:/home/fvallejo/testdata/PDS4_PSA_1F00_1300.sch, file:/home/fvallejo/testdata/PDS4_GEOM_1F00_1910.sch, file:/home/fvallejo/testdata/PDS4_DISP_1F00_1500.sch, file:/home/fvallejo/testdata/PDS4_BC_1F00_1100.sch, file:/home/fvallejo/testdata/PDS4_PDS_1F00.sch]
   Severity Level                WARNING
   Recurse Directories           true
   File Filters Used             [*.xml, *.XML]
   Data Content Validation       on
   Product Level Validation      on
   Max Errors                    100000
   Registered Contexts File      /home/fvallejo/Software/validate-2.1.0/resources/registered_context_products.json
   Non Registered Contexts File  /home/fvallejo/testdata/local_context_products.json

Product Level Validation Results

  PASS: gov.nasa.pds.validate.ValidateLauncher
      WARNING  [warning.product_not_registered]   Non-registered context products should only be used during archive development. All context products must be registered for a valid, released archive bundle. 
        1 product validation(s) completed

  FAIL: file:/home/fvallejo/testdata/mix_raw_calib_mixs-c_sw_offset_table_20160301.xml
    Begin Content Validation: file:/home/fvallejo/testdata/mix_raw_calib_mixs-c_sw_offset_table_20160301.fits
      ERROR  [error.table.fields_mismatch]   table 1: The table offset 5760 for object 'METADATA' is invalid.  The previously defined object ends at byte 17280
    End Content Validation: file:/home/fvallejo/testdata/mix_raw_calib_mixs-c_sw_offset_table_20160301.fits
        2 product validation(s) completed

Summary:

  1 error(s)
  1 warning(s)

  Product Validation Summary:
    1          product(s) passed
    1          product(s) failed
    0          product(s) skipped

  Referential Integrity Check Summary:
    0          check(s) passed
    0          check(s) failed
    0          check(s) skipped

  Message Types:
    1            error.table.fields_mismatch
    1            warning.product_not_registered

End of Report
Completed execution in 3801 ms

🖥 System Info

OS: Ubuntu 20.04
Java: Oracle JDK 1.8.0_291

🦄 Related requirements

⚙️ Engineering Details

msbentley commented 3 years ago

Probably this is related to ticket https://github.com/NASA-PDS/validate/issues/361 ? In any case, this is a critical issue that prevents us updating to this version of validate.

jordanpadams commented 3 years ago

@qchaupds please add this to the top of your list once you have wrapped up your current task

fran-vallejo commented 3 years ago

Please see phe_misc_temperature_reference_20190524.zip as additional test data for this ticket as validate raises the same message when validating it. This product has fits as its data file format too, but the label declares a table delimited instead of an table binary.

jpl-jengelke commented 2 years ago

This passes tests with validate 2.2.0-SNAPSHOT with the error resolved.

viviant100 commented 2 years ago

@jordanpadams @tloubrieu-jpl this bug ticket refers to "validate 2.1.0", are you good with John tested with validate 2.2.0-SNAPSHOT with the error resolved?

jordanpadams commented 2 years ago

@viviant100 yes. The but happened with validate 2.1.0, but the fix happened int he latest SNAPSHOT

NASA-PDS / validate