asdf-format / asdf

ASDF (Advanced Scientific Data Format) is a next generation interchange format for scientific data
http://asdf.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
523 stars 57 forks source link

Investigate enabling `validate_checksum` as default `True` #1784

Open braingram opened 5 months ago

braingram commented 5 months ago

Description of the problem

To address issues of data corruption (like the pre-commit whitespace stripping that held up: https://github.com/spacetelescope/romancal/pull/1161) investigate enabling validate_checksum as default True.

If not already the behavior (I don't recall at the moment) the validation could occur only when the block is read (so a file opened with lazy_load=True and corrupt data would not raise an error until the corrupt array data is accessed). This may require some changes to how validate_checksum behaves and documentation updates. Depending on the impact of the change a deprecation period could be considered.