gwpy / gwsumm

Gravitational-wave interferometer summary information system
GNU General Public License v3.0
12 stars 23 forks source link

Ensure data integrity by verifying newly created file for corruption. #372

Closed iaraota closed 1 year ago

iaraota commented 1 year ago
  1. If corruption is detected, recover from backup. If no backup is found, delete the corrupted file.

  2. Initialize logger for consistent warning messages throughout the package

Testing

To intentionally corrupt the newly created file, the following code snippet can be used (before the committed script):

# Manually introduce corruption in the file by modifying the file structure
with open(outfile, 'r+b') as file:
    # Seek to a specific position in the file
    file.seek(0)

    # Write some random data to introduce corruption
    file.write(b'\x00\x01\x02\x03\x04')

To validate the changes, I executed the following steps:

  1. Confirmation that the pages are being run for the first time, which guarantees the absence of hdf5 files for backup.
  2. Execution of the code with the corruption script, resulting in the expected warning message: "Error occurred during file creation and no backup file is available."
  3. Removal of the corruption script, followed by code execution. This ensures that an uncorrupted file is saved
  4. Re-add the corruption script and rerunning the code. With the presence of backup files, the code successfully restores the backup, displaying the message: "Failed to write the {outfile}, restoring the backup...".
codecov[bot] commented 1 year ago

Codecov Report

Merging #372 (6eaa3c0) into master (7a3f4d3) will decrease coverage by 0.02%. The diff coverage is 40.91%.

@@            Coverage Diff             @@
##           master     #372      +/-   ##
==========================================
- Coverage   50.29%   50.28%   -0.02%     
==========================================
  Files          60       60              
  Lines        8669     8690      +21     
==========================================
+ Hits         4360     4369       +9     
- Misses       4309     4321      +12     
Flag Coverage Δ
Linux 50.28% <40.91%> (-0.02%) :arrow_down:
python3.10 50.28% <40.91%> (-0.02%) :arrow_down:
python3.7 49.47% <40.91%> (-0.02%) :arrow_down:
python3.8 50.28% <40.91%> (-0.02%) :arrow_down:
python3.9 50.28% <40.91%> (-0.02%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
gwsumm/archive.py 70.67% <40.91%> (-3.12%) :arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

eagoetz commented 1 year ago

@iaraota There are a few flake8 errors that you can easily fix. Check the CI test output. Also, is there are particular reason to add the logger here?