search

digital-preservation / csv-validator

CSV Validation Tool and API (CSV Schema RI)
http://digital-preservation.github.io/csv-validator
Mozilla Public License 2.0
202 stars 54 forks source link

Performance throttling #228

Open DavidUnderdown opened 4 years ago

DavidUnderdown commented 4 years ago

Having been thinking about the possibility of using CSV Validator with DROID reports as the basis of ongoing integrity checking of digital archive holdings (in the case where an archive is in the early stages of digital work and doesn't have a digital preservation system as such) - see https://openpreservation.org/blog/2019/05/28/droid-report-as-basis-for-collection-integrity-checks/ It's occurred to me that implementing performance throttling as DROID does could be a useful feature to reduce the potential impact of checksum verification for larger collections.

adamretter commented 4 years ago

@DavidUnderdown for someone who is not aware of what DROID does for "Performance Throttling" could you describe what you would want in the CSV Validator?

Are you talking about limiting CPU, Memory, IO, power consumption, or something else? And how so, over time or to some pre-defined unit of measurement?

DavidUnderdown commented 4 years ago

Basically you can specify a delay between reading each file (in the GUI implemented via a slider in the bottom right of the interface)