This is a bad bug. By default, Silo prevents object overwrites. However, you can disable this by calling DBSetOverwrite() with a non-zero arg. On PDB driver, this means object data is just re-written. However, on HDF5 driver, instead of re-writting these objects, new datasets are written and the old ones orphaned. If there is a bug in a data producer, this can lead to unusually large files with no explanation as to cause.
-----------------------REDMINE MIGRATION-----------------------
This ticket was migrated from Redmine. As such, not all
information was able to be captured in the transition. Below is
a complete record of the original redmine ticket.
Ticket number: 2717
Status: New
Project: VisIt
Tracker: Bug
Priority: High
Subject: overwrite on HDF5 driver orphans old datasets
Assigned to: -
Category: -
Target version: -
Author: Mark Miller
Start: 11/22/2016
Due date:
% Done: 0%
Estimated time:
Created: 11/22/2016 11:28 am
Updated: 11/29/2016 02:00 pm
Likelihood: 3 - Occasional
Severity: 2 - Minor Irritation
Found in version: 4.8
Impact:
Expected Use:
OS: All
Support Group: Any
Description:
This is a bad bug. By default, Silo prevents object overwrites. However, you can disable this by calling DBSetOverwrite() with a non-zero arg. On PDB driver, this means object data is just re-written. However, on HDF5 driver, instead of re-writting these objects, new datasets are written and the old ones orphaned. If there is a bug in a data producer, this can lead to unusually large files with no explanation as to cause.
Comments:
I've developed an overall strategy on how to correct this.One issue is that I don't want the correction to significantly impact performance of normal usage. The solution I've developed involves one additional call to H5Lexist prior to any DBPutXXX operation in the HDF5 driver. This should have negligible impact on performance for normal usage.I have attached here an example of the modified driver code.Note that I've discovered other issues with this as well including even in the PDB driver. For example, any array of strings (which gets written as a single charactervalued dataset of semicolon separated strings), can wind up causing problems on readback. If the new array of strings contains more chars than the original, overwriting will be silently truncated. But, then when the string is parsed into the array of strings on read back, that parsing will return a number of strings that isn't consistent with the rest of the object and the read will likely fail or the object returned be corrupt. If the new array of strings contains fewer characters than the original, then a similar issue can occur except that it winds up being a catenation of the original string list and what is already in the file perhaps resulting in either incorrect individual strings or too many strings. The PDB driver doesn't nullterminate such strings whereas the HDF5 driver does. Modiying the PDB driver to null terminate its strings can fix this case.We can also add logic in the HDF5 driver to detect and prevent overwrite where new data size is bigger than original.But, all this represents more of change than I can make to the lib at this time. So, it needs to be scheduled for a future release. I am attaching example code I gave a user to deal with overwrite on multi-block objects directly via Silo's public interface
markcmiller86
commented
5 years ago
This is a bad bug. By default, Silo prevents object overwrites. However, you can disable this by calling DBSetOverwrite() with a non-zero arg. On PDB driver, this means object data is just re-written. However, on HDF5 driver, instead of re-writting these objects, new datasets are written and the old ones orphaned. If there is a bug in a data producer, this can lead to unusually large files with no explanation as to cause.
-----------------------REDMINE MIGRATION----------------------- This ticket was migrated from Redmine. As such, not all information was able to be captured in the transition. Below is a complete record of the original redmine ticket.
Ticket number: 2717 Status: New Project: VisIt Tracker: Bug Priority: High Subject: overwrite on HDF5 driver orphans old datasets Assigned to: - Category: - Target version: - Author: Mark Miller Start: 11/22/2016 Due date: % Done: 0% Estimated time: Created: 11/22/2016 11:28 am Updated: 11/29/2016 02:00 pm Likelihood: 3 - Occasional Severity: 2 - Minor Irritation Found in version: 4.8 Impact: Expected Use: OS: All Support Group: Any Description: This is a bad bug. By default, Silo prevents object overwrites. However, you can disable this by calling DBSetOverwrite() with a non-zero arg. On PDB driver, this means object data is just re-written. However, on HDF5 driver, instead of re-writting these objects, new datasets are written and the old ones orphaned. If there is a bug in a data producer, this can lead to unusually large files with no explanation as to cause.
Comments: I've developed an overall strategy on how to correct this.One issue is that I don't want the correction to significantly impact performance of normal usage. The solution I've developed involves one additional call to H5Lexist prior to any DBPutXXX operation in the HDF5 driver. This should have negligible impact on performance for normal usage.I have attached here an example of the modified driver code.Note that I've discovered other issues with this as well including even in the PDB driver. For example, any array of strings (which gets written as a single charactervalued dataset of semicolon separated strings), can wind up causing problems on readback. If the new array of strings contains more chars than the original, overwriting will be silently truncated. But, then when the string is parsed into the array of strings on read back, that parsing will return a number of strings that isn't consistent with the rest of the object and the read will likely fail or the object returned be corrupt. If the new array of strings contains fewer characters than the original, then a similar issue can occur except that it winds up being a catenation of the original string list and what is already in the file perhaps resulting in either incorrect individual strings or too many strings. The PDB driver doesn't nullterminate such strings whereas the HDF5 driver does. Modiying the PDB driver to null terminate its strings can fix this case.We can also add logic in the HDF5 driver to detect and prevent overwrite where new data size is bigger than original.But, all this represents more of change than I can make to the lib at this time. So, it needs to be scheduled for a future release. I am attaching example code I gave a user to deal with overwrite on multi-block objects directly via Silo's public interface