search

hpc / mpifileutils

File utilities designed for scalability and performance.
https://hpc.github.io/mpifileutils
BSD 3-Clause "New" or "Revised" License
164 stars 64 forks source link

add daos-preserve option for preserving daos metadata #486

Closed dsikich closed 3 years ago

dsikich commented 3 years ago

This option depends on the hdf5 library, and can only be used when moving data to or from a POSIX filesystem. If a user moves data from a POSIX DAOS Container to NFS or Lustre they can write the container properties and user attributes to an hdf5 file. When a user is ready to move their data back into a container they can pass the metadata file into the copy and the properties will be read from the hdf5 file. The container properties and user attributes are set on container creation in the case the the data is moving from a POSIX filesystem to a DAOS POSIX container.

If a user is moving data from one DAOS container to another the metadata is already preserved from the source to destination, and passing the option is not necessary.

This option can be used by setting --daos-preserve <path/to/use> where the filename will be appended to the end of the path.

Signed-off-by: Danielle Sikich danielle.sikich@intel.com

dsikich commented 3 years ago

@daltonbohning I've updated based on your comments, and my changes are in the "updates" commit so you can see what I've updated.

dsikich commented 3 years ago

@dsikich Functionally, this seems to be working (for some simple manual tests, anyway). I just had some comments about errors/docs/etc.

@daltonbohning thanks! I updated the docs and addressed your comments in my latest commit.

adammoody commented 3 years ago

Hi all, I'll take a look at this by Monday.

dsikich commented 3 years ago

Thanks @dsikich . I found a few minor things to check, otherwise this looks good to me.

Thanks @dsikich . I found a few minor things to check, otherwise this looks good to me.

thanks @adammoody I addressed your comments.