Metadata extractor to report files from compressed archives

datalad / datalad-metalad

Next generation metadata handling

Other

11 stars 11 forks source link

Metadata extractor to report files from compressed archives #59

Open mih opened 3 years ago

mih commented 3 years ago

My concrete usecase is a single dataset comprised to large 7z archives (several hundred) that cumulatively content more than 100M files. Given that investigating the structure in this form would require a download (in my case 80TB), it would be instrumental to have a metadata extractor that can pull out such information and maintain it for a leaner access.

Interesting information:

path within the archive
uncompressed size

yarikoptic commented 3 years ago

https://github.com/datalad/datalad-neuroimaging/issues/32 "support DICOMs tarballs"
https://github.com/datalad/datalad/issues/2368 as a generalization of levels of metadata beyond dataset + file.