We need to come up with a way to track provenance of data as it is processed or analyzed. pyUSID forced all Processes to write data into the same HDF5 file. Databases like DataFed have inherent provenance tracking capabilities. However, for those who do not want to use or do not have access to DataFed, we may need to build on the existing pyUSID methodology while reconciling the need to track analyses that may have multiple input datasets - something pyUSID did not really solve all that elegantly.
We need to come up with a way to track provenance of data as it is processed or analyzed. pyUSID forced all Processes to write data into the same HDF5 file. Databases like DataFed have inherent provenance tracking capabilities. However, for those who do not want to use or do not have access to DataFed, we may need to build on the existing pyUSID methodology while reconciling the need to track analyses that may have multiple input datasets - something pyUSID did not really solve all that elegantly.