search

esgf2-us / esgf-1.5-design

Apache License 2.0
0 stars 1 forks source link

Indexing policy for legacy unreplicated data #8

Open lliming opened 2 months ago

lliming commented 2 months ago

LLNL's Solr index served as an index for several ESGF data nodes that didn't have their own index. Consequently, LLNL's index contains metadata for datasets that have never been stored at LLNL, ANL, or ORNL. Q: Should ANL's or ORNL's Phase I indices contain these entries? Q: Should the Phase II consolidated index contain these entries?

bstrdsmkr commented 2 months ago

ORNL believes these datasets have been replicated, does anyone have evidence to the contrary? If so, we'd like to get them replicated. This means the likely answer to both questions should be yes, though that might be a policy question for @climate-dude ?

sashakames commented 2 months ago

I think there might be some misunderstanding. These are datasets published by NASA, NOAA, CCCma (canadians) DIAS (Japanese) sites, Taiwan, and several of the Korean and Chinese sites. Not all these datasets are replicated. What is crucial are the records that are hosted in the LLNL Solr. We don't need multiple copies of the records migrated to all the DOE site indexes. It would probably be easiest to just migrate those along with the LLNL records when migration time comes. I will produce a list of dataset and file counts for the data nodes.