Upload counts are different than the search result count in some profiles

[laurenwalker]

Example: https://knb.ecoinformatics.org/#profile/CN=SASAP,DC=dataone,DC=org

The discrepancy between the metadata docs in the search results and the metadata doc count in the uploads chart is a difference between querying for objects that are not obsoleted (i.e. we find the latest version for search results list) and objects that do not obsolete anything (i.e. we find the first version for the uploads chart) . This is likely happening when the first version of a metadata document is not publicly readable.

What we ideally would like to do is base the Uploads chart off of the first accessible version in a metadata doc version chain.

Im not sure this is something that can be fixed in MetacatUI right now. I think this is a larger issue caused by not tracking versions chains in the Solr index. There is no way for me to send a query that says, “give me the dateUploaded for the first accessible version of each Metadata doc.” To do that, we would need to traverse the obsolescence chain of many metadata pids, which could be hundreds to thousands of queries, depending on the profile. (for SASAP just 10 but for all of KNB, it would be up to ~4,000). One compromise we can do is change the Uploads chart to a Last Updated chart, which would always match the search result number exactly.

In addition, we could show a “First Upload” stat which just shows the first date anything was ever uploaded (earliest dateUploaded in Solr).

Unless I hear any objections, I'm going to change the chart to "Last Updated" and add the "First Upload" date to the profiles.

[laurenwalker]

@datadavev - You may want to look at this ticket since it will effect the DataONE profiles.

[laurenwalker]

Changes to the profiles are now in the 1.14 branch. Here is what the profiles look like now:

Group and User Profiles:

DataONE CN Profile:

Arctic Data Center Profile:

KNB Profile:

[laurenwalker]

More changes. The changes from what we have in production are:

• Removed the Uploads chart
• Added "Last Updated" charts, one for metadata and one for data objects. These chart the number of latest-version objects updated in each month. This is a temporary solution until we have some sort of server-side solution to caching statistics on all objects in the system, including private objects. When that is ready we can add the Uploads chart back.
• Added two quick stats to the top of the page - the total number of datasets (number of current-version metadata objects) and total size of current-version metadata and data files.

[laurenwalker]

The latest version of these changes to the profiles have been open to review here on Github and via email for 12 days, and I haven't heard any negative feedback, so I'm releasing these changes and updating KNB, Arctic Data Center, DataONE, etc.