RSS entries without sources?

efc commented 9 years ago

I've been playing around with the RSS feed and finding a number of entries without any source field in the description. See the list below, which is pull from RSS data with a lastBuildDate of 2014-12-04 14:48:09.978009.

The sources I found in this pull from RSS included {'dataone': 133, 'spdataverse': 2, 'upenn': 22, 'pushtest': 2, 'no source': 62, 'cmu': 2, 'stcloud': 1, 'mit': 25, 'trinity': 1}

That "no source" is from records that were missing the "source" element in their description. The "link" for these records are in the list below. They seem to be from quite a few of our providers (I'm not worried about the "anexample" one, which seems to be from push testing), most of whom also have records that included sources.

Any idea why these seem to be sneaking through without the source being attributed?

erinspace commented 9 years ago

Huh! This is puzzling indeed... will definitely look into it and see what we can find. Thanks for the detailed report!

efc commented 9 years ago

OK, I've found what I think is the culprit, but I still don't understand why this is happening.

It looks like the records without sources are duplicates of records with sources, but not every record has one of the duplicates. Why are we getting two records in the RSS feed for the same item? Are we somehow keeping outdated versions of some records where scrAPI ingests them? Why would we be creating outdated version of relatively new records? Do we have multiple (old and new) harvests running?

Here is an example (please forgive the slightly odd output, it comes from a python script I've been fiddling with):

title {} Two-phase westward encroachment of Basin and Range extension into the northern Sierra Nevada
link {} http://digitalcommons.trinity.edu/geo_faculty/6
description {} {
    "_id": "5480f601b5e9d7413507b733", 
    "attached": {
        "pmid": "5480f600b5e9d7413507b731"
    }, 
    "category": "metadata", 
    "collisionCategory": 1, 
    "contributors": [
        {
            "ORCID": "", 
            "email": "", 
            "family": "Surpless", 
            "given": "Benjamin", 
            "middle": "E", 
            "prefix": "", 
            "suffix": ""
        }, 
        {
            "ORCID": "", 
            "email": "", 
            "family": "Stockli", 
            "given": "Daniel", 
            "middle": "F", 
            "prefix": "", 
            "suffix": ""
        }, 
        {
            "ORCID": "", 
            "email": "", 
            "family": "Dumitru", 
            "given": "Trevor", 
            "middle": "A", 
            "prefix": "", 
            "suffix": ""
        }, 
        {
            "ORCID": "", 
            "email": "", 
            "family": "Miller", 
            "given": "Elizabeth", 
            "middle": "L", 
            "prefix": "", 
            "suffix": ""
        }
    ], 
    "dateCreated": "2002-01-01T08:00:00+00:00", 
    "dateUpdated": "2014-12-04T22:30:08+00:00", 
    "description": "", 
    "id": {
        "doi": "", 
        "serviceID": "oai:digitalcommons.trinity.edu:geo_faculty-1005", 
        "url": "http://digitalcommons.trinity.edu/geo_faculty/6"
    }, 
    "meta": {
        "docHash": "68d8ced20b61aff29cfce2f610b50fb2", 
        "uids": [
            "2fed17df83cf2aac3a34288994e48e15", 
            "8ee1920781067996fd16b942d7685c2c", 
            "2fed17df83cf2aac3a34288994e48e15", 
            "oaidigitalcommonstrinityedugeofaculty1005"
        ]
    }, 
    "properties": {
        "format": "application/pdf", 
        "identifiers": [
            "http://digitalcommons.trinity.edu/geo_faculty/6", 
            "http://digitalcommons.trinity.edu/cgi/viewcontent.cgi?article=1005&amp;context=geo_faculty"
        ], 
        "publisher": "Digital Commons @ Trinity", 
        "set_spec": "publication:geo_faculty", 
        "source": "Geosciences Faculty Research", 
        "type": "text"
    }, 
    "source": "trinity", 
    "tags": [
        "earth sciences"
    ], 
    "timestamps": {
        "consumeFinished": "2014-12-04T23:59:04.517867", 
        "consumeStarted": "2014-12-04T23:59:01.158067", 
        "consumeTaskCreated": "2014-12-04T23:59:00.088331", 
        "normalizeFinished": "2014-12-05T00:01:52.938798", 
        "normalizeStarted": "2014-12-05T00:01:52.936621", 
        "normalizeTaskCreated": "2014-12-05T00:00:33.726819"
    }, 
    "title": "Two-phase westward encroachment of Basin and Range extension into the northern Sierra Nevada"
}
author {} trinity
category {} earth sciences
guid {} oai:digitalcommons.trinity.edu:geo_faculty-1005
pubDate {} Thu, 04 Dec 2014 22:30:08 GMT

...and just following that in the RSS feed...

title {} Two-phase westward encroachment of Basin and Range extension into the northern Sierra Nevada
link {} http://digitalcommons.trinity.edu/geo_faculty/6
description {} {
    "_id": "5480f600b5e9d7413507b731", 
    "attached": {
        "cmids": [
            "5480f601b5e9d7413507b733"
        ]
    }, 
    "category": "metadata", 
    "collisionCategory": 1, 
    "contributors": [
        {
            "ORCID": "", 
            "email": "", 
            "family": "Surpless", 
            "given": "Benjamin", 
            "middle": "E", 
            "prefix": "", 
            "suffix": ""
        }, 
        {
            "ORCID": "", 
            "email": "", 
            "family": "Stockli", 
            "given": "Daniel", 
            "middle": "F", 
            "prefix": "", 
            "suffix": ""
        }, 
        {
            "ORCID": "", 
            "email": "", 
            "family": "Dumitru", 
            "given": "Trevor", 
            "middle": "A", 
            "prefix": "", 
            "suffix": ""
        }, 
        {
            "ORCID": "", 
            "email": "", 
            "family": "Miller", 
            "given": "Elizabeth", 
            "middle": "L", 
            "prefix": "", 
            "suffix": ""
        }
    ], 
    "dateCreated": "2002-01-01T08:00:00+00:00", 
    "dateUpdated": "2014-12-04T22:30:08+00:00", 
    "description": "", 
    "isResource": true, 
    "links": [
        {
            "longName": "Digital Commons@Trinity", 
            "shortName": "trinity", 
            "url": "http://digitalcommons.trinity.edu/geo_faculty/6"
        }
    ], 
    "meta": {
        "uids": [
            "8ee1920781067996fd16b942d7685c2c", 
            "2fed17df83cf2aac3a34288994e48e15"
        ]
    }, 
    "properties": {
        "format": "application/pdf", 
        "identifiers": [
            "http://digitalcommons.trinity.edu/geo_faculty/6", 
            "http://digitalcommons.trinity.edu/cgi/viewcontent.cgi?article=1005&amp;context=geo_faculty"
        ], 
        "publisher": "Digital Commons @ Trinity", 
        "set_spec": "publication:geo_faculty", 
        "source": "Geosciences Faculty Research", 
        "type": "text"
    }, 
    "tags": [
        "earth sciences"
    ], 
    "timestamps": {
        "consumeFinished": "2014-12-04T23:59:04.517867", 
        "consumeStarted": "2014-12-04T23:59:01.158067", 
        "consumeTaskCreated": "2014-12-04T23:59:00.088331", 
        "normalizeFinished": "2014-12-05T00:01:52.938798", 
        "normalizeStarted": "2014-12-05T00:01:52.936621", 
        "normalizeTaskCreated": "2014-12-05T00:00:33.726819"
    }, 
    "title": "Two-phase westward encroachment of Basin and Range extension into the northern Sierra Nevada"
}
category {} earth sciences
guid {} 5480f600b5e9d7413507b731
pubDate {} Thu, 04 Dec 2014 22:30:08 GMT

The differences between these two include:

Nonmatching Lines (first record; Line 5; second record; Line 5)
Nonmatching Lines (first record; Line 7; second record; Line 7:9)
Nonmatching Lines (first record; Line 52:56; second record; Line 54:61)
Extra lines in New before 63 in Old (first record; Line 58; second record; Line 63)
Extra lines in New before 64 in Old (first record; Line 60; second record; Line 64)
Nonmatching Lines (first record; Line 62:63; second record; Line 65)
Extra lines in New before 79 in Old (first record; Line 77; second record; Line 79)
Extra lines in New before 92 in Old (first record; Line 91; second record; Line 92)
Nonmatching Lines (first record; Line 93; second record; Line 93)

For example, all the timestamps match exactly, and these are clearly from the same source reporting the same resource because the links also match. But the normalized ("description") "_id" does not match, the first has a description/id and the second a description/links, the first has a "docHash" and more "uids" than the second, they have different "guid", and of course only the first has the description/source.

chrisseto commented 9 years ago

@efc @erinspace The records without sources are "resources." Resources are the collision detected aggregate metadata for each unique event. The isResource field is there for easily filtering out events/records or visa versa.

Resources will be updated when new information comes into the system referring to an event that already exists but the original events will not be altered.

So if we get three events:

Original Event from source 1
Original Event from source 2
Event Update from source 1

There will be four records in the system, one for each event, and then a single resource which contains the data from all the events merged into one.

efc commented 9 years ago

@chrisseto, what you explained to @erinspace and I about the cause of these "no source" records raises a number of questions for me. One of the fundamental tenets of the notification service is that we get one report of an event in, we send one report out. As envisioned, the NS also would not try to "update" records, but I think some of this collision detection work is changing that.

You have also used the word "resource" in a way I am not familiar with. I usually talk about "providers" (or "sources") sending us "reports" (or our "harvesting" from them and generating the "reports" from our harvests) about "research release events" that refer to "resources" created in the research ecosystem. With this in mind:

Provider or Source = St. Cloud State Digital Commons
Harvest = OAI-PMH results
Reports = Normalized records generated by harvester code
Resources = The items to which those reports refer

In your usage, "resource" is something else, it appears to be some kind of aggregation of information about related events.

Furthermore, the resource records you describe span multiple "sources" (or "providers").

CASE A: Lets say we harvest a provider twice, and the records they provide overlap somewhat. In that case me may get duplicate records in the OAI-PMH feed that should NOT become duplicate "event" reports, because they all refer to the same event, the deposit of a given paper.

CASE B: Lets say we harvest from a given provider at two different times and for their own reasons they have updated a record we got earlier. In this case the record may not be a duplicate record, but the record still does not refer to a new "event." Rather, it modifies our understanding of an event we already knew about and recorded. In that case we could decide to discard the updated information and stick with our previous understanding of the event, or to update our event "report" to incorporate the new information. But we still must understand this as a single "event" relating to a given provider. The clue that this is still the same event is probably the persistent link to the "resource" (my meaning of the word) itself.

CASE C: Lets say that we harvest a report of a given resource (my meaning again) from provider A, and then later get a report of what appears to be the same resource from provider B (for example, a preprint with two authors at two different institutions deposited to both institutional repositories). In this case we should maintain reports of each "event" (the appearance of the paper in A and also its appearance in B). This is NOT a duplicate "event" even though the paper may be the same.

From what you've described, right now the NS does not behave this way. In each of these three cases we would have two "event" records and one "resource" (in your sense of the word) record.

However, from the RSS feed and other "subscription" sources the notification system should actually provide in CASE A just one notification, in CASE B just one notification (either updated or not), and in CASE C two notifications (one from each provider).

It is OK if internally things are represented differently to assist in collision detection, but to the user of the service it has to appear that our getting a report about one "research release event" results in one "notification."

efc commented 9 years ago

By the way, the collision of the term "resource" turns out to be evident already in our glossary. This is just the first time I've come across it in real life.

efc commented 9 years ago

A few more questions about the "resource records." I'm not necessarily looking for quick answers, I just don't want to forget the questions!

How does "isResource" help filter these records, since more commonly I imagine we would need to find those things that are NOT resources. Since the "isResource" field is not even present in non-resource records, this makes filtering a bit awkward (though doable). If the dataset is to have records of multiple "types" mixed together, wouldn't it be more flexible to have a "type" field with entries like "event" and "resource"? That would also allow us to grow beyond two types, if we ever needed to.

I've found it is easier to just filter for items that have an actual source. That technique, however, leaves me wondering whether all resources are sourceless? And why don't the resource records have a "source" since they clearly come from a source?

How does additional information brought to the notification service though these resource records get incorporated into results? What kind of post-query processing do you envision to tie events and their halo of collision-inspired resources together?

JeffSpies commented 9 years ago

We're removing events with isResource from default output. As such, closing.

CenterForOpenScience / SHARE

RSS entries without sources? #115