BIBFRAME in Sinopia issue: Duplicate data "in Sinopia."

[gerontakos]

It appears description sets were loaded May 4 then re-loaded May 11. The May 4 do not display in Sinopia RTs but their IRIs still dereference. The new IRIs do display in Sinopia RTs and do dereference (as JSON-LD) so this appears to be OK; the new IRIs do not display in Sinopia RTs. So nearly-duplicate data is in fact live and dereferencing, but nobody will probably ever know about it. This issue has been created to open a discussion. Probably requires no action.

[mcm104]

Hm. I'll have to look into this, because I was only loading things into Sinopia Stage until 5/13... Are these stage IRIs, or did the data make it into production somehow?

[gerontakos]

Production. bf:AdminMetadata>bf:creationDate=5/11, bf:AdminMetadata>bf:changeDate=5/13.

[mcm104]

I'm realizing now some dates are going to be a little different because I was running the transform one step at a time...

• There are four records that have a creationDate of 5/10, as these were the test records I first loaded into production. The IRIs generated during this transform get recorded in a CSV file, and are treated as the "official" BIBFRAME IRIs for that dataset, meaning that in the future when the transform is run, for RDA resource X, the transform will go into that CSV file and see if X already has an "official" IRI -- if so, it will use that IRI for the output, maintain the original bf:creationDate, and add a new value for bf:changeDate. If not, it will generate a new IRI and create a bf:createDate.
• Everything else should have a creationDate of 5/11, after running the transform on all records on that day. The resources transformed during the 5/10 transform maintain their IRIs and get a new bf:changeDate of 5/11. The rest of the resources get a new IRI and a bf:creationDate of 5/10, and these are recorded as their "official" BIBFRAME IRIs. However, because I was still testing out the code, these records were not actually loaded into Sinopia on 5/11.
• The transform was run in its entirety on all records on 5/13. All resources had been assigned their "official" BIBFRAME IRIs on 5/10 or 5/11, so they maintained those IRIs, maintained a bf:creationDate of 5/10 or 5/11, and got a new bf:changeDate of 5/13 before being loaded into Sinopia. For the four 5/10 test records, this overwrote the data I had loaded on 5/10. However, for the rest of the records, this was actually the first time they were loaded into production.

Hopefully that explanation makes sense. The creation-5/11-change-5/13 makes sense to me, but anything in production from 5/4 does not... If you see this again, could you send me the IRI?

Edit: I'll also add that the reason for the "official" IRI system was specifically to avoid uploading the same data into production multiple times under different IRIs

[gerontakos]

That's an interesting solution: we assume the RDA identifiers are stable; they do not change. Create a table row each time a BF resource is derived from an RDA resource. When there is new data, or the transform is run again, the lookup of the stable RDA ID is performed to see if the old BF ID should be retained (and a new change date added) or, if the RDA ID is not found, a new BF ID should be created. That should work! I'll close this issue now and if we see a problem, we'll hopefully remember this as a potentially serious issue and possibly re-open.