Open jggautier opened 7 months ago
I saw that it failed to harvest any records in prod. on Friday. I believe we need to remove all the SRDA records harvested from Datacite first, before trying to re-harvest their metadata into the same collection. I.e., we will need to delete that client, then re-create it from scratch. I unscheduled it for now, let's handle this after 6.1 is deployed (in the next couple of days, hopefully).
BTW, the 6.1 release patch that I'm deploying in prod. is already running on demo.
So you can see the redirects working for the records harvested from https://srda.sinica.edu.tw/
there:
https://demo.dataverse.org/dataverse/srda
Also, note that the old records harvested via Datacite in https://dataverse.harvard.edu/dataverse/srda_harvested are now redirecting properly. Let's coordinate about re-harvesting their records directly, later this week maybe?
Nice! Yeah anytime this week would be fine I think. If you don't mind, could you delete the client, harvest directly from them later this week, and let me know if that went okay? Or I could do it when you think it's best this week. I'll have blocks of time this Thursday and Friday afternoon that would work well.
A quick update: it took a good portion of the day on Fri. to delete all the datacite-harvested records. An attempt to re-harvest directly did not succeed, unfortunately. I did re-run that direct harvest on demo, just as a quick check, and it worked like a charm again there. I have no explanation of what's going on. Another production mystery that will need to be resolved.
https://dataverse.harvard.edu/dataverse/srda_harvested
I left the client scheduled, weekly, Sun. 10pm. Let's keep an eye on it and check what happens over the weekend.
Our contact at SRDA emailed us that he noticed that those 2926 records were harvested. That's one short of the 2,927 records in https://srda.sinica.edu.tw/oai_pmh/oai?verb=ListRecords&metadataPrefix=oai_dc
(which interestingly has no pagination, so it takes a while to load the metadata of all 2,927 records).
I let him know that you told Harvard Dataverse to try again this Sunday and that we'll see what happens then 🤞
2928 actually, 2 failed. I can give them the 2 identifiers - it looks like it may be an intermittent problem on their end.
I would maybe also ask them why they are formatting their identifiers as <dc:identifier>10.6141/TW-SRDA-AA000001-1</dc:identifier>
; instead of <dc:identifier>https://doi.org/10.6141/TW-SRDA-AA000001-1</dc:identifier>
or <dc:identifier>doi:10.6141/TW-SRDA-AA000001-1</dc:identifier>
? - that was the part that was causing most of the problems with their archive so far.
Those may all be moot-er points. On a closer look, I am very apprehensive about having their archive harvested regularly on a schedule. Their server does not appear to understand the incremental harvesting parameter (from=
). Sending ListIdentifiers
with from=2024-02-06
returns the same list of 2928 records. (Our Harvester doesn't use ListRecords
; it does ListIdentifiers
, then GetRecord
individually). The <datestamp>
entries don't seem to represent the modification times of the records - it is simply the current date/time for all the records.
I am not ok with re-harvesting ~3000 records from scratch every week. It would be preferable if they could fix this.
No pagination is another thing, yes; but less of a big deal compared to this.
Okay, in the email thread I asked about the identifier format and about the datestamp.
OK, great, saw your message. If they ask followup questions about the identifier format - that part is not a problem for us anymore; Dataverse was confused by it, but I worked around it. It's just something they may want to consider - since the identifiers in question are registered and resolvable DOIs, why not advertise them as such?
The "from=" parameter on the other hand is a fairly important thing.
@landreev, our contact, Chucky, wrote back that "regarding the date stamp issue, we have already instructed our IT personnel to update the date to reflect the latest data version release date in https://srda.sinica.edu.tw/oai_pmh/oai?verb=ListRecords&metadataPrefix=oai_dc," and that they'd make corrections to the identifier format (dc: identifier).
I'm not sure if it matters that they're referring to a ListRecords request and not the ListIdentifiers request that you said Dataverse uses. Should I ask them to clarify?
I scheduled it again for Sun. night, it's running now, we'll check on it tomorrow and see what it does. It's usually the same code underneath that drives the behavior of ListRecords and ListIdentifiers, when it comes to dates. I didn't think it was necessary to ask, unless we see something weird. I'm a little worried about how they phrased it - "update the date to reflect the latest data version". With "the date" possibly implying that it's the same date for all their records; which would again mean having to reharvest all their records every time. But let's see what/how many records we get tonight.
It did re-harvest all their records last night, i.e., still not incremental. I'm assuming we can just harvest their stuff like this once in a while manually for the time being, until they get the from= parameter to work on their end. One small improvement is that there were no broken records this time (there were 2 that failed to import the last time around).
Nice! Okay. I got the sense that they hadn't updated the dates, yet, and Chucky will email again when they have. I can reach out in a month, on Mar 26, to ask if they've made the change, too. If they have by then we can set the schedule to weekly. If they haven't yet, we can just run another manual harvest then.
How's that sound?
2024/03/12
I see different dates in the <datestamp>
of each record in https://srda.sinica.edu.tw/oai_pmh/oai?verb=ListRecords&metadataPrefix=oai_dc, but I'm not sure if that means that the datestamp issue has been resolved so that Harvard Dataverse can harvest incrementally and we can schedule weekly harvest.
I've emailed Chucky to ask if it's been resolved and if not, to email us when it has been so that I don't forget. See the email thread in RT
@landreev, Chucky's replied and I'm concerned that I've cause some confusion, maybe because I don't understand enough about how incremental harvesting works.
I've CC'ed you on the email thread. Could you reply when you get the chance?
@jggautier @landreev bumping this to find out status
What a coincidence! I was just writing the comment at https://github.com/IQSS/dataverse-pm/issues/171#issuecomment-2189129040 about following up with all of the repository contacts who've emailed me, although in SRDA's case I see that there are some issues unique to them to follow up on.
Looks like there are 2970 records in SRDA's OAI-PMH feed, and Harvard Dataverse has 2929 of them (https://dataverse.harvard.edu/dataverse/srda_harvested). I think we turned off scheduled harvesting - so Harvard Dataverse isn't trying to update harvested records each week - while indexing-related issues are being worked on, so it's been a while since Harvard Dataverse has tried to update the records it has from SRDA.
In late March our contact Chucky emailed:
In addition, I noticed that your colleague raised an issue regarding the ineffectiveness of the incremental harvesting parameter (from=) in the GitHub issue at https://github.com/IQSS/dataverse.harvard.edu/issues/243. I have already asked our IT team to look into this and resolve the issue.
I think the "from" parameter is working now for SRDA's OAI-PMH feed, at least from what I can tell when I try to use it, such as https://srda.sinica.edu.tw/oai_pmh/oai?verb=ListRecords&from=2024-02-25T00:00:00Z&metadataPrefix=oai_dc.
In addition to letting them know that we continue to work on indexing-related issues that are affecting harvesting, I can ask them to confirm that they've resolved this "from" parameter issue.
@cmbz and @landreev, what do you think?
@jggautier sounds good to me.
@landreev, Chucky from SRDA confirmed in the email that the "from" parameter is working now. 🎉
OK, I'll try to harvest from them and see if it is now incremental. But yes, scheduled regular harvesting is still off, pending fixing reindexing for harvested content, unfortunately. This is a blocker r/n for most issues that track harvesting content from specific remote archives.
After Dataverse v6.1 is applied to Harvard Dataverse, we'll need to re-harvest SRDA's metadata into the collection at https://dataverse.harvard.edu/dataverse/srda_harvested
OAI URL:
https://srda.sinica.edu.tw/oai_pmh/oai
Set: None
This is following the work discussed in the GitHub issue at https://github.com/IQSS/dataverse/issues/7624