search

IQSS / dataverse.harvard.edu

Custom code for dataverse.harvard.edu and an issue tracker for the IQSS Dataverse team's operational work, for better tracking on https://github.com/orgs/IQSS/projects/34
5 stars 1 forks source link

Harvest metadata from SRDA #243

Open jggautier opened 7 months ago

jggautier commented 7 months ago

After Dataverse v6.1 is applied to Harvard Dataverse, we'll need to re-harvest SRDA's metadata into the collection at https://dataverse.harvard.edu/dataverse/srda_harvested

OAI URL: https://srda.sinica.edu.tw/oai_pmh/oai

Set: None

This is following the work discussed in the GitHub issue at https://github.com/IQSS/dataverse/issues/7624

landreev commented 7 months ago

I saw that it failed to harvest any records in prod. on Friday. I believe we need to remove all the SRDA records harvested from Datacite first, before trying to re-harvest their metadata into the same collection. I.e., we will need to delete that client, then re-create it from scratch. I unscheduled it for now, let's handle this after 6.1 is deployed (in the next couple of days, hopefully).

landreev commented 7 months ago

BTW, the 6.1 release patch that I'm deploying in prod. is already running on demo. So you can see the redirects working for the records harvested from https://srda.sinica.edu.tw/ there: https://demo.dataverse.org/dataverse/srda

landreev commented 7 months ago

Also, note that the old records harvested via Datacite in https://dataverse.harvard.edu/dataverse/srda_harvested are now redirecting properly. Let's coordinate about re-harvesting their records directly, later this week maybe?

jggautier commented 7 months ago

Nice! Yeah anytime this week would be fine I think. If you don't mind, could you delete the client, harvest directly from them later this week, and let me know if that went okay? Or I could do it when you think it's best this week. I'll have blocks of time this Thursday and Friday afternoon that would work well.

landreev commented 7 months ago

A quick update: it took a good portion of the day on Fri. to delete all the datacite-harvested records. An attempt to re-harvest directly did not succeed, unfortunately. I did re-run that direct harvest on demo, just as a quick check, and it worked like a charm again there. I have no explanation of what's going on. Another production mystery that will need to be resolved.

landreev commented 7 months ago

Screen Shot 2024-02-05 at 10 29 10 PM

https://dataverse.harvard.edu/dataverse/srda_harvested

I left the client scheduled, weekly, Sun. 10pm. Let's keep an eye on it and check what happens over the weekend.

jggautier commented 7 months ago

Our contact at SRDA emailed us that he noticed that those 2926 records were harvested. That's one short of the 2,927 records in https://srda.sinica.edu.tw/oai_pmh/oai?verb=ListRecords&metadataPrefix=oai_dc (which interestingly has no pagination, so it takes a while to load the metadata of all 2,927 records).

I let him know that you told Harvard Dataverse to try again this Sunday and that we'll see what happens then 🤞

landreev commented 7 months ago

2928 actually, 2 failed. I can give them the 2 identifiers - it looks like it may be an intermittent problem on their end. I would maybe also ask them why they are formatting their identifiers as <dc:identifier>10.6141/TW-SRDA-AA000001-1</dc:identifier>; instead of <dc:identifier>https://doi.org/10.6141/TW-SRDA-AA000001-1</dc:identifier> or <dc:identifier>doi:10.6141/TW-SRDA-AA000001-1</dc:identifier>? - that was the part that was causing most of the problems with their archive so far.

landreev commented 7 months ago

Those may all be moot-er points. On a closer look, I am very apprehensive about having their archive harvested regularly on a schedule. Their server does not appear to understand the incremental harvesting parameter (from=). Sending ListIdentifiers with from=2024-02-06 returns the same list of 2928 records. (Our Harvester doesn't use ListRecords; it does ListIdentifiers, then GetRecord individually). The <datestamp> entries don't seem to represent the modification times of the records - it is simply the current date/time for all the records. I am not ok with re-harvesting ~3000 records from scratch every week. It would be preferable if they could fix this. No pagination is another thing, yes; but less of a big deal compared to this.

jggautier commented 7 months ago

Okay, in the email thread I asked about the identifier format and about the datestamp.

landreev commented 7 months ago

OK, great, saw your message. If they ask followup questions about the identifier format - that part is not a problem for us anymore; Dataverse was confused by it, but I worked around it. It's just something they may want to consider - since the identifiers in question are registered and resolvable DOIs, why not advertise them as such?

The "from=" parameter on the other hand is a fairly important thing.

jggautier commented 6 months ago

@landreev, our contact, Chucky, wrote back that "regarding the date stamp issue, we have already instructed our IT personnel to update the date to reflect the latest data version release date in https://srda.sinica.edu.tw/oai_pmh/oai?verb=ListRecords&metadataPrefix=oai_dc," and that they'd make corrections to the identifier format (dc: identifier).

I'm not sure if it matters that they're referring to a ListRecords request and not the ListIdentifiers request that you said Dataverse uses. Should I ask them to clarify?

landreev commented 6 months ago

I scheduled it again for Sun. night, it's running now, we'll check on it tomorrow and see what it does. It's usually the same code underneath that drives the behavior of ListRecords and ListIdentifiers, when it comes to dates. I didn't think it was necessary to ask, unless we see something weird. I'm a little worried about how they phrased it - "update the date to reflect the latest data version". With "the date" possibly implying that it's the same date for all their records; which would again mean having to reharvest all their records every time. But let's see what/how many records we get tonight.

landreev commented 6 months ago

It did re-harvest all their records last night, i.e., still not incremental. I'm assuming we can just harvest their stuff like this once in a while manually for the time being, until they get the from= parameter to work on their end. One small improvement is that there were no broken records this time (there were 2 that failed to import the last time around).

jggautier commented 6 months ago

Nice! Okay. I got the sense that they hadn't updated the dates, yet, and Chucky will email again when they have. I can reach out in a month, on Mar 26, to ask if they've made the change, too. If they have by then we can set the schedule to weekly. If they haven't yet, we can just run another manual harvest then.

How's that sound?

cmbz commented 5 months ago

2024/03/12

jggautier commented 5 months ago

I see different dates in the <datestamp> of each record in https://srda.sinica.edu.tw/oai_pmh/oai?verb=ListRecords&metadataPrefix=oai_dc, but I'm not sure if that means that the datestamp issue has been resolved so that Harvard Dataverse can harvest incrementally and we can schedule weekly harvest.

I've emailed Chucky to ask if it's been resolved and if not, to email us when it has been so that I don't forget. See the email thread in RT

jggautier commented 5 months ago

@landreev, Chucky's replied and I'm concerned that I've cause some confusion, maybe because I don't understand enough about how incremental harvesting works.

I've CC'ed you on the email thread. Could you reply when you get the chance?

cmbz commented 2 months ago

@jggautier @landreev bumping this to find out status

jggautier commented 2 months ago

What a coincidence! I was just writing the comment at https://github.com/IQSS/dataverse-pm/issues/171#issuecomment-2189129040 about following up with all of the repository contacts who've emailed me, although in SRDA's case I see that there are some issues unique to them to follow up on.

jggautier commented 2 months ago

Looks like there are 2970 records in SRDA's OAI-PMH feed, and Harvard Dataverse has 2929 of them (https://dataverse.harvard.edu/dataverse/srda_harvested). I think we turned off scheduled harvesting - so Harvard Dataverse isn't trying to update harvested records each week - while indexing-related issues are being worked on, so it's been a while since Harvard Dataverse has tried to update the records it has from SRDA.

In late March our contact Chucky emailed:

In addition, I noticed that your colleague raised an issue regarding the ineffectiveness of the incremental harvesting parameter (from=) in the GitHub issue at https://github.com/IQSS/dataverse.harvard.edu/issues/243. I have already asked our IT team to look into this and resolve the issue.

I think the "from" parameter is working now for SRDA's OAI-PMH feed, at least from what I can tell when I try to use it, such as https://srda.sinica.edu.tw/oai_pmh/oai?verb=ListRecords&from=2024-02-25T00:00:00Z&metadataPrefix=oai_dc.

In addition to letting them know that we continue to work on indexing-related issues that are affecting harvesting, I can ask them to confirm that they've resolved this "from" parameter issue.

@cmbz and @landreev, what do you think?

cmbz commented 2 months ago

@jggautier sounds good to me.

jggautier commented 2 months ago

@landreev, Chucky from SRDA confirmed in the email that the "from" parameter is working now. 🎉

landreev commented 2 months ago

OK, I'll try to harvest from them and see if it is now incremental. But yes, scheduled regular harvesting is still off, pending fixing reindexing for harvested content, unfortunately. This is a blocker r/n for most issues that track harvesting content from specific remote archives.