cjlee112
commented
10 years ago It seems to me that DOI recognition is turning into a bit of a nightmare, because it consists of TWO incompatible standards:
- pre-2009: "Legal characters are the legal graphic characters of Unicode", extremely problematic because strictly speaking there no way to know where the DOI ends (any character allowed, so no character can be treated as terminator).
- post-2009: "CrossRef limits a DOI to only using the following characters: "a-z", "A-Z", "0-9" and "-._;()/" This rule was put into effect on January 1, 2009 and affects only the creation of new DOIs".
The problem is that when reading a string that starts doi:... there's no way to know whether it's going to be a pre or post-2009 DOI. This creates an impossible dilemma:
- if you terminate according to post-2009 rules, but the DOI is actually pre-2009, you'll terminate too early and get only a fragment of the correct DOI.
- if you allow any characters not in the post-2009 character set, but the DOI is actually post-2009, you'll append extra crap that isn't supposed to be part of the DOI.
Conundrums:
- There's no way to resolve this other than just first trying the post-2009 rule, and if that fails, try some heuristic for terminating the (in theory unterminatable) pre-2009 DOI format. And "try" means having to query external databases (e.g. crossref) to see if the putative DOI exists or not. Possible but seriously klugey.
- Is there any good heuristic for terminating pre-2009 DOIs? Given how DOIs are displayed on papers, journals, etc., whitespace seemed like a good terminator. But then G+ stuck an HTML tag immediately after a DOI (with no whitespace), breaking that rule. And unfortunately the HTML tag characters
<>are legal in pre-2009 DOIs... Is there any way to find out whether any publisher actually used<in a DOI? - there is one ironclad way to escape this whole set of dilemmas, and it's one that the original spnet design strongly urged on people: give the shortDOI instead of the DOI. This solves the whole problem. But will people do it? I think we should strongly urge people to do this, and warn them that if they don't, recognizing their DOI could fail for many possible reasons (link to page explaining all these technical complications in case they really want to know).
- Ideally we could tell users to paste pre-2009 DOIs as URLs, since all the above issues would then be taken care of by urlencoding (i.e. we use URL character rules to find the termination of the URL, the de-urlencode to get the DOI). But it's too much to expect all users to learn and follow this special rule.
- conceivably we could tell users they HAVE to give all DOIs as URLs... but I think they'd find this incomprehensible and infuriating. "I provided the right DOI, why won't spnet process it?!"
One more thought: we need a way to report to users that spnet hit a problem while trying to index their post. Right now people get frustrated because their post vanishes into a black hole (doesn't get indexed) and they have no way to know why. Unfortunately we have no direct way to communicate with them -- can't communicate with them on G+ (G+ gives no mechanism to do that; can't even send them email (we can look to see whether G+ gives us a user email address, I guess). Indeed no way even to know whether they'll log in to our site.
semorrison
The regex for DOI references can fail if the DOI is not followed by whitespace. This happens, for instance, if it is at the end of an html line (so it is immediately followed by
).
Currently, this doesn't break anything. The error must get corrected somewhere in the DOI lookup.