Add RIS (refman) import

[dsifford]

See #24

cc: @someonehere15

[dsifford]

Note to self: This seems to work as a halfway decent bibtex-to-json parser.. Run with it.

(assuming bib = a string a bibtex)

let x = bib.split(/(^\s*$)/gm).map(i => i.trim()).filter(i => i != "");

let y = [];

x.forEach(i => {
    let z = {};
    let a = i.split(/\n/m);
    a.forEach((j, ii) => {
        if (ii == 0 || ii == a.length - 1) { return; }
        let key = j.split('=', 1)[0].trim();
        let val = j.substr(j.split('=', 1)[0].length + 1).trim();
        val = val.match(/{.+}/) 
            ? val.match(/{.+}/)[0].replace(/[{}]/g, '')
            : val.substr(0, val.length - 1).replace(/[{}]/g, '');
        z[key] = val;
    });
    y.push(z);
})

[someonehere15]

Here's an example:

@inproceedings{Liu1999, abstract = {For any re-engineering of the legacy system such as alteration, modification or complete rebuild, to understand the original requirements can be important and sometimes critical. The semiotic approach we present aims at recovering requirements by studying the legacy system's behaviour. Using this approach, the requirements recovery consists of investigation activities at three major stages with a set of techniques for analysis and representation. The approach guides the analysis to be conducted from a number of angles at certain levels of detail using various software engineering and semiotic techniques. These techniques enable one, for derivation of the requirements, to study a legacy system from perspectives of different stakeholders, from its interaction with the users in the business context, and from information contents and processes of system operations. A case study applying this approach to recover requirements for a university's multi-site library is presented}, author = {Liu, Kecheng and Alderson, Albert and Qureshi, Zubair}, booktitle = {Proceedings IEEE International Conference on Software Maintenance (ICSM' 99)}, doi = {10.1109/ICSM.1999.792485}, isbn = {0-7695-0016-1}, issn = {1063-6773}, keywords = {Cost function,Electrical capacitance tomography,Environmental factors,Read only memory,Reverse engineering,System analysis and design,legacy systems,library automation,organisational semiotics,requirements engineering,requirements recovery,reverse engineering,software engineering,system behaviour modelling,systems re-engineering,systems reengineering,university library}, pages = {3--12}, publisher = {IEEE}, title = {{Requirements Recovery from Legacy Systems by Analysing and Modelling Behaviour}}, url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=792485}, year = {1999} }

and one with special characters (See the authors):

@inproceedings{Waljas2010, abstract = {Many web-based services utilize both desktop and mobile terminals in delivering content and functionality to their users. In terms of user experience (UX), the overall chain of interactions, including mobile and non-mobile settings, becomes a central design target. The aim of this study was to investigate, what are the key elements of user experience associated with these, cross-platform interactions. This paper presents the findings from a four week long field study with three web-based cross-platform services. During the study, participants used the services on both their PCs and mobile devices. Diaries and interviews were used for gathering users' experiences with the services. Based on our findings and reflection with related work, we argue that central elements of cross-platform service UX include fit for cross-contextual activities, flow of interactions and content, and perceived service coherence. We propose an initial conceptual framework of cross-platform user experience. The framework can be used to guide the design of cross-platform web services, as it draws attention to elements of user experience that are essentially influenced by the characteristics of cross-platform settings.}, address = {New York}, author = {W{\"{a}}ljas, Minna and Segerst{\aa}hl, Katarina and V{\"{a}}{\"{a}}n{\"{a}}nen-Vainio-Mattila, Kaisa and Oinas-Kukkonen, Harri}, booktitle = {Proceedings of the 12th international conference on Human computer interaction with mobile devices and services - MobileHCI '10}, doi = {10.1145/1851600.1851637}, isbn = {978-1-60558-835-3}, keywords = {cross-platform web services,crossmedial,user experience,ux}, pages = {219--228}, publisher = {ACM}, title = {{Cross-platform service user experience: a field study and an initial framework}}, url = {http://doi.acm.org/10.1145/1851600.1851637}, year = {2010} }

Every academic database can export to bibtex, because this , but you can write me if you need anything. Keep in mind to also test the special characters, they might become a problem.

[someonehere15]

BTW, on your plugin theres Journal, Book, and Web Page. I dont know how you would classify other types, like the example bibtex I sent. But it would be nice to add some more options, like Conference Proceedings. Or at least keep in mind to assign types without a template to similar known types, e.g. if a bibtex contains a Working Paper, it should be formatted as a Journal Article.

[dsifford]

Is there any way you can send me a list of all known bibtex types?

[dsifford]

I'll have to see if I can work on special characters, but the above snippet I posted was able to convert the bibtex you posted to this:

[[object Object] {
  abstract: "For any re-engineering of the legacy system such as alteration, modification or complete rebuild, to understand the original requirements can be important and sometimes critical. The semiotic approach we present aims at recovering requirements by studying the legacy system's behaviour. Using this approach, the requirements recovery consists of investigation activities at three major stages with a set of techniques for analysis and representation. The approach guides the analysis to be conducted from a number of angles at certain levels of detail using various software engineering and semiotic techniques. These techniques enable one, for derivation of the requirements, to study a legacy system from perspectives of different stakeholders, from its interaction with the users in the business context, and from information contents and processes of system operations. A case study applying this approach to recover requirements for a university's multi-site library is presented",
  author: "Liu, Kecheng and Alderson, Albert and Qureshi, Zubair",
  booktitle: "Proceedings IEEE International Conference on Software Maintenance (ICSM' 99)",
  doi: "10.1109/ICSM.1999.792485",
  isbn: "0-7695-0016-1",
  issn: "1063-6773",
  keywords: "Cost function,Electrical capacitance tomography,Environmental factors,Read only memory,Reverse engineering,System analysis and design,legacy systems,library automation,organisational semiotics,requirements engineering,requirements recovery,reverse engineering,software engineering,system behaviour modelling,systems re-engineering,systems reengineering,university library",
  pages: "3--12",
  publisher: "IEEE",
  title: "Requirements Recovery from Legacy Systems by Analysing and Modelling Behaviour",
  url: "http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=792485",
  year: "1999"
}, [object Object] { ... }, [object Object] {
  abstract: "Many web-based services utilize both desktop and mobile terminals in delivering content and functionality to their users. In terms of user experience (UX), the overall chain of interactions, including mobile and non-mobile settings, becomes a central design target. The aim of this study was to investigate, what are the key elements of user experience associated with these, cross-platform interactions. This paper presents the findings from a four week long field study with three web-based cross-platform services. During the study, participants used the services on both their PCs and mobile devices. Diaries and interviews were used for gathering users' experiences with the services. Based on our findings and reflection with related work, we argue that central elements of cross-platform service UX include fit for cross-contextual activities, flow of interactions and content, and perceived service coherence. We propose an initial conceptual framework of cross-platform user experience. The framework can be used to guide the design of cross-platform web services, as it draws attention to elements of user experience that are essentially influenced by the characteristics of cross-platform settings.",
  address: "New York",
  author: "W\"aljas, Minna and Segerstaahl, Katarina and V\"a\"an\"anen-Vainio-Mattila, Kaisa and Oinas-Kukkonen, Harri",
  booktitle: "Proceedings of the 12th international conference on Human computer interaction with mobile devices and services - MobileHCI '10",
  doi: "10.1145/1851600.1851637",
  isbn: "978-1-60558-835-3",
  keywords: "cross-platform web services,crossmedial,user experience,ux",
  pages: "219--228",
  publisher: "ACM",
  title: "Cross-platform service user experience: a field study and an initial framework",
  url: "http://doi.acm.org/10.1145/1851600.1851637",
  year: "2010"
}]

[dsifford]

Also, nevermind on posting the list of types. I found this:

https://www.andy-roberts.net/res/writing/latex/bibentries.pdf

If that doesn't include all of the types, let me know

[someonehere15]

The conversion looks great! (except the special chars)

Looks like the types are all there on your pdf, but the fields maybe not, there's no URL, DOI fields, although I couldnt find any bigger list than yours. Maybe they are custom fields, but very used. Nice job

[dsifford]

@someonehere15 Thanks. I'll keep you updated on the status.

Also, looking at the test references you sent, it appears that you have background in computer science? Is that correct? Forgive me if my previous comments were insulting your intelligence.

[someonehere15]

No man, no problem at all. You've done a really good job here, seriously. It's strange that there are some reference tools for WP, but none of them does a full job. Something is always missing.

[dsifford]

@someonehere15 Gonna be starting on the bibtex parser soon. In the meantime, let me know what you think of this:

https://vid.me/JYCo

Automatically adjusts both inline references and the citation list if you click and drag citations around the side list. Still very "alpha" but, it seems to be passing the tests that I've thrown at it. Oh, and it also gets rid of shortcodes completely. 👍

cc: @metallikat36

[dsifford]

Stress test: https://vid.me/DKIi

[someonehere15]

Nice, I love it! Very practical. Its a very good addition. I think this would help a lot. Watched the stress test while writing here, very responsive. Well done. Can you still also add a title for the list though, that can be set manually. I though it is better than people writing "Bibliography" manually on every post.

Let me also throw another thought out there, just an idea from an actual case: At the moment, on a site, we use a custom footnotes plugin to do the references. Its of course just text, no reference capabilities. The footnotes however are also used for images. We have a custom field for the images, which saves the source, and when the image is inserted in a post, the source is added on the image title with a number and the source link at the end list of footnotes. So what I'm saying with this is the idea of somehow also including image sources, maybe in the same, or a separate list. Not a must, like I said, just a thought, maybe if you want to expand in the future. There are some plugins, e.g. Image Source Control, but are separate, and don't add the source in an end list. This may actually be better depending on the case, to have images separately through ISC, and not mix them with the references, but again, just a thought.

[dsifford]

@someonehere15

Can you still also add a title for the list though, that can be set manually. I though it is better than people writing "Bibliography" manually on every post.

That can be done with css.

Add this to your stylesheet and adjust however you like best:

ol#abt-smart-bib:before {
  content: 'Bibliography';  
  margin-left: -20px;
  font-weight: 900;
  line-height: 50px;    
}

Demo here

---

As for your other request, I'm not quite sure I'm understanding what it is you're describing. Any way you can send me a screenshot or something?

[someonehere15]

I actually tried with css but it was not working. Maybe I was using the wrong class, I'll check again.

I'll send you something later, I'm writing from my phone now.

[metallikat36]

Awesome Derek! I'll be happy to test it out thoroughly once it's done.

[dsifford]

@someonehere15 Can you email me a sample bibtex file? Ideally, one that has a good mixture of reference types?

I'm gonna get started on the bibtex upload feature this week.

dereksifford@gmail.com

Thanks.

[dsifford]

Also, just realized that most reference managers also support RIS and EndNote (xml) export formats... Either of these will be light years easier to implement than bibtex. Would that be an acceptable option? Or is BibTeX mandatory for your workflow?

If either of these are acceptable, I'm going to go with RIS.

[someonehere15]

Hey, sorry for the delay. The reason for the bibtex is because it is also supported by latex, and besides, its the oldest and most used type. It would not be a problem personally for me, but maybe for others. If the reference types are your worry, you don't need to implement all of them anyway, there are more defined than necessary, and besides, the types would be the same with RIS, that's not really format specific. I would not recommend EndNote though, RIS is ok. Maybe you can even provide both RIS and BibTex later ;), but as I said, for me RIS is also ok. Let me know if you still need a bibtex file or you'll proceed with RIS. Don't forget to provide a download button for each reference.

As for what I wrote before, I meant two things:

1. Footnotes: Beside actual references, authors may want to add remarks as footnotes in their posts. There are plugins for footnotes, but why not one for all? I think it would be easy to implement, you can just add it as a type of the manual formatted reference function, but with only a text field, or a text and link, like a web page, and maybe add them in a separate list, not together with the references. (e.g. Simple Footnotes is a plugin).
2. Image Sources: Images also need to be referenced if you got it from another website or someone, right? There is no native way to do that. In my case, we have done it like this: We add a custom field to images, where you put the source link to each uploaded image. When the image is placed in a post, a number is assigned and added to the image caption, and the source link is shown as a footnote together with the other footnotes at the end of the post, as you can see from the pictures below. As I said this is automatic, only the source is entered in the image custom Source field.

I hope it is clear now. This way, your plugin would substitute two plugins and be a full informal and formal reference plugin. The image thing was just an idea and not really necessary since there are other plugins, but maybe the simple footnotes would be more useful.

[dsifford]

@someonehere15 Thanks for the clarification.

The reason I'm leaning toward RIS is that it is just much easier to reason about programatically than bibtex.

Here's an example of the format

TY  - JOUR
AU  - Shannon,Claude E.
PY  - 1948/07//
TI  - A Mathematical Theory of Communication
T2  - Bell System Technical Journal
SP  - 379
EP  - 423
VL  - 27
ER  - 

Much easier; I can just split each line at the - and then trim the whitespace and know for certain that I have all of the information that I need. The bibtex script I wrote works fairly well too, but I'm sure there will be edge cases that sneak past it.

Re: footnotes and image sources -- I'm not opposed to the ideas, but it won't happen on this next update. If you'd like, open a separate issue for both of those things and I'll work on it in the future. 👍

[dsifford]

@someonehere15 I checked into it and it looks like zotero, mendeley, endnote, and papers allows you to export in RIS format (some call it Refman format). Because it'll be easier from a maintainability standpoint, I'm going to go that route..

There are RIS to bibtex converters online that you can use to convert if you need to paste it into a latex document. http://www.bruot.org/ris2bib/ is just one example of many.

[someonehere15]

No problem, if it's easier for you then go with it. I know the format and yes, it is supported by most, so don't worry, you can use it.

[dsifford]

@someonehere15 Which types did you say you would like added (to go along with website, journal, and book)?

[dsifford]

@someonehere15 Early prototype: https://vid.me/jhpr

It's still not perfect (as you'll see), but it's coming along nicely.

Let me know your thoughts.

[someonehere15]

Wonderful! It's coming along nicely, right?

For me probably the types conference proceedings, report, maybe also book chapter and thesis. For the types that you don't include separately, you can just add a generic type, and parse the fields that you can recognize, e.g. author, title, journal, url, etc. (eventually editors, publisher, etc.), which you also use on other types, and reference it in a general way. I think RIS and APA already have a generic reference type. Also, it would be nice if you include the DOI number to all the reference types, if the imported reference has it of course. You can show the DOI number, which is hyperlinked to the doi resolver (check here, you use the standard URL and just add the DOI number). Beside that, as I said, maybe also add a "footnote" type, with just a text field, maybe also a link, which is not formatted in anyway, just manually added through your plugin in the editor, and also not downloadable like the other references.

[someonehere15]

Can I give you another idea, and please keep in mind these are just ideas, so don't feel any pressure or anything, we are just discussing here after all. I think one nice additional feature would be to add an attachment link to references, e.g. by using the WordPress Media Library, or just entering a url to a file, in case e.g. someone wants to include a paper as a PDF with the reference. You can e.g. check out the papercite plugin that does it, just as a reference.

[dsifford]

conference proceedings, report, maybe also book chapter and thesis

Book chapter already exists - I fused it together with "book"

I think RIS and APA already have a generic reference type

You are correct. I'll think about how to best handle that.

it would be nice if you include the DOI number to all the reference types, if the imported reference has it of course. You can show the DOI number, which is hyperlinked to the doi resolver

Not a bad idea. I'll add in.

downloadable like the other references

Can you give me an example of how you vision this to work?

attachment link to references

Open a separate issue for this

[dsifford]

@someonehere15 @metallikat36 There's going to be an additional delay on 2.5.0 because I'm in the process of converting the current way that I parse citations into CSL format (as defined by the Citation Style Language spec) -- This will allow me to add the option for users to define their own citation style which I think will be better for the long-term.

[someonehere15]

Very nice thinking on CSL, it is pretty much standard nowadays. For the footnotes: Beside the actual reference types, book, journal, etc., you can add a "Footnote" type, where I can just add text, and a link field, for a link. More specifically, you can format it like this: If there is only text and no link, just show the text of course, if there is a link too, you can show it "text: link" for instance. Notice the colon (:) that you automatically add. The other references are going to be downloadable by the user, in a RIS format. What I meant was that the "footnote" type will not have a download option, because it is not a formal reference.

[dsifford]

@someonehere15 Re: footnotes, that'll have to come in the next major release. It'll take a bit of work to pull off (correctly)

Also, I checked out the KCite plugin for giggles and... wow. The amount of errors that it throws in debug mode is scary. If you have it installed, strongly consider removing it.

[someonehere15]

hehe, no, haven't used it much, but the idea is nice. you can check out papercite though, that is a good reference.

[dsifford]

@someonehere15 @metallikat36 -- Happy to report, I managed to get full CSL support working (without having to upload the 200Mb of bloat files)

What does that mean? Next update will have 1300 citation styles, and 40 reference types supported.

Here's a demo of RIS import (using a range of different reference types)

https://vid.me/WE9U

Let me know what you think 👍

Correction: There's 8147 citation styles to choose from 👍 https://vid.me/vUDE

[dsifford]

Also -- I'm adding DOI support through Crossref as well. 👍 Thanks for the suggestions @someonehere15 @metallikat36

[someonehere15]

Nice touch on the unending list video ;). You did a wonderful job on everything, so I'm speechless now. Believe me I tried every plugin possible, and no one is as polished and this functional, so congrats. Can't wait to try it.

[dsifford]

@someonehere15 Thanks for the kind words. Glad you like it!

The plugin update is pretty much complete from a literal standpoint. All that's left for me to do is work on better test coverage, some last minute file cleanup, and work on the documentation.

I'll probably have the update pushed out by the end of this weekend at the latest 👍 (assuming I don't find any difficult edge cases while running tests). I'm making this a "major" upgrade to 3.0.0, rather than a minor due to how extensive the rewriting needed to be.

Thanks again for your suggestions (and @metallikat36)

[someonehere15]

Sure, it's a pleasure with someone who listens, so thank you.

[someonehere15]

Hey just a question: will all the references be downloadable, or only the ones that have been imported from RIS? For instance if I have entered a reference manually, will that also be available to download as a RIS? It would be nice if all of them are downloadable, not partially. Will you also provide a link to download a RIS file with all the references together?

One last thing I thought of: Beside the local upload of a RIS file to be imported, you could add an option to just provide the url to a ris file, and the plugin imports it directly from there.

[dsifford]

@someonehere15

I won't be adding reference downloading to this next release because there's already too many changes. I'd like the next release out into the wild first so that I can be sure it's working properly before spreading myself out too thin.

That said, All references (no matter where they come from) in this plugin are converted to CSL. I'd be able to easily export the references to CSL in a future release, but things become a bit more challenging if you'd need CSL => RIS export. This is due to mendeley, scopus, and some others not following the rules and creating their own field definitions. Basically, what you put in won't necessarily be what you get out because I have to shim each of their oddball fields to a standard field (and by doing so, information will be lost in the process)..

I believe you should be able to import CSL directly to mendeley (and likely zotero), Is that correct?

Re: URL import, open an issue for it and I'll tackle on a future release (same with anything else you might be able to think of)

[someonehere15]

Not to sound bad here, but to be honest downloadable references are the single most important feature I needed. Of course it's clear that you have done a wonderful job with the implementation and the listening to others like us, and you improved the plugin a lot. But as I said, exporting is the only thing still keeping me from this. I was really looking forward to everything, but it won't be useful for my prototype. Still, great job. If I get any ideas I will surely let you know.

[dsifford]

@someonehere15 Open an issue for that and I'll look into it. Just because it won't be on the next release doesn't mean that it will be a long way away.

Also, can you clarify the question that I asked above Re: exporting CSL?

[dsifford]

@someonehere15 Also, I apologize if it has appeared as though I wasn't putting much work into this project over the last few days. Truth be told, it's been quite the contrary.

Please sit tight. As soon as I get this next release tested and pushed out, I'll then be able to focus on new features. Settling for some of the other options you've mentioned may seem appealing now, but I can assure you that it'll be the equivalent of throwing a hand grenade at your database.

So, again. Please, please be patient and open a new issue for the features you are still in need of. 👍 Thank you.

[someonehere15]

I hope you didn't take my comment the wrong way, I didn't mean at all that you weren't putting a lot of effort to it. On the contrary I also think that you actually listened and tried to solve a lot more than anyone I know in such a short time. Don't get me wrong there. I just meant that in the specific case I couldn't use it, not because I don't want to, but because that specific feature was the main requirement (the downloading I mean, the rest was just casual improvements to consider). It is of course not the end of the world, but since it's due in a month I thought your plugin would help. But again, no problem and no pressure, please let me tell you that I am impressed from the whole interaction and your efforts.

As for the CSL, maybe I'm wrong but from what I know, that is a standard for the formatting and structure of the reference, not a reference holder itself containing specific publication data. Mendeley for instance also supports it as a formatting style, but I am not aware of a feature to use it as a container for actual references, like ris or bibtex. So I don't really know what you mean. Again, maybe I'm wrong or maybe I didn't correctly understand you.

I will open an issue for the downloadable files and url import as you asked,and again, thanks for listening either way.

[dsifford]

@someonehere15 No problem. Just didn't want you thinking that I wasn't considering your suggestions.

Re: CSL -- I was referring to the JSON (or XML, I suppose) schema. Wasn't sure if the bigger reference managers accepted that as an import. If not, I suppose I could just export back to endnote RIS format and hope that the other reference managers prepared themselves for this sort of thing (I would assume they would have).

Thanks for opening the issues. I'm getting ready to push version 3.0.0 live here in a few (pending me finishing the readme file...ugh). Once that happens, I'll start looking into those things. 👍

Closing this issue. We can keep the conversation moving in the new issues you opened.