Text mining

[sckott]

We have a number of packages that provide an interface to either metadata or full text of academic articles. And there are other packages outside of ropensci to leverage.

However, we haven't done much work on these packages. They

• could be easier to use
• integrated with the tm package
• need use cases to demonstrate what can be done with them
• possible use cases or even integration with taxonomic data (from taxize) to extract taxonomic data from literature, then pass on to get phylogeny, occurrence data, etc.

[sckott]

Seems like there's no interest here, speak up if you are

[cpsievert]

I'm interested (especially in the full text of academic articles). I've been working on a tool to visualize and help interpret output from a topic model fit via Latent Dirichlet Allocation.

I think I could contribute to the "need use cases to demonstrate what can be done with them" point with something similar to what I did for xkcd. The end result could help us understand what topics are being discussed as well as trends over time. Since this is a hot topic (no pun intended :), it might be cool to make some related rMaps (if we can link geographical data to the articles).

[sckott]

@cpsievert Cool! Yeah, use cases would be great, and during that process surely you'll discover bugs/come up with things we can do better. Maybe a guest blog post on our blog too? What we have:

• elife (@IanMulvany) has an API for full text (and you can get XML full text files) - we have an alpha pkg here https://github.com/ropensci/elife that's not on CRAN
• PLOS with full text of all articles across their journals - rplos
• BioMed Central full text bmc is very early alpha - their API needs some work - I've been giving them feedback to improve interface/etc.
• Biodiversity Heritage Library rbhl - has full text OCR text from scanned books/etc. floras, species accounts, etc.
• We tried to work with Springer, but their web services are a mess
• PeerJ I imagine will provide web services soon, but at least you can get xml dumps of papers if you want - no R interface

[cpsievert]

Great! @karthik also mentioned he would like to work done on arXiv, but fitting a topic model to eLife articles might be more relevant for a blog post, right?

[sckott]

I don't know. Thoughts @karthik

[karthik]

I think I could contribute to the "need use cases to demonstrate what can be done with them" point with something similar to what I did for xkcd. The end result could help us understand what topics are being discussed as well as trends over time. Since this is a hot topic (no pun intended :), it might be cool to make some related rMaps (if we can link geographical data to the articles).

Sounds great. Working on arxiv would be great time permitting. But if you want to work on the elife thing, that's great too.

[benmarwick]

@sckott @cpsievert I have a package for working with JSTOR's Data for Research service here: https://github.com/benmarwick/JSTORr

[Ironholds]

Could be fun.

[szeitlin]

Sounds useful. And kind of similar to somethings I've used a lot, e.g. iHOP? http://www.ihop-net.org/UniPub/iHOP/