Idea: direct sync with Zotero/Mendeley sqlite DB

[mangecoeur]

Both Zotero and Mendeley store their references in sqlite database files (Mendeley already offers to sync directly with Zotero's file).

It would be interesting to offer direct syncing to this file for getting citekeys and paper data instead of doing a round-trip though bibtex export functionalities of either program. Reading sqlite is straightforward and both programs store this DB's in predictable places, and have fairly straightforward table layouts (e.g. reading Mendeley's "Documents" table gets you all you need).

[tpoisot]

I don't think I like the idea. It would require to write code for Mendeley and Zotero, so a lot of code will be for things many users will not have. Zotero and Mendeley can change things in their DB layouts, whereas bibtex and citeproc are stable.

[mangecoeur]

I don't suggest removing bibtex support, just adding sqlite as an option. Sqlite support gives much more reliable access to the original data (bibtex file format is not very well defined). I doubt that either Zotero or Mendeley are going to change their file formats very often.

Another idea would be to used their web APIs (handy if you find yourself trying to sync bibtex files between machines) - but this means setting up the login authentication, which is a pain...

[tpoisot]

I get your point, but it would probably double the amount of code in the package. Besides, Mendeley and Zotero have export functions (and it can be automated in both cases). I vote against.

[lodestone]

I think it would complicate this codebase greatly. It's a good idea, but seems like a better fit for a new package.

[mangecoeur]

I'm not convinced it would be very complicated - reading SQLITE reliably is much much simpler than a reliable bibtex parser. I think a separate package would mean a lot more code, rather than less. I mean the code to actually get the data would look something like:

var db = new sqlite3.Database(pathToSqliteFromConfig);
db.all("SELECT * FROM Documents", function(rows) {
   bib =[]
   for (row in rows){
     citation = {entryTags:{}};
     citation.entryTags.title = row.title;
     citation.entryTags.author = row.author;
     citation.entryTags.editor = row.editor;
     bib.push(citation)
}
  });
db.close();

//pass the bib list to the rest of the package as if it was read from a bibtext file

It's not exactly mind-bending stuff ;)

[lodestone]

It's not exactly mind-bending stuff ;)

@mangecoeur take a look at the schema of Zotero. I just did and my mind bent. LOL

[tpoisot]

Ok, so maybe when the panel thing is pulled, submit a PR for this one too. If it's not too much new code, then we can always include it. 

Sent from a mobile device - pardon the brevity/typos.

[apcshields]

I think, if we're going to continue to increase the number of possible data sources, we should consider making this project manage the presentation of data which is supplied through source-specific plugins.

If we design the interface for the plugins well, it shouldn't significantly increase the amount of code required (beyond a standard wrapper to turn each plugin into an Atom package).

Andrew

— Sent from Mailbox

On Fri, Oct 30, 2015 at 3:44 PM, Timothée Poisot notifications@github.com wrote:

Ok, so maybe when the panel thing is pulled, submit a PR for this one too. If it's not too much new code, then we can always include it. 

Sent from a mobile device - pardon the brevity/typos.

Reply to this email directly or view it on GitHub: https://github.com/apcshields/autocomplete-bibtex/issues/44#issuecomment-152633752

[NoxMortem]

Zotero supports a feature where it keeps the bibtex file updated on changes. However, autocomplete-bibtex is not able to process the file changes yet (it does not reflect the new file)

[rriemann]

@NoxMortem , are you sure? The code reads as if the package internal database is updated on bibtex file change:

https://github.com/apcshields/autocomplete-bibtex/blob/master/lib/provider.js#L84

The Zotero Better BibLatex plugin is about to integrate code that allows to retrieve suggestions via a JSON-RPC API: https://github.com/retorquere/zotero-better-bibtex/issues/852

I have tested it. Please find the example below. I suggest to add a new option (tick box) to the settings to activate as additional source this API to fetch suggestions for a given search query.

POST request (with curl) for the search term Helios:

curl http://localhost:23119/better-bibtex/json-rpc -X POST -H "Content-Type: application/json" -H "Accept: application/json" --data-binary '{"jsonrpc": "2.0", "method": "item.search", "params": ["Helios"] }'

Beautified Response:

{
  "jsonrpc": "2.0",
  "result": [
    {
      "id": "http://zotero.org/users/2043877/items/IWN9RADQ",
      "type": "article-journal",
      "title": "Electing a university president using open-audit voting: Analysis of real-world use of Helios",
      "container-title": "EVT/WOTE",
      "volume": "9",
      "source": "Google Scholar",
      "URL": "https://www.usenix.org/event/evtwote09/tech/full_papers/adida-helios.pdf",
      "shortTitle": "Electing a university president using open-audit voting",
      "author": [
        {
          "family": "Adida",
          "given": "Ben"
        },
        {
          "family": "De Marneffe",
          "given": "Olivier"
        },
        {
          "family": "Pereira",
          "given": "Olivier"
        },
        {
          "family": "Quisquater",
          "given": "Jean-Jacques"
        },
        {
          "literal": "others"
        }
      ],
      "issued": {
        "date-parts": [
          [
            "2009"
          ]
        ]
      },
      "accessed": {
        "date-parts": [
          [
            "2014",
            9,
            2
          ]
        ]
      },
      "library": "My Library",
      "citekey": "AdidaElectinguniversitypresident2009"
    },
    {
      "id": "http://zotero.org/users/2043877/items/XC6XBRK9",
      "type": "article-journal",
      "title": "Helios: Web-based Open-Audit Voting.",
      "container-title": "USENIX Security Symposium",
      "page": "335–348",
      "volume": "17",
      "source": "Google Scholar",
      "abstract": "Voting with cryptographic auditing, sometimes called\nopen-audit voting, has remained, for the most part, a theoretical endeavor. In spite of dozens of fascinating pro-\ntocols and recent ground-breaking advances in the field,\nthere exist only a handful of specialized implementations\nthat few people have experienced directly. As a result, the benefits of cryptographically audited elections have remained elusive.\nWe present Helios, the first web-based, open-audit\nvoting system. Helios is publicly accessible today: any-\none can create and run an election, and any willing observer can audit the entire process. Helios is ideal for online software communities, local clubs, student government, and other environments where trustworthy, secret-ballot elections are required but coercion is not a serious concern. With Helios, we hope to expose many to the power of open-audit elections.",
      "URL": "https://www.usenix.org/legacy/confadmin/sec08/papers/S97/content.pdf",
      "shortTitle": "Helios",
      "author": [
        {
          "family": "Adida",
          "given": "Ben"
        }
      ],
      "issued": {
        "date-parts": [
          [
            "2008"
          ]
        ]
      },
      "accessed": {
        "date-parts": [
          [
            "2014",
            9,
            2
          ]
        ]
      },
      "library": "My Library",
      "citekey": "AdidaHeliosWebbasedOpenAudit2008"
    }
  ],
  "id": null
}