LibraryID Error for Globally Unique Keys

[jpshanno]

Thanks for the package, this seems like a great tool. I was trying to get it working and I ran into a problem that seemed dependent on my BBT settings in Zotero. When I have "Keep keys unique" set to "across all libraries" the bbt_write_bib() fails with the error "Error: keyscope is global, do not provide a library ID". If I change that option to "within each library" then bbt_write_bib() runs fine.

I think I've traced the problem back to bbt_call_json_rpc(). It looks like for globally unique keys, a libraryID parameter has to be passed with the item.export request as a third parameter. With some trials it looks like an empty libraryID ("//") or "My Library" will work. The docs for BBT say that 'My Library" is assumed, but unless I manually specify it I get the above error. I haven't forked rbbt to try writing an actual solution, but after some quick tests I'm pretty confident that's the general solution. In the coming weeks I may have time to take a closer look. Here is the test code that worked for me (an edited version of bbt_call_json_rpc) with a single citation:

httr::content(httr::POST(rbbt:::bbt_url("json-rpc"), body = list(jsonrpc = "2.0", 
    method = "item.export", params = list(list("moore-2016"), "csljson", "My Library")), encode = "json"), 
    as = "parsed", encoding = "UTF-8")

Relevant Session Info: R 3.6.0 rbbt 0.0.0.9000 httr 1.4.1 Zotero 5.0.87

[paleolimbot]

Sorry this took me a second...I can't get either "//" or "My Library" to work:

httr::content(
  httr::POST(
    rbbt:::bbt_url("json-rpc"), 
    body = list(
      jsonrpc = "2.0",
      method = "item.export", 
      params = list(list("dunnington_etal18"), "csljson", "My Library")), 
    encode = "json"
  ), 
  as = "parsed", 
  encoding = "UTF-8"
)
#> $jsonrpc
#> [1] "2.0"
#> 
#> $error
#> $error$code
#> [1] -32602
#> 
#> $error$message
#> [1] "keyscope is per-library, you should provide a library ID"
#> 
#> 
#> $id
#> NULL

^{Created on 2020-05-30 by the reprex package (v0.3.0)}

Any ideas?

[jpshanno]

It turns out BBT does not use the library name in the request, it identifies each library with a number. I wrote a function to get the libraryID given the library name (which would let users specify a library for rbbt to query). Passing the libraryID as a number works when the keys are unique within a library, but gives the same error I originally posted when the keys are globally unique (though my original solution of "//" still works). Both the "keyscope is per-library, you should provide a library ID" and "keyscope is global, do not provide a library ID" have the same error code. So if the results return the error code -32602 then request could be resent after dropping libraryID. It looks like the BBT default is to have library-unique keys, so I passed the three parameter option first, then on error pass the two parameter option. If speed is ever became a problem you could also pass just the first key and check the error message. Let me know if you would prefer some of this as a PR.

# Get Library ID from Supplied Name ---------------------------------------

bbt_get_library_id <- 
  function(libraryID){

    if(is.numeric(libraryID)){
      libraryID
    }

    libraries <- 
      httr::content(
        httr::POST(
          rbbt:::bbt_url("json-rpc"),
          body = list(
            jsonrpc = "2.0",
            method = "user.groups"),
          encode = "json"
        ),
        as = "parsed",
        encoding = "UTF-8"
      )

    if (!is.null(libraries$error)) {
      stop(libraries$error$message, call. = FALSE) 
      }

    matching <- 
      Filter(function(x){x$name == libraryID}, 
             libraries$result)

    n_matches <- 
      length(matching)

    if(n_matches != 1){
      if(n_matches == 0){
        stop("The library '", libraryID, "' was not found in Zotero")
      } else {
        # I'm not sure how to deal with multiple group libraries with the same
        # name, so I am treating it as an error here. One option would be to try
        # both libraries to find the citation keys.
        stop("Multiple libraries with the ID '", libraryID, "' were found in Zotero")
      }

    }

    matching[[1]]$id
  }

# Editted bbt_call_json_rpc -----------------------------------------------

keys <- 
  list("bonan-1995", "barthold-2011")
translator <- 
  "csljson"
libraryID <- 
  bbt_get_library_id("My Library")

result <- 
  httr::content(
    httr::POST(
      rbbt:::bbt_url("json-rpc"), 
      body = list(
        jsonrpc = "2.0",
        method = "item.export", 
        params = list(keys, translator, libraryID)), 
      encode = "json"
    ), 
    as = "parsed", 
    encoding = "UTF-8"
  )

if (!is.null(result$error)) {

  if(result$error$code != -32602){
    stop(result$error$message, call. = FALSE) 
  }

  result <- 
    httr::content(
      httr::POST(
        rbbt:::bbt_url("json-rpc"), 
        body = list(
          jsonrpc = "2.0",
          method = "item.export", 
          params = list(keys, translator, "//")), 
          encode = "json"
        ), 
        as = "parsed", 
        encoding = "UTF-8"
      )
}

result
#> $jsonrpc
#> [1] "2.0"
#> 
#> $result
#> $result[[1]]
#> [1] 200
#> 
#> $result[[2]]
#> [1] "text/plain"
#> 
#> $result[[3]]
#> [1] "[\n  {\"id\":\"barthold-2011\",\"accessed\":{\"date-parts\":[[2020,1,15]]},\"author\":[{\"family\":\"Barthold\",\"given\":\"Frauke K.\"},{\"family\":\"Tyralla\",\"given\":\"Christoph\"},{\"family\":\"Schneider\",\"given\":\"Katrin\"},{\"family\":\"Vaché\",\"given\":\"Kellie B.\"},{\"family\":\"Frede\",\"given\":\"Hans-Georg\"},{\"family\":\"Breuer\",\"given\":\"Lutz\"}],\"container-title\":\"Water Resources Research\",\"container-title-short\":\"Water Resour. Res.\",\"DOI\":\"10.1029/2011WR010604\",\"ISSN\":\"00431397\",\"issue\":\"8\",\"issued\":{\"date-parts\":[[2011,8]]},\"language\":\"en\",\"source\":\"DOI.org (Crossref)\",\"title\":\"How many tracers do we need for end member mixing analysis (EMMA)? A sensitivity analysis: HOW MANY TRACERS FOR EMMA? SENSITIVTY ANALYSIS\",\"title-short\":\"How many tracers do we need for end member mixing analysis (EMMA)?\",\"type\":\"article-journal\",\"URL\":\"http://doi.wiley.com/10.1029/2011WR010604\",\"volume\":\"47\"},\n  {\"id\":\"bonan-1995\",\"abstract\":\"Abstract A land surface model that includes a subgrid parameterization for inland water (lake, swamp, marsh) was coupled to a modified version of the NCAR CCM2. The coupled model was run for 5 yr with and without inland water subgrid points to determine the importance of inland water for global climate simulation. In July, the inclusion of these water bodies resulted in a spatially consistent signal in which high inland water regions were 2°–3°C cooler, had increased latent heat flux (10–45 W m-2), and decreased sensible heat flux (5–30 W m-2) compared to the simulation without these water bodies. These changes were statistically significant in the lake region of northwest Canada, the Great Lakes region of North America, the swamp and marsh region of the Siberian lowlands, and the lake region of East Africa, but were not significantly different in the swamp and marsh region of Finland and northwest Russia. The effect on Northern Hemisphere January air temperature was difficult to interpret due to large interannual variability. In tropical lake regions (East Africa), the response to lakes was less in the rainy season (January) than in the dry season (July). Precipitation was unchanged in both months except for the Great Lakes region where precipitation increased in January. These changes in temperature, precipitation, and surface fluxes were consistent with mesoscale modeling studies of the effects of lakes on climate and tended to bring the model closer to observations. In particular, the summer cooling in North America helped reduce a large warm temperature bias in the model, but did not eliminate the bias. The lakes had little effect on atmospheric moisture, radiation, or zonal circulation. These results show that subgrid-scale inland water bodies can be successfully added to global land surface models for use with GCMS.\",\"author\":[{\"family\":\"Bonan\",\"given\":\"Gordon B.\"}],\"container-title\":\"Journal of Climate\",\"DOI\":\"10.1175/1520-0442(1995)008<2691:SOAGST>2.0.CO;2\",\"issue\":\"11\",\"issued\":{\"date-parts\":[[1995]]},\"page\":\"2691-2704\",\"title\":\"Sensitivity of a GCM simulation to inclusion of inland water surfaces\",\"type\":\"article-journal\",\"URL\":\"https://doi.org/10.1175/1520-0442(1995)008<2691:SOAGST>2.0.CO;2\",\"volume\":\"8\"}\n]\n"
#> 
#> 
#> $id
#> NULL

^{Created on 2020-06-01 by the reprex package (v0.3.0)}

[paleolimbot]

Thanks! When I get a minute I'll add this in (although if you PR it's likely it'll get added more quickly!).

[paleolimbot]

I've added a slightly more minimal fix for this (I think), but I really do appreciate the PR! I think this will let you set an option like options(rbbt.default.library_id = "//") (or pass directly to bbt_bib() or bbt_write_bib()). The options thing might be helpful if a future addin uses this (hard to specify arguments in an addin). Could you confirm that this works for you?

[jpshanno]

Just had a chance to test. Specifying library_id in bbt_bib and setting options(rbbt.default.library_id = "//") both worked with globally unique keys. Good call on setting it as an option, that's a better fix.

[jpshanno]

I was just trying to use bbt_write_bib and came across the keyscope is global error again. It looks like bbt_write_bib accepts library_id, but does not pass it on when it calls bbt_bib

[paleolimbot]

Thanks for testing and reporting!! Major oversight on my part.

[Sciurus365]

I met a similar problem with the newest better bibtex (6.7.10). I suspect it is related to https://github.com/paleolimbot/rbbt/issues/32. Unfortunately I have no knowledge about the js so can't help much. The test I did include:

With Better bibtex 6.7.10: "Keep keys unique" "within each library" solves the problem; (thanks for providing this) options(rbbt.default.library_id = "//") does NOT solve the problem

With an earlier version of Better bibtex: options(rbbt.default.library_id = "//") did solve the problem.

(Sorry I don't remember the exact version; I can't retest now either because earlier versions are not compatible with the current Zotero core. But I remember clearly that last year the combination of rolling back the BBT version PLUS using this option could solve the problem.)

[Fred-White94]

I suddenly as of 2024 get a similar issue with:

bibliography: "`r rbbt::bbt_write_bib('refs.json', overwrite = TRUE)`" (with zotero) although the error message is as follows:

Error in `result$result[[3]]`: ! subscript out of bounds Backtrace:

1. rbbt::bbt_write_bib("refs.json", overwrite = TRUE)
2. rbbt::bbt_bib(...)
3. base (local) .action(result$result[[3]])

Trying to set library_id within the function call does not seem to solve the issue, any help would be appreciated!

I also get the same error message when point and clicking through to "Update bibliography for current document from Zotero"

Cheers!

Edit the issue is with the latest release of betterbibtex. I rolled this back and now it is solved