Closed rkrug closed 1 year ago
It's more so that one of the papers contained within that JSON has connections to a paper that's failing to parse. So to replicate that last request, you can run:
x <- URLdecode("%3AW2132133878%7CW21
16272411%7CW2147324701%7CW1964905403%7CW2158562479%7CW2106351369%7CW2496196582%7CW148510
6432%7CW2163664734%7CW2080589306%7CW2554000176%7CW2005480114%7CW1982115724%7CW2025640596
%7CW2007643437%7CW2101770051%7CW1973422738%7CW1982596679%7CW2000741911%7CW2340594102%7CW
1974150937%7CW2148879975%7CW1829030053%7CW4256553588%7CW2560732491%7CW2208397176%7CW2127
626302%7CW2523325469%7CW2345739406%7CW2099626405%7CW2166465333%7CW2087638546%7CW19704395
84%7CW2611319929%7CW1965859094%7CW2034370381%7CW2112239236%7CW2571226805%7CW2028624884%7
CW1693516977%7CW2028649465%7CW2067915749%7CW2047995703%7CW2406241095%7CW2163789349%7CW21
64576867%7CW2526369002%7CW2188515674%7CW2016538523%7CW3023837187")
library(stringr)
out <- str_extract_all(str_remove_all(x, "\\s"), "W\\d+")[[1]] |>
oa_snowball(verbose = TRUE)
However I actually can't reproduce that error. I can fetch all 5113 (citing) records and the 1876 (cited_by) records (which you would've gotten to if it didn't error early).
Could it be a network connection? It happens sometimes (happened to me just a few hours ago too). Can you try just running the snippet of code above and see if you get this back?
out
#> $nodes
#> # A tibble: 7,005 × 31
#> id display_name author ab publication_date so so_id host_organization issn_l url pdf_url license version
#> <chr> <chr> <list> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
#> 1 W2132133878 Shrub Invasions… <df> "▪ A… 2000-11-01 Annu… http… Annual Reviews 0066-… http… <NA> <NA> <NA>
#> 2 W2116272411 Organic chemica… <df> "Sig… 2014-06-16 Proc… http… <NA> <NA> http… <NA> <NA> publis…
#> 3 W2147324701 CONTESTED TERRA… <df> "▪ A… 2004-11-21 Annu… http… Annual Reviews 1543-… http… <NA> <NA> publis…
#> 4 W1964905403 Decentralizatio… <df> "Thi… 2008-11-01 Annu… http… Annual Reviews 1543-… http… <NA> pd publis…
#> 5 W2158562479 Energy access i… <df> "Thi… 2010-05-01 Ener… http… Elsevier BV 0301-… http… <NA> <NA> <NA>
#> 6 W2106351369 <NA> <df> "" 2003-01-01 Huma… http… Springer Science… 0300-… http… <NA> <NA> <NA>
#> 7 W2496196582 The Potential R… <df> "Exp… 2016-07-26 Fron… http… Frontiers Media 1664-… http… <NA> cc-by publis…
#> 8 W1485106432 Policy Options … <df> "We … 2014-08-14 Cons… http… Wiley 1755-… http… <NA> <NA> <NA>
#> 9 W2163664734 Testing the use… <df> "Sum… 2008-08-01 Jour… http… Wiley-Blackwell 0021-… http… <NA> <NA> <NA>
#> 10 W2080589306 Land degradatio… <df> "Rec… 2007-01-01 Land… http… Wiley-Blackwell 1085-… http… <NA> <NA> <NA>
#> # ℹ 6,995 more rows
#> # ℹ 18 more variables: first_page <chr>, last_page <chr>, volume <chr>, issue <chr>, is_oa <lgl>, cited_by_count <int>,
#> # counts_by_year <list>, publication_year <int>, cited_by_api_url <chr>, ids <list>, doi <chr>, type <chr>,
#> # referenced_works <list>, related_works <list>, is_paratext <lgl>, is_retracted <lgl>, concepts <list>, oa_input <lgl>
#>
#> $edges
#> # A tibble: 7,068 × 2
#> from to
#> <chr> <chr>
#> 1 W2168682319 W2080589306
#> 2 W2097686432 W2000741911
#> 3 W2160537686 W2132133878
#> 4 W2336998050 W2158562479
#> 5 W2116950002 W2147324701
#> 6 W2147169550 W2132133878
#> 7 W2147716564 W2132133878
#> 8 W2157062135 W2132133878
#> 9 W2130737082 W2116272411
#> 10 W2007900882 W2132133878
#> # ℹ 7,058 more rows
OK. Worked for me as well. I am, while trying this, re-running the snowball with per_page = 50
- might help as well.
Thanks.
Great! Thanks for the tip!
Hm. Same error with:
Getting 23 pages of results with a total of 4475 records...
OpenAlex downloading [=====================] 100% eta: 0s
Requesting url: https://api.openalex.org/works?filter=cites%3AW2589424942%7CW2594233224%7CW2109428973%7CW2108887391%7CW2118353368%7CW2265892259%7CW2115119268%7CW2071911076%7CW1942798807%7CW2027109090%7CW2097039354%7CW2051207904%7CW2016044675%7CW2549733934%7CW2127806428%7CW2088887692%7CW3016484654%7CW2118476021%7CW2068064564%7CW2015353307%7CW2033502030%7CW2045404798%7CW2077285015%7CW2588584062%7CW1929584539%7CW2078907443%7CW2766942546%7CW2789191954%7CW2507663396%7CW2004014131%7CW2011151752%7CW1994021028%7CW2411108857%7CW2165933828%7CW4235181723%7CW2553303766%7CW2766957840%7CW1978887695%7CW2512305553%7CW1975749733%7CW2559981416%7CW2560959851%7CW2604706251%7CW1963540873%7CW2070099228%7CW2024320912%7CW3088921768%7CW2093809036%7CW2161073656%7CW4212887679
Getting 21 pages of results with a total of 4116 records...
OpenAlex downloading [=======>-------------] 38% eta: 22sError: lexical error: invalid char in json text.
There was an error processing y
(right here) ------^
And it works with the code you provided.
It again works for me with the same code, so unfortunately I can't diagnose the error. I'm not really sure what's going on but I think it's bouncing you somewhere with some information that we don't know the structure of, so you just get a vague parsing error.
I'll make sure to keep an eye out for that error for when it happens to me again. For now, would you like this output to be sent to you somehow?
x <- URLdecode("https://api.openalex.org/works?filter=cites%3AW2589424942%7CW2594233224%7CW2109428973%7CW2108887391%7CW2118353368%7CW2265892259%7CW2115119268%7CW2071911076%7CW1942798807%7CW2027109090%7CW2097039354%7CW2051207904%7CW2016044675%7CW2549733934%7CW2127806428%7CW2088887692%7CW3016484654%7CW2118476021%7CW2068064564%7CW2015353307%7CW2033502030%7CW2045404798%7CW2077285015%7CW2588584062%7CW1929584539%7CW2078907443%7CW2766942546%7CW2789191954%7CW2507663396%7CW2004014131%7CW2011151752%7CW1994021028%7CW2411108857%7CW2165933828%7CW4235181723%7CW2553303766%7CW2766957840%7CW1978887695%7CW2512305553%7CW1975749733%7CW2559981416%7CW2560959851%7CW2604706251%7CW1963540873%7CW2070099228%7CW2024320912%7CW3088921768%7CW2093809036%7CW2161073656%7CW4212887679")
library(stringr)
out <- str_extract_all(str_remove_all(x, "\\s"), "W\\d+")[[1]] |>
oa_snowball(verbose = TRUE)
# Requesting url: https://api.openalex.org/works?filter=openalex_id%3AW2589424942%7CW2594233224%7CW2109428973%7CW2108887391%7CW2118353368%7CW2265892259%7CW2115119268%7CW2071911076%7CW1942798807%7CW2027109090%7CW2097039354%7CW2051207904%7CW2016044675%7CW2549733934%7CW2127806428%7CW2088887692%7CW3016484654%7CW2118476021%7CW2068064564%7CW2015353307%7CW2033502030%7CW2045404798%7CW2077285015%7CW2588584062%7CW1929584539%7CW2078907443%7CW2766942546%7CW2789191954%7CW2507663396%7CW2004014131%7CW2011151752%7CW1994021028%7CW2411108857%7CW2165933828%7CW4235181723%7CW2553303766%7CW2766957840%7CW1978887695%7CW2512305553%7CW1975749733%7CW2559981416%7CW2560959851%7CW2604706251%7CW1963540873%7CW2070099228%7CW2024320912%7CW3088921768%7CW2093809036%7CW2161073656%7CW4212887679
# Getting 1 page of results with a total of 50 records...
# converting [===============================] 100% eta: 0s
# Collecting all documents citing the target papers...
# Requesting url: https://api.openalex.org/works?filter=cites%3AW2589424942%7CW2594233224%7CW2109428973%7CW2108887391%7CW2118353368%7CW2265892259%7CW2115119268%7CW2071911076%7CW1942798807%7CW2027109090%7CW2097039354%7CW2051207904%7CW2016044675%7CW2549733934%7CW2127806428%7CW2088887692%7CW3016484654%7CW2118476021%7CW2068064564%7CW2015353307%7CW2033502030%7CW2045404798%7CW2077285015%7CW2588584062%7CW1929584539%7CW2078907443%7CW2766942546%7CW2789191954%7CW2507663396%7CW2004014131%7CW2011151752%7CW1994021028%7CW2411108857%7CW2165933828%7CW4235181723%7CW2553303766%7CW2766957840%7CW1978887695%7CW2512305553%7CW1975749733%7CW2559981416%7CW2560959851%7CW2604706251%7CW1963540873%7CW2070099228%7CW2024320912%7CW3088921768%7CW2093809036%7CW2161073656%7CW4212887679
# Getting 21 pages of results with a total of 4116 records...
# OpenAlex downloading [=====================] 100% eta: 0s
# converting [===============================] 100% eta: 0s
# Collecting all documents cited by the target papers...
# Requesting url: https://api.openalex.org/works?filter=cited_by%3AW2589424942%7CW2594233224%7CW2109428973%7CW2108887391%7CW2118353368%7CW2265892259%7CW2115119268%7CW2071911076%7CW1942798807%7CW2027109090%7CW2097039354%7CW2051207904%7CW2016044675%7CW2549733934%7CW2127806428%7CW2088887692%7CW3016484654%7CW2118476021%7CW2068064564%7CW2015353307%7CW2033502030%7CW2045404798%7CW2077285015%7CW2588584062%7CW1929584539%7CW2078907443%7CW2766942546%7CW2789191954%7CW2507663396%7CW2004014131%7CW2011151752%7CW1994021028%7CW2411108857%7CW2165933828%7CW4235181723%7CW2553303766%7CW2766957840%7CW1978887695%7CW2512305553%7CW1975749733%7CW2559981416%7CW2560959851%7CW2604706251%7CW1963540873%7CW2070099228%7CW2024320912%7CW3088921768%7CW2093809036%7CW2161073656%7CW4212887679
# Getting 6 pages of results with a total of 1199 records...
# OpenAlex downloading [=====================] 100% eta: 0s
# converting [===============================] 100% eta: 0s
out
#> $nodes
#> # A tibble: 5,349 × 31
#> id display_name author ab publication_date so so_id host_organization issn_l url pdf_url license version
#> <chr> <chr> <list> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
#> 1 W2589424942 Valuing nature’… <df> "Nat… 2017-06-01 Curr… http… Elsevier BV 1877-… http… <NA> cc-by-… publis…
#> 2 W2594233224 Persistent effe… <df> "Pas… 2017-03-03 Scie… http… American Associa… 0036-… http… <NA> <NA> <NA>
#> 3 W2109428973 Home gardens: n… <df> "" 2010-09-17 Biod… http… Springer Science… 0960-… http… <NA> <NA> <NA>
#> 4 W2108887391 Fisheries asses… <df> "Fis… 1999-10-01 Cana… http… Canadian Science… 0706-… http… <NA> <NA> <NA>
#> 5 W2118353368 Arsenic contami… <df> "In … 2010-03-01 Envi… http… Elsevier BV 0269-… http… <NA> <NA> <NA>
#> 6 W2265892259 Ecosystem servi… <df> "In … 2016-08-01 Envi… http… Elsevier BV 1462-… http… <NA> <NA> <NA>
#> 7 W2115119268 Ecological Comp… <df> "Man… 2010-07-01 BioS… http… American Institu… 0006-… http… <NA> <NA> <NA>
#> 8 W2071911076 Short-term soil… <df> "The… 2006-12-01 Soil… http… Elsevier BV 0167-… http… <NA> <NA> <NA>
#> 9 W1942798807 No-take Marine … <df> "" 2006-08-01 AMBI… http… Royal Swedish Ac… 0044-… http… <NA> <NA> <NA>
#> 10 W2027109090 Lessons from In… <df> "" 2008-10-23 Envi… http… Springer Science… 0364-… http… <NA> <NA> <NA>
#> # ℹ 5,339 more rows
#> # ℹ 18 more variables: first_page <chr>, last_page <chr>, volume <chr>, issue <chr>, is_oa <lgl>, cited_by_count <int>,
#> # counts_by_year <list>, publication_year <int>, cited_by_api_url <chr>, ids <list>, doi <chr>, type <chr>,
#> # referenced_works <list>, related_works <list>, is_paratext <lgl>, is_retracted <lgl>, concepts <list>, oa_input <lgl>
#>
#> $edges
#> # A tibble: 5,455 × 2
#> from to
#> <chr> <chr>
#> 1 W2756791498 W2589424942
#> 2 W2782917446 W2589424942
#> 3 W2105691566 W2115119268
#> 4 W1973981085 W2115119268
#> 5 W2057646086 W2097039354
#> 6 W2122925767 W2016044675
#> 7 W1982116679 W2118353368
#> 8 W2788008213 W2589424942
#> 9 W2159632829 W2108887391
#> 10 W2343199820 W2071911076
#> # ℹ 5,445 more rows
As I said - this code runs for me too. I am at the moment limiting the works I use - the error is still there when limiting it to the first 1000. Will post an example if it does not go away.
Oh oops sorry I misread! Actually do you mind sharing the full context that's causing oa_snowball()
to error?
Any way for us to reproduce the invalid char in json text.
would help a lot!
Will do so. I am narrowing it down at the moment.
Hi @rkrug, I have seen this "Error: lexical error: invalid char in json text." before in #117 and #122. Could you make sure that you install the current version of openalexR from GitHub and try again?
install.packages("remotes")
remotes::install_github("ropensci/openalexR")
OK - reinstalled from GitHub newest version and re-running it.
OK - I still got the error. Can I somebody on your side send the works ids so that you can check on your side? I would not like to post it here as it should be treated as semi confidential...
OK - I still got the error. Can I somebody on your side send the works ids so that you can check on your side? I would not like to post it here as it should be treated as semi confidential...
Received. I'll take a look.
Send it to @trangdata . Here is my session info:
r$> devtools::session_info()
─ Session info ────────────────────────────────────────────────────────────────────
setting value
version R version 4.3.1 (2023-06-16)
os macOS Ventura 13.4.1
system aarch64, darwin20
ui X11
language (EN)
collate en_US.UTF-8
ctype en_US.UTF-8
tz Europe/Zurich
date 2023-07-19
pandoc 3.1.5 @ /opt/homebrew/bin/pandoc
─ Packages ────────────────────────────────────────────────────────────────────────
package * version date (UTC) lib source
cachem 1.0.7 2023-02-24 [1] CRAN (R 4.3.0)
callr 3.7.3 2022-11-02 [1] CRAN (R 4.3.0)
cli 3.6.1 2023-03-23 [1] CRAN (R 4.3.0)
colorspace 2.1-0 2023-01-23 [1] CRAN (R 4.3.0)
crayon 1.5.2 2022-09-29 [1] CRAN (R 4.3.0)
devtools 2.4.5 2022-10-11 [1] CRAN (R 4.3.0)
digest 0.6.31 2022-12-11 [1] CRAN (R 4.3.0)
dplyr * 1.1.2 2023-04-20 [1] CRAN (R 4.3.0)
ellipsis 0.3.2 2021-04-29 [1] CRAN (R 4.3.0)
fansi 1.0.4 2023-01-22 [1] CRAN (R 4.3.0)
farver 2.1.1 2022-07-06 [1] CRAN (R 4.3.0)
fastmap 1.1.1 2023-02-24 [1] CRAN (R 4.3.0)
fs 1.6.1 2023-02-06 [1] CRAN (R 4.3.0)
generics 0.1.3 2022-07-05 [1] CRAN (R 4.3.0)
ggforce 0.4.1 2022-10-04 [1] CRAN (R 4.3.0)
ggplot2 * 3.4.2 2023-04-03 [1] CRAN (R 4.3.0)
ggraph * 2.1.0 2022-10-09 [1] CRAN (R 4.3.0)
ggrepel 0.9.3 2023-02-03 [1] CRAN (R 4.3.0)
glue 1.6.2 2022-02-24 [1] CRAN (R 4.3.0)
graphlayouts 1.0.0 2023-05-01 [1] CRAN (R 4.3.0)
gridExtra 2.3 2017-09-09 [1] CRAN (R 4.3.0)
gtable 0.3.3 2023-03-21 [1] CRAN (R 4.3.0)
htmltools 0.5.5 2023-03-23 [1] CRAN (R 4.3.0)
htmlwidgets 1.6.2 2023-03-17 [1] CRAN (R 4.3.0)
httpuv 1.6.9 2023-02-14 [1] CRAN (R 4.3.0)
igraph 1.4.2 2023-04-07 [1] CRAN (R 4.3.0)
jsonlite 1.8.7 2023-06-29 [1] CRAN (R 4.3.0)
knitr * 1.42 2023-01-25 [1] CRAN (R 4.3.0)
later 1.3.0 2021-08-18 [1] CRAN (R 4.3.0)
lifecycle 1.0.3 2022-10-07 [1] CRAN (R 4.3.0)
magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.3.0)
MASS 7.3-59 2023-04-21 [1] CRAN (R 4.3.0)
memoise 2.0.1 2021-11-26 [1] CRAN (R 4.3.0)
mime 0.12 2021-09-28 [1] CRAN (R 4.3.0)
miniUI 0.1.1.1 2018-05-18 [1] CRAN (R 4.3.0)
munsell 0.5.0 2018-06-12 [1] CRAN (R 4.3.0)
openalexR * 1.1.2.9 2023-07-19 [1] Github (ropensci/openalexR@66f0743)
pillar 1.9.0 2023-03-22 [1] CRAN (R 4.3.0)
pkgbuild 1.4.0 2022-11-27 [1] CRAN (R 4.3.0)
pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.3.0)
pkgload 1.3.2 2022-11-16 [1] CRAN (R 4.3.0)
polyclip 1.10-4 2022-10-20 [1] CRAN (R 4.3.0)
prettyunits 1.1.1 2020-01-24 [1] CRAN (R 4.3.0)
processx 3.8.1 2023-04-18 [1] CRAN (R 4.3.0)
profvis 0.3.7 2020-11-02 [1] CRAN (R 4.3.0)
promises 1.2.0.1 2021-02-11 [1] CRAN (R 4.3.0)
ps 1.7.5 2023-04-18 [1] CRAN (R 4.3.0)
purrr 1.0.1 2023-01-10 [1] CRAN (R 4.3.0)
R6 2.5.1 2021-08-19 [1] CRAN (R 4.3.0)
Rcpp 1.0.10 2023-01-22 [1] CRAN (R 4.3.0)
remotes 2.4.2 2021-11-30 [1] CRAN (R 4.3.0)
rlang 1.1.1 2023-04-28 [1] CRAN (R 4.3.0)
scales 1.2.1 2022-08-20 [1] CRAN (R 4.3.0)
sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.3.0)
shiny 1.7.4 2022-12-15 [1] CRAN (R 4.3.0)
stringi 1.7.12 2023-01-11 [1] CRAN (R 4.3.0)
stringr 1.5.0 2022-12-02 [1] CRAN (R 4.3.0)
tibble 3.2.1 2023-03-20 [1] CRAN (R 4.3.0)
tidygraph * 1.2.3 2023-02-01 [1] CRAN (R 4.3.0)
tidyr 1.3.0 2023-01-24 [1] CRAN (R 4.3.0)
tidyselect 1.2.0 2022-10-10 [1] CRAN (R 4.3.0)
tweenr 2.0.2 2022-09-06 [1] CRAN (R 4.3.0)
urlchecker 1.0.1 2021-11-30 [1] CRAN (R 4.3.0)
usethis 2.1.6 2022-05-25 [1] CRAN (R 4.3.0)
utf8 1.2.3 2023-01-31 [1] CRAN (R 4.3.0)
vctrs 0.6.3 2023-06-14 [1] CRAN (R 4.3.0)
viridis 0.6.2 2021-10-13 [1] CRAN (R 4.3.0)
viridisLite 0.4.1 2022-08-22 [1] CRAN (R 4.3.0)
withr 2.5.0 2022-03-03 [1] CRAN (R 4.3.0)
xfun 0.39 2023-04-20 [1] CRAN (R 4.3.0)
xtable 1.8-4 2019-04-21 [1] CRAN (R 4.3.0)
[1] /Users/rainerkrug/R/library/aarch64-apple-darwin20/4.3
[2] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
───────────────────────────────────────────────────────────────────────────────────
I did it in batches, and it worked. So I assume, it was a memory issue.
Hi
I have about 200 articles from which I want to use
snowball()
which works fine, but in the middle:Assuming that the last URL mentioned is causing the problem, I tried
and it worked with no problems.
Any suggestions how I can debug this?
Thanks.