search

crsh / citr

RStudio Addin to Insert Markdown Citations
Other
408 stars 46 forks source link

Umlaute not correctly displayed in preview #44

Open TobiasHeycke opened 5 years ago

TobiasHeycke commented 5 years ago

When loading the following citation, the citr plugin does not correctly display the ü. The bib file is utf-8 encoded.

@article{hutter_dissociating_2012,
    title = {Dissociating contingency awareness and conditioned attitudes: {Evidence} of contingency-unaware evaluative conditioning.},
    volume = {141},
    issn = {1939-2222, 0096-3445},
    shorttitle = {Dissociating contingency awareness and conditioned attitudes},
    url = {http://doi.apa.org/getdoi.cfm?doi=10.1037/a0026477},
    doi = {10.1037/a0026477},
    number = {3},
    urldate = {2013-03-07},
    journal = {Journal of Experimental Psychology: General},
    author = {Hütter, Mandy and Sweldens, Steven and Stahl, Christoph and Unkelbach, Christian and Klauer, Karl Christoph},
    year = {2012},
    keywords = {*Associative Processes, *Attitude Formation, *Awareness, *Conditioning, *Evaluation, Affective Valence, Associative processes, Awareness, Conditioning, Contingency awareness, Evaluative conditioning, Attitude formation, Evaluation, Process-dissociation procedure, \_tablet, \_tablet\_modified, multinomial modeling},
    pages = {539--557},
    file = {Hütter_etal_2012.pdf:/home/theycke/Zotero/storage/48DPA3WI/Hütter_etal_2012.pdf:application/pdf;Hütter_etal_2012.pdf:/home/theycke/Zotero/storage/6HZQQ2HA/Hütter_etal_2012.pdf:application/pdf;Hütter_etal_2012.pdf:/home/theycke/Zotero/storage/7AFEWMN5/Hütter_etal_2012.pdf:application/pdf}
}
crsh commented 5 years ago

Thanks, Tobias. I just tried to recreate this problem on my Mac and I can't. It seems this issue may be related to #41. Could you try to create a reference file on your machine that includes just the above reference and upload it here, so I can take a look at it?

crsh commented 5 years ago

I just pushed a commit that might help with this. Before upload the file, could you check wether the problem persists?

TobiasHeycke commented 5 years ago

I installed the current version of citr from github and exported the attached biblatex file from zotero using utf-8 encoding (I also tested bibtex), the issue is still there. RStudio 1.1.463 (R 3.5.1)

File: citr_umlaute.zip

crsh commented 5 years ago

Thanks, I'll take another look some time today.

crsh commented 5 years ago

Hmm, still no luck reproducing the error. Could you try setting the locale to the following:

Sys.setlocale(category="LC_ALL", locale = "de_DE.UTF-8")

Also, could you provide the output of devtools::session_info()?

crsh commented 5 years ago

I think I found it. This seems to be a Windows-specific problem. I just pushed a commit that should fix this. Let me know if this works for you.

TobiasHeycke commented 5 years ago

I still have the same issue. I tried setting sys.setlocal to German:

Sys.setlocale("LC_ALL", "German")

(The other sys.setlocal call does not work on a Windows OS)

My session info is the following:

Session info -----------------------------------------------------------------------------------------------------------
 setting  value                       
 version  R version 3.5.1 (2018-07-02)
 os       Windows 7 x64 SP 1          
 system   x86_64, mingw32             
 ui       RStudio                     
 language (EN)                        
 collate  German_Germany.1252         
 ctype    German_Germany.1252         
 tz       Europe/Berlin               
 date     2018-12-04                  

Packages -----------------------------------------------------------------------------------------------------------
 package     * version    date       lib source                    
 assertthat    0.2.0      2017-04-11 [1] CRAN (R 3.5.1)            
 backports     1.1.2      2017-12-13 [1] CRAN (R 3.5.0)            
 base64enc     0.1-3      2015-07-28 [1] CRAN (R 3.5.0)            
 bibtex        0.4.2      2017-06-30 [1] CRAN (R 3.5.1)            
 callr         3.0.0      2018-08-24 [1] CRAN (R 3.5.1)            
 citr          0.2.0.9055 2018-12-04 [1] Github (crsh/citr@456ff20)
 cli           1.0.1      2018-09-25 [1] CRAN (R 3.5.1)            
 crayon        1.3.4      2017-09-16 [1] CRAN (R 3.5.1)            
 curl          3.2        2018-03-28 [1] CRAN (R 3.5.1)            
 desc          1.2.0      2018-05-01 [1] CRAN (R 3.5.1)            
 devtools      2.0.1      2018-10-26 [1] CRAN (R 3.5.1)            
 digest        0.6.18     2018-10-10 [1] CRAN (R 3.5.1)            
 fs            1.2.6      2018-08-23 [1] CRAN (R 3.5.1)            
 glue          1.3.0      2018-07-17 [1] CRAN (R 3.5.1)            
 htmltools     0.3.6      2017-04-28 [1] CRAN (R 3.5.1)            
 httpuv        1.4.5      2018-07-19 [1] CRAN (R 3.5.1)            
 httr          1.3.1      2017-08-20 [1] CRAN (R 3.5.1)            
 jsonlite      1.5        2017-06-01 [1] CRAN (R 3.5.1)            
 later         0.7.5      2018-09-18 [1] CRAN (R 3.5.1)            
 lubridate     1.7.4      2018-04-11 [1] CRAN (R 3.5.1)            
 magrittr      1.5        2014-11-22 [1] CRAN (R 3.5.1)            
 memoise       1.1.0      2017-04-21 [1] CRAN (R 3.5.1)            
 mime          0.6        2018-10-05 [1] CRAN (R 3.5.1)            
 miniUI        0.1.1.1    2018-05-18 [1] CRAN (R 3.5.1)            
 pkgbuild      1.0.2      2018-10-16 [1] CRAN (R 3.5.1)            
 pkgload       1.0.2      2018-10-29 [1] CRAN (R 3.5.1)            
 plyr          1.8.4      2016-06-08 [1] CRAN (R 3.5.1)            
 prettyunits   1.0.2      2015-07-13 [1] CRAN (R 3.5.1)            
 processx      3.2.0      2018-08-16 [1] CRAN (R 3.5.1)            
 promises      1.0.1      2018-04-13 [1] CRAN (R 3.5.1)            
 ps            1.2.1      2018-11-06 [1] CRAN (R 3.5.1)            
 R6            2.3.0      2018-10-04 [1] CRAN (R 3.5.1)            
 Rcpp          1.0.0      2018-11-07 [1] CRAN (R 3.5.1)            
 RefManageR    1.2.0      2018-04-25 [1] CRAN (R 3.5.1)            
 remotes       2.0.2      2018-10-30 [1] CRAN (R 3.5.1)            
 rlang         0.3.0.1    2018-10-25 [1] CRAN (R 3.5.1)            
 rprojroot     1.3-2      2018-01-03 [1] CRAN (R 3.5.1)            
 rstudioapi    0.8        2018-10-02 [1] CRAN (R 3.5.1)            
 sessioninfo   1.1.1      2018-11-05 [1] CRAN (R 3.5.1)            
 shiny       * 1.2.0      2018-11-02 [1] CRAN (R 3.5.1)            
 shinyjs       1.0        2018-01-08 [1] CRAN (R 3.5.1)            
 stringi       1.2.4      2018-07-20 [1] CRAN (R 3.5.1)            
 stringr       1.3.1      2018-05-10 [1] CRAN (R 3.5.1)            
 testthat      2.0.1      2018-10-13 [1] CRAN (R 3.5.1)            
 usethis       1.4.0      2018-08-14 [1] CRAN (R 3.5.1)            
 withr         2.1.2      2018-03-15 [1] CRAN (R 3.5.1)            
 xml2          1.2.0      2018-01-24 [1] CRAN (R 3.5.1)            
 xtable        1.8-3      2018-08-29 [1] CRAN (R 3.5.1)            
 yaml          2.2.0      2018-07-25 [1] CRAN (R 3.5.1)
TobiasHeycke commented 5 years ago

The Umlaute are also not correctly displayed in my Win10 machine. Just to make sure I didn't forget anything, I did the following:

  1. install citr from github: devtools::install_github("crsh/citr")
  2. restart R
  3. used the bib file I uploaded here
  4. start the citr plugin

I also tested it with Sys.setlocale("LC_ALL", "German").

The session infos:

setting  value                       
 version  R version 3.4.1 (2017-06-30)
 system   x86_64, mingw32             
 ui       RStudio (1.1.463)           
 language (EN)                        
 collate  German_Germany.1252         
 tz       Europe/Berlin               
 date     2018-12-07                  

Packages ------------------------------------------------------------------------------------------------
 package    * version    date       source                    
[...]      
 citr         0.2.0.9055 2018-12-07 Github (crsh/citr@b976667)
[...]`
crsh commented 5 years ago

Thanks for the update. I have tried to reproduce this on the Windows machine I have available (Windows 8, R 3.4.2 and the same set of packages and versions as reported above) and it worked as expected. To be honest, I'm somewhat at a loss, why it's not working for you. But I'll keep looking.

crsh commented 5 years ago

Hi Tobi, thanks for helping me debug this on your machine! 🙇I think I finally fixed the problem. If you find the time, please try again and confirm that it indeed works as expected.

TobiasHeycke commented 5 years ago

Actually, I now only get the cite key displayed in the preview (and not the full reference). Can you reproduce this error?

crsh commented 5 years ago

You are right, of course. Sorry about that. I just pushed a fix for that. Could you try (hopefully) one last time?

TobiasHeycke commented 5 years ago

It works! Thank you

TobiasHeycke commented 5 years ago

I have the same problem again (most current citr version from github). Did you change anything back by accident?

crsh commented 5 years ago

Hmm, no I haven't touched those bits of code. Can I use the above reproducible example to debug this?

TobiasHeycke commented 5 years ago

I had the encoding error on multiple machines. I can reproduce it with the bib file I uploaded (as a zip file) on November 26th (see above). Sys.setlocale("LC_ALL", "German") also did not help. Anything you need so I can help? Session info below. 

- Session info --------------------------------------------------------------------------------------------------------------
 setting  value                       
 version  R version 3.5.3 (2019-03-11)
 os       Windows 7 x64 SP 1          
 system   x86_64, mingw32             
 ui       RStudio                     
 language (EN)                        
 collate  German_Germany.1252         
 ctype    German_Germany.1252         
 tz       Europe/Berlin               
 date     2019-04-17                  

- Packages ------------------------------------------------------------------------------------------------------------------
 package     * version date       lib source                    
 assertthat    0.2.1   2019-03-21 [1] CRAN (R 3.5.3)            
 backports     1.1.4   2019-04-10 [1] CRAN (R 3.5.3)            
 bibtex        0.4.2   2017-06-30 [1] CRAN (R 3.5.3)            
 callr         3.2.0   2019-03-15 [1] CRAN (R 3.5.3)            
 citr          0.3.0   2019-04-17 [1] Github (crsh/citr@d4696f4)
 cli           1.1.0   2019-03-19 [1] CRAN (R 3.5.3)            
 crayon        1.3.4   2017-09-16 [1] CRAN (R 3.5.3)            
 curl          3.3     2019-01-10 [1] CRAN (R 3.5.3)            
 desc          1.2.0   2018-05-01 [1] CRAN (R 3.5.3)            
 devtools      2.0.2   2019-04-08 [1] CRAN (R 3.5.3)            
 digest        0.6.18  2018-10-10 [1] CRAN (R 3.5.3)            
 fs            1.2.7   2019-03-19 [1] CRAN (R 3.5.3)            
 glue          1.3.1   2019-03-12 [1] CRAN (R 3.5.3)            
 htmltools     0.3.6   2017-04-28 [1] CRAN (R 3.5.3)            
 httpuv        1.5.1   2019-04-05 [1] CRAN (R 3.5.3)            
 httr          1.4.0   2018-12-11 [1] CRAN (R 3.5.3)            
 jsonlite      1.6     2018-12-07 [1] CRAN (R 3.5.3)            
 later         0.8.0   2019-02-11 [1] CRAN (R 3.5.3)            
 lubridate     1.7.4   2018-04-11 [1] CRAN (R 3.5.3)            
 magrittr      1.5     2014-11-22 [1] CRAN (R 3.5.3)            
 memoise       1.1.0   2017-04-21 [1] CRAN (R 3.5.3)            
 mime          0.6     2018-10-05 [1] CRAN (R 3.5.2)            
 miniUI        0.1.1.1 2018-05-18 [1] CRAN (R 3.5.3)            
 pkgbuild      1.0.3   2019-03-20 [1] CRAN (R 3.5.3)            
 pkgload       1.0.2   2018-10-29 [1] CRAN (R 3.5.3)            
 plyr          1.8.4   2016-06-08 [1] CRAN (R 3.5.3)            
 prettyunits   1.0.2   2015-07-13 [1] CRAN (R 3.5.3)            
 processx      3.3.0   2019-03-10 [1] CRAN (R 3.5.3)            
 promises      1.0.1   2018-04-13 [1] CRAN (R 3.5.3)            
 ps            1.3.0   2018-12-21 [1] CRAN (R 3.5.3)            
 R6            2.4.0   2019-02-14 [1] CRAN (R 3.5.3)            
 Rcpp          1.0.1   2019-03-17 [1] CRAN (R 3.5.3)            
 RefManageR    1.2.12  2019-04-03 [1] CRAN (R 3.5.3)            
 remotes       2.0.4   2019-04-10 [1] CRAN (R 3.5.3)            
 rlang         0.3.4   2019-04-07 [1] CRAN (R 3.5.3)            
 rprojroot     1.3-2   2018-01-03 [1] CRAN (R 3.5.3)            
 rstudioapi    0.10    2019-03-19 [1] CRAN (R 3.5.3)            
 sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 3.5.3)            
 shiny       * 1.3.1   2019-04-12 [1] CRAN (R 3.5.3)            
 shinyjs       1.0     2018-01-08 [1] CRAN (R 3.5.3)            
 stringi       1.4.3   2019-03-12 [1] CRAN (R 3.5.3)            
 stringr       1.4.0   2019-02-10 [1] CRAN (R 3.5.3)            
 usethis       1.5.0   2019-04-07 [1] CRAN (R 3.5.3)            
 withr         2.1.2   2018-03-15 [1] CRAN (R 3.5.3)            
 xml2          1.2.0   2018-01-24 [1] CRAN (R 3.5.3)            
 xtable        1.8-3   2018-08-29 [1] CRAN (R 3.5.3)            
 yaml          2.2.0   2018-07-25 [1] CRAN (R 3.5.2)
crsh commented 5 years ago

Hmm, I'm not sure, but this may be a problem with RefManageR::ReadBib() on Windows:

bib <- RefManageR::ReadBib("citr_umlaute.bib", .Encoding = "UTF-8")
bib$author[1]$family
[1] "Högden"

I can read the file into R without problems using readLines():

> readLines("citr_umlaute.bib", encoding = "UTF-8")[12]
[1] "\tauthor = {Högden, Fabia and Hütter, Mandy and Unkelbach, Christian},"

cc @mwmclean

crsh commented 5 years ago

@TobiasHeycke can you confirm that the issue can be observed within RefManageR::ReadBib() (see post above)? If so, would you be willing to raise the issue with the maintainers of RefManageR?

TobiasHeycke commented 5 years ago

I can confirm that I can replicate the problem using ReadBib (and using readLines works fine). If I am not mistaken the problem has been raised already: https://github.com/ropensci/RefManageR/issues/61

christianlindell commented 5 years ago

I can confim the bug. I have the same problem with swedish characters with umlauts. I'm running R 3.6.1, Windows 10 and RStudio 1.2.1578.

mwmclean commented 5 years ago

@crsh I'm currently unable to reproduce the output you obtain with readLines() on my Windows machine. Does bibtext::read.bib() give the correct output? I'm a bit rusty, but I've wasted many, many hours in the past trying to help people with encoding issues with R/RefManageR and Windows with extremely little success, unfortunately.

R Under development (unstable) (2019-10-02 r77250)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17134)

l10n_info()["UTF-8"] is FALSE for me.