rstudio / rstudio

RStudio is an integrated development environment (IDE) for R
https://posit.co/products/open-source/rstudio/
Other
4.65k stars 1.09k forks source link

Backslashes are added to citations when using the RMarkdown Visual Editor, resulting in the incorrect rendering of documents #10075

Closed pedrohbraga closed 2 years ago

pedrohbraga commented 2 years ago

System details

RStudio Edition : Desktop
RStudio Version :  2021.09.0 Build 351 - Ghost Orchid" Release (077589bc, 2021-09-20) for Windows
OS Version      :  Windows 10 x64 (build 22000)
R Version       : R version 4.1.2 (2021-11-01)

Steps to reproduce the problem

  1. Create and save the following .Rmd document within the desired directory in your system:
---
title: "Reproduce issue with backslashes in citations"
author: "Pedro Henrique P. Braga"
date: "11/11/2021"
bibliography: references.bib  
output: pdf_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)

R Markdown

This is an R Markdown document. Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents. For more details on using R Markdown see http://rmarkdown.rstudio.com.


2. Create a `references.bib` file containing the code chunk below and save it to the same directory as the `.Rmd` file:

@Manual{R-rmarkdown, title = {rmarkdown: Dynamic Documents for R}, author = {JJ Allaire and Yihui Xie and Jonathan McPherson and Javier Luraschi and Kevin Ushey and Aron Atkins and Hadley Wickham and Joe Cheng and Winston Chang and Richard Iannone}, year = {2020}, note = {R package version 2.6}, url = {https://github.com/rstudio/rmarkdown}, }



3.  Open the `.Rmd` file in RStudio and switch from the script-based view to the "Visual Editor" (shortcut *Ctrl or Command+Shift+F4*);

4. Add a citation by typing `[@R-rmarkdown]` after the first paragraph in the prose written below the YAML;

![image](https://user-images.githubusercontent.com/8599229/141369911-2e1d1e90-6719-48da-937e-e5672170993a.png)

5. Knit the document.

6. Disable the "Visual Editor" (shortcut *`Ctrl` or `Command`+`Shift`+`F4`*).

### Describe the problem in detail

Backslashes are added to citations created in the "Visual Editor" mode. They are apparent when switching back from the Visual Editor mode, and this results in the incorrect rendering of citations and references. See the output of the document below:

![image](https://user-images.githubusercontent.com/8599229/141370071-a7e76ac0-bddb-402e-a4f5-2aeab0f1f6a3.png)

![image](https://user-images.githubusercontent.com/8599229/141369439-52e193db-cdb0-4b7c-a053-39c7fba1fa4a.png)

### Describe the behaviour you expected

The expectation is that citations added when using the "Visual Editor" mode are reflected correctly in the code-based view of the `.Rmd` script and that citations and references are rendered correctly within the PDF or HTML document.

<!-- 
Please keep the below portion in your issue, and check `[x]` the applicable boxes.
-->

- [x] I have read the guide for [submitting good bug reports](https://github.com/rstudio/rstudio/wiki/Writing-Good-Bug-Reports).
- [x] I have installed the latest version of RStudio, and confirmed that the issue still persists.
- [x] If I am reporting an RStudio crash, I have included a [diagnostics report](https://support.rstudio.com/hc/en-us/articles/200321257-Running-a-Diagnostics-Report).
- [x] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
pedrohbraga commented 2 years ago

I am also adding another example, non-reproducible, which also demonstrates this same issue I am having, at a larger scale.

  1. Here is the text I am writing in the Visual Editor:

image

  1. Here is what appears when I deactivate the Visual Editor:

image

  1. Here is the rendered document:

image

Note how citations are not correctly rendered and how backslashes are added to the citations within the .Rmd document.

Dijkie85 commented 2 years ago

Not sure if it's related to this, but I found that working with multiple child Rmd's will result in duplicate references between documents, because the visual editor numbers each reference in each document without regard for previous children. So, even if the first reference in 2.Rmd is "[^2-1]" it will be converted to "[^1"] and overlap with references from 1.Rmd.

ronblum commented 2 years ago

@pedrohbraga Thank you for raising this! We've certain cases in which the backslashes are added incorrectly. However, I can't reproduce this problem: It works for me in both

Did this problem occur for you in previous versions of RStudio, such as 1.4.1717 or earlier?

@Dijkie85 The issue with citations in children files appears to be a different problem. Can you file a new issue for this? It would also be helpful to include sample files, both the parent and child files. Thanks!

balthasars commented 2 years ago

I have the same issue in RStudio version 2021.9.1.372 on macOS 11.6. @ronblum is there a way of switching into the visual mode reproducible? In my case, somewhat strangely, not all citations in the document end up being formatted like this, however. Some retain the correct formatting

Dijkie85 commented 2 years ago

@ronblum Will do. Thanks!

edit: opened #10078

ronblum commented 2 years ago

@balthasars Would you be willing to provide an example of a file in which some citations keep the correct formatting and others don't? I can try it and see if I see the same issue.

pedrohbraga commented 2 years ago

@ronblum and others,

Here follows another example that should be reproducible in your environment and that (hopefully) brings to the same outcome I am having here.

  1. Save the content below to an .Rmd file:
---
output:
  bookdown::pdf_document2:
    latex_engine: xelatex
    toc: false
    self_contained: true
  bookdown::html_document2:
    default
title: "Little reproducible example"
author: "Pedro Henrique P. Braga"
date: "`r Sys.Date()`"
abstract: "Answering how quickly species climatic niches evolve is essential to understand biogeographical patterns and to predict how species will respond to climate change."
geometry: margin = 1in
fontsize: 11pt
spacing: double
documentclass: article
mainfont: "Times New Roman"
editor_options:
  markdown:
    wrap: sentence
bibliography: main_references.bib
csl: https://raw.githubusercontent.com/marlonecobos/kuenm/master/ecography.csl
link-citations: yes
header-includes:
  - \usepackage[left]{lineno}
  - \linenumbers
  - \usepackage{setspace}\doublespacing
  - \setlength{\parindent}{4em}
  - \setlength{\parskip}{0em}
---

```{r setup, include=FALSE}
library(bookdown)

knitr::opts_chunk$set(tidy = FALSE, 
                      comment = "#",
                      cache = TRUE,
                      warning = FALSE,
                      message = FALSE,
                      fig.width = 6, 
                      fig.height = 6
)

options(htmltools.dir.version = FALSE)

Introduction

Every species has a realized climatic niche (henceforward, "climatic niche"), which is determined by their responses and underlying tolerances to the set of long-term meteorological conditions that allow them to occur when and where they are [@holt2009p; @hutchinson1957cshsqb; @soberon2017po]. Hence, answering questions such as "What determines species climatic affinities and limitations? [@carscadden2020qrb]", "How climatic niches contribute to the formation and maintenance of biodiversity patterns? [@cooney2016jae; @kozak2010el]", "Do certain climatic niche characteristics make species more vulnerable to the ongoing and projected climate change? [@rinnan2019e; @smith2020jb]" , "Can rapid climatic niche shifts attenuate climate change driven extinctions? [@quintero2013el; @jezkova2016pbs]", "Is the evolution of climatic niches similar across groups? [@liu2020]" has been of critical interest for biogeographers, ecologists and conservation biologists.

References


2. Save the content below to a `main_references.bib` file, within the same directory as the `.Rmd` file :

@article{holt2009p, title = {Bringing the Hutchinsonian niche into the 21st century: Ecological and evolutionary perspectives}, author = {{Holt}, {Robert D.}}, year = {2009}, month = {11}, date = {2009-11-17}, journal = {Proceedings of the National Academy of Sciences}, pages = {19659--19665}, volume = {106}, number = {Supplement 2}, doi = {10.1073/pnas.0905137106}, url = {https://www.pnas.org/content/106/Supplement_2/19659}, note = {Publisher: National Academy of Sciences Section: Colloquium Paper PMID: 19903876}, langid = {en} }

@article{hutchinson1957cshsqb, title = {Concluding Remarks}, author = {{Hutchinson}, {G. Evelyn}}, year = {1957}, month = {01}, date = {1957-01-01}, journal = {Cold Spring Harbor Symposia on Quantitative Biology}, pages = {415--427}, volume = {22}, doi = {10.1101/SQB.1957.022.01.039}, url = {http://symposium.cshlp.org/content/22/415}, note = {Publisher: Cold Spring Harbor Laboratory Press}, langid = {en} }

@article{soberon2017po, title = {Are fundamental niches larger than the realized? Testing a 50-year-old prediction by Hutchinson}, author = {{Soberón}, {J.} and {Arroyo-Peña}, {B.}}, year = {2017}, month = {04}, date = {2017-04-12}, journal = {PLOS ONE}, pages = {e0175138}, volume = {12}, number = {4}, doi = {10.1371/journal.pone.0175138}, url = {https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0175138}, note = {Publisher: Public Library of Science}, langid = {en} }

@article{liu2020, title = {Climatic-niche evolution follows similar rules in plants and animals}, author = {{Liu}, {Hui} and {Ye}, {Qing} and {Wiens}, {John J.}}, year = {2020}, month = {05}, date = {2020-05}, journal = {Nature Ecology & Evolution}, pages = {753--763}, volume = {4}, number = {5}, doi = {10.1038/s41559-020-1158-x}, url = {http://www.nature.com/articles/s41559-020-1158-x}, note = {Bandiera{_}abtest: a Cg{_}type: Nature Research Journals Number: 5 Primary{_}atype: Research Publisher: Nature Publishing Group Subject{_}term: Ecology;Evolution Subject{_}term{_}id: ecology;evolution}, langid = {en} }

@article{rangel2018s, title = {Modeling the ecology and evolution of biodiversity: Biogeographical cradles, museums, and graves}, author = {{Rangel}, {Thiago F.} and {Edwards}, {Neil R.} and {Holden}, {Philip B.} and {Diniz-Filho}, {José Alexandre F.} and {Gosling}, {William D.} and {Coelho}, {Marco Túlio P.} and {Cassemiro}, {Fernanda A. S.} and {Rahbek}, {Carsten} and {Colwell}, {Robert K.}}, year = {2018}, month = {07}, date = {2018-07-20}, journal = {Science}, pages = {eaar5452}, volume = {361}, number = {6399}, doi = {10.1126/science.aar5452}, url = {http://www.science.org/doi/10.1126/science.aar5452}, note = {Publisher: American Association for the Advancement of Science} }

@article{dimarco2021e, title = {Drivers of change in the realised climatic niche of terrestrial mammals}, author = {{Di Marco}, {Moreno} and {Pacifici}, {Michela} and {Maiorano}, {Luigi} and {Rondinini}, {Carlo}}, year = {2021}, date = {2021}, journal = {Ecography}, pages = {1180--1190}, volume = {44}, number = {8}, doi = {10.1111/ecog.05414}, url = {http://onlinelibrary.wiley.com/doi/abs/10.1111/ecog.05414}, note = {{_}eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/ecog.05414}, langid = {en} }

@article{liu2017jb, title = {Realized climatic niches are conserved along maximum temperatures among herpetofaunal invaders}, author = {{Liu}, {Xuan} and {Petitpierre}, {Blaise} and {Broennimann}, {Olivier} and {Li}, {Xianping} and {Guisan}, {Antoine} and {Li}, {Yiming}}, year = {2017}, date = {2017}, journal = {Journal of Biogeography}, pages = {111--121}, volume = {44}, number = {1}, doi = {10.1111/jbi.12808}, url = {http://onlinelibrary.wiley.com/doi/abs/10.1111/jbi.12808}, note = {{_}eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/jbi.12808}, langid = {en} }

@article{quintero2013el, title = {Rates of projected climate change dramatically exceed past rates of climatic niche evolution among vertebrate species}, author = {Quintero, Ignacio and Wiens, John J.}, year = {2013}, month = {08}, date = {2013-08}, journal = {Ecology Letters}, pages = {1095--1103}, volume = {16}, number = {8}, doi = {10.1111/ele.12144}, note = {PMID: 23800223}, langid = {eng} }

@article{carscadden2020qrb, title = {Niche Breadth: Causes and Consequences for Ecology, Evolution, and Conservation}, author = {Carscadden, Kelly A. and Emery, Nancy C. and Arnillas, Carlos A. and Cadotte, Marc W. and Afkhami, Michelle E. and Gravel, Dominique and Livingstone, Stuart W. and Wiens, John J.}, year = {2020}, month = {09}, date = {2020-09-01}, journal = {The Quarterly Review of Biology}, pages = {179--214}, volume = {95}, number = {3}, doi = {10.1086/710388}, url = {http://www.journals.uchicago.edu/doi/10.1086/710388}, note = {Publisher: The University of Chicago Press} }

@article{cooney2016jae, title = {Widespread correlations between climatic niche evolution and species diversification in birds}, author = {Cooney, Christopher R. and Seddon, Nathalie and Tobias, Joseph A.}, year = {2016}, date = {2016}, journal = {Journal of Animal Ecology}, pages = {869--878}, volume = {85}, number = {4}, doi = {10.1111/1365-2656.12530}, url = {http://onlinelibrary.wiley.com/doi/abs/10.1111/1365-2656.12530}, note = {{_}eprint: https://besjournals.onlinelibrary.wiley.com/doi/pdf/10.1111/1365-2656.12530}, langid = {en} }

@article{jezkova2016pbs, title = {Rates of change in climatic niches in plant and animal populations are much slower than projected climate change}, author = {Jezkova, Tereza and Wiens, John J.}, year = {2016}, month = {11}, date = {2016-11-30}, journal = {Proceedings of the Royal Society B: Biological Sciences}, pages = {20162104}, volume = {283}, number = {1843}, doi = {10.1098/rspb.2016.2104}, url = {https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5136599/}, note = {PMID: 27881748 PMCID: PMC5136599} }

@article{rinnan2019e, title = {Climate-niche factor analysis: a spatial approach to quantifying species vulnerability to climate change}, author = {Rinnan, D. Scott and Lawler, Joshua}, year = {2019}, date = {2019}, journal = {Ecography}, pages = {1494--1503}, volume = {42}, number = {9}, doi = {10.1111/ecog.03937}, url = {http://onlinelibrary.wiley.com/doi/abs/10.1111/ecog.03937}, note = {{_}eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/ecog.03937}, langid = {en} }

@article{kozak2010el, title = {Accelerated rates of climatic-niche evolution underlie rapid species diversification}, author = {Kozak, Kenneth H. and Wiens, John J.}, year = {2010}, date = {2010}, journal = {Ecology Letters}, pages = {1378--1389}, volume = {13}, number = {11}, doi = {10.1111/j.1461-0248.2010.01530.x}, url = {http://onlinelibrary.wiley.com/doi/abs/10.1111/j.1461-0248.2010.01530.x}, note = {{_}eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1461-0248.2010.01530.x}, langid = {en} }

@article{castro-insua2018sr, title = {Climatic niche divergence drives patterns of diversification and richness among mammal families}, author = {Castro-Insua, {Adrián} and {Gómez-Rodríguez}, Carola and Wiens, John J. and Baselga, {Andrés}}, year = {2018}, month = {06}, date = {2018-06-08}, journal = {Scientific Reports}, pages = {8781}, volume = {8}, number = {1}, doi = {10.1038/s41598-018-27068-y}, note = {PMID: 29884843 PMCID: PMC5993713}, langid = {eng} }

@article{smith2020jb, title = {Climatic niche limits and community-level vulnerability of obligate symbioses}, author = {Smith, Robert J. and Jovan, Sarah and McCune, Bruce}, year = {2020}, date = {2020}, journal = {Journal of Biogeography}, pages = {382--395}, volume = {47}, number = {2}, doi = {10.1111/jbi.13719}, url = {http://onlinelibrary.wiley.com/doi/abs/10.1111/jbi.13719}, note = {{_}eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/jbi.13719}, langid = {en} }


3. Switch to Visual Editor mode;

4. Add the citation `@castro-insua2018sr` to the sentence "_How climatic niches contribute to the formation and maintenance of biodiversity patterns?_"`, as follows:

"*How climatic niches contribute to the formation and maintenance of biodiversity patterns?* [@cooney2016jae; @kozak2010el; @castro-insua2018sr]"


Note: Instead of copying and pasting, type `; @castro-insua2018sr` within the citation tag that already exists there.

5. Disable the Visual Editor.

## Expected behaviour

Following Step 5, you should see that backslashes have been added before the square brackets in all citations.

![image](https://user-images.githubusercontent.com/8599229/141825705-ef9c0dc7-3621-41b3-b37b-2065f2cae63c.png)

Please let me know if you need a video to be recorded or further information about my system and about my RStudio and R sessions.
bryanpmayfield commented 2 years ago

I am having the same exact issue using RStudio version 2021.09.0 on MaCOS 12.0.1.

Also, not sure if this has anything to do with the canonical visual mode setting, but just in case I have tried it both with and without that setting set to true (in both global options and project options) and the problem persists.

@pedrohbraga Have you found a work around for this?

ronblum commented 2 years ago

@pedrohbraga Thank you! I can reproduce this issue based on your code and precise steps—I appreciate it. Specifically, tested using RStudio Desktop 2021.09.0+351 on MacOS 12. I was also able to reproduce is it by switching back and forth between source and visual mode—sometimes once was enough, sometimes twice.

mikebessuille commented 2 years ago

@dragonstyle not sure who best could look at this? Is this a visual markdown problem?

dragonstyle commented 2 years ago

Let me take a look and triage - might need to have JJ take a look.

jjallaire commented 2 years ago

Addressed with https://github.com/rstudio/rstudio/commit/aa08d43d76294920c98052538bb7d4b62dfcde28

ronblum commented 2 years ago

Verified using several of the examples in

ronblum commented 2 years ago

Reopening to consider adding an automated test. @rich-rstudio do you think this warrants it, measuring the effort? It would require mimicking typing the entry, not just the original text. Note: If it does, then we should consider including #9938, which can be the same test.

[Rich—I assigned it to you to assess, not necessarily to create.]

melissa-barca commented 2 years ago

@rich-rstudio I can add automation for this one so I unassigned you.

astayleraz commented 2 years ago

Closing issue as automation test added has been merged and the fix has been verified manually as well.

pedrohbraga commented 2 years ago

Thank you for this!!! 🎉 Excited to see next year's version update!! Happy holidays!!

ronblum commented 2 years ago

Note: Verified the backport in Ghost 2021.09.2+382.

@pedrohbraga While Prairie Trillium (2022.02.0) is still in progress, we've also released Ghost Orchid update 2021.09.2+382 which contains this fix. If you update to it, please let us know if you continue to run into any issues. And, I hope your holidays went well!

wuttkea commented 2 years ago

I continue to have this problem with basically all my scientific papers in one way or the other. Particularly when I use [] somewhere in the text or some other formatting or special cases / punctuation close to the citations such as here where undesired backslashes are added:

image image

As I did in other papers, I will now remove all [] from the text and replace them with () although my preference would be to keep them in there.

mikebessuille commented 2 years ago

@wuttkea which version are you using? We've backported this to the latest Ghost Orchid (2021.09.2) patch, and it's fixed in Prairie Trillium which will be releasing any day now... (2022.02.0) and whose previews are available. Are you still having problems with the latest preview?

wuttkea commented 2 years ago

Fantastic. I have updated to the preview version and, indeed, have not been able to produce this problem. Great! Thank you.

ronblum commented 2 years ago

@wuttkea I'm glad that this is now working for you, and thank you for letting us know that this now working! And for you, @pedrohbraga, and others—we just released an official, stable release of Prairie Trillium 2022.02.0+443, available for download at https://www.rstudio.com/products/rstudio/download/#download . Enjoy! 🙂

tseidl commented 2 years ago

I continue to experience the problem even with the newest version although switching to visual and back no longer adds backlashes before curly brackets - now it's only @ symbols. For example, [@TheEconomist.2020] becomes [\@TheEconomist.2020]

ronblum commented 2 years ago

@tseidl Thank you! This works in Ghost Orchid 2021.09.2+382, and I can reproduce it in Prairie Trillium 2022.02.0+4443, so reopening this is a regression.

n8craig commented 2 years ago

I am experiencing this issue with 2022.02.0 Build 443 in Win 10 when working with .qmd files.

Steps to reproduce:

test.qmd

---
title: "Untitled"
format: html
editor: visual
bibliography: 
  references.bib
---

This is some text [@abelsen1993, 93].

references.bib


@article{abelsen1993,
    title = {Schopenhauer and Buddhism},
    author = {Abelsen, Peter},
    year = {1993},
    date = {1993},
    journal = {Philosophy East and West},
    pages = {255--278},
    volume = {43},
    number = {2},
    doi = {10.2307/1399616},
    url = {http://www.jstor.org/stable/1399616}
}

From the Source editor, this renders properly, producing:

This is some text (Abelsen 1993, 93)

Switching to the visual editor and switching back results in a new test.qmd. Now the file looks like:

---
title: "Untitled"
format: html
editor: visual
bibliography: 
  references.bib
---

This is some text \[@abelsen1993, 93\].

Upon render the new output now looks like

This is some text [Abelsen (1993), 93].

Dijkie85 commented 2 years ago

I'm also experiencing the issue @n8craig described when editing .qmd files within a newly created Quarto project. I can add that it does not happen for me when switching back from visual editor to source on regular .Rmd files (in a RMarkdown project).

EDIT: I'm using RStudio 2022.02.1+461 "Prairie Trillium" Release (8aaa5d470dd82d615130dbf663ace5c7992d48e3, 2022-03-17) for Ubuntu Bionic Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.82 Safari/537.36

jjallaire commented 2 years ago

Thanks for everyone's patience and diligence in following up on this issue. I'm hopeful that we will finally put this issue to rest with these commits:

And this PR:

In terms of the nature of the fixes, there are two general causes of these problems:

(1) Our pattern for detecting citation IDs has recently become too restrictive;

(2) In .qmd files backslashes can be escaped (whereas in Rmd they can't because the +tex_math_single_backslash is part of the Rmd markdown spec). Our logic for determining when something is a valid citation was also overly restrictive (resulting in the backslashes for .qmd files only).

Note that anyone experiencing this second issue in the Visual Editor with .qmd files can work around it by adding the following to their document YAML options:

---
from: markdown+tex_math_single_backslash
---

This basically puts the editor in the same posture towards backslashes as it already has with Rmd files. Hopefully we can merge these fixes into a patch release that will come out fairly soon.

jjallaire commented 2 years ago

@ronblum All of these fixes are now on main and PT and ready for testing/verification.

ronblum commented 2 years ago

Verified the fix for the citations in the .qmd file, using test.qmd and references.bib from above.

Recommendation for automated test:

  1. Add the test for this condition to the existing automated tests of the R Markdown file (.Rmd).
  2. Repeat all of the tests with a Quarto file (.qmd).
ronblum commented 2 years ago

Reopening because it looks like there's another version of this issue, which came up in #11198 and I can reproduce in PT and SW. Let me know if I should open a separate issue instead.


[Copied from @robinlovelace's note in #11198.]

travis-main.zip

No problem. See attached another maybe reproducible example. Every time I open this up in visual mode all the citations get changed from something like this:

[@wood_visualizing_2011; @vonschmidt_webbased_2019; @morgan2020; @beecham2022]

to this:

 [\@wood_visualizing_2011; \@vonschmidt_webbased_2019; \@morgan2020; \@beecham2022].

which is frustrating to say the least...

ronblum commented 2 years ago

Closed in favor of opening #11279.