Windows and first use of devtools

[jennybc]

STAT 545 students: I'm not expecting you to answer this, rather am hoping to attract answers from the community.

I'm seeking advice on what to tell STAT 545 windows users prior to next week's lessons on package development. Last year, we had some regrets about our instructions, but we never decided what would have been best.

We will be using devtools. If that offends you, please just ignore this thread. That's not up for debate.

Should they install Rtools?

YES: technically of course they should! Last I checked devtools slaps you on the wrist if you have not done this? And you won't be able to install packages requiring compilation from GitHub with install_github() until you do this.

NO: I'm pretty sure we can get by w/o this since our toy packages will not require compilation.

MORE NO: Rtools seems to install fairly old versions of some Unix stuff. We've already had them install Git for Windows, aka Git Bash. Soon many of them might even install msysgit OMG no maybe that's Git for Windows SDK now (?), in order to get current Make and other things you might want to call from Make. I hate all this re-installation and potential for confusion.

Should we be looking at...

• chocolatey which seems to be apt-get or homebrew but for Windows
• installr: "Using R to Install Stuff (Such As: R, Rtools, RStudio, Git, and More!)"

@jeroenooms @STAT545-UBC/2014_ta @STAT545-UBC/2015_ta

[soodoku]

afaik, RTools essential if they need to install pkgs with src

[mmparker]

Hmm... I've used Git Bash and Rtools on Windows XP, 7, 8, 8.1, and 10 and I don't think I've ever had a conflict between Git's Unix stuff and Rtools. Better to have a bit more installation misery than mysterious-compilation-error misery. Maybe schedule a happy hour when students can get together outside of class and help each other troubleshoot?

[gmbecker]

I'd strongly advise against encouraging them to install things that are going to conflict with Rtools if they are ever going to want to install packages from source.

Rtools has old versions of things, but newer versions often cause weird, impossible-for-students-to-debug errors at various places in the R build chain. In particular, within the context of R, they should definitely be using the version of make in Rtools, not a newer one.

That said, I thought git bash was it's own separate shell which wouldn't be used during package building, so that should be fine, I think. Just tell them they can't build their packages there, they have to use the normal commandline or RStudio or other IDE.

[npjc]

Just being forward looking here. Whatever solution is picked should have a way to store or somehow log specifics of issues we encounter ("ahem... banging on the we need more data drum").

[jennybc]

If you look at this issues in this repo from November 2014, you'll see a lot of our problems. They come up once we start using Make. And people get different results and errors when running make through RStudio vs directly in bash.

But you're right @npjc: we need to carefully record our problems, so we can either answer this ourselves or clearly articulate how "just use Rtools!" is problematic. I feel it assumes that one's entire unix life is centered around R, which is presumptuous. But I'm also not completely sure that characterization is accurate.

[kevinushey]

What about instructing students to install VirtualBox and just use Ubuntu? (if only...)

One of the big headaches is the setting of PATH with Rtools (to ensure the compiler + required utilities are on the PATH). RStudio and devtools both do their best to ensure Rtools is on the PATH if it's installed, but this doesn't always work (mainly if installed in a non-standard directory).

My overarching suggestion for Windows users:

1. Install git for windows in the default directory; ie, C:\Program Files\Git;
2. Install RTools in the default directory; ie, C:\Rtools,
3. Try to ensure that the Rtools directories come first on the PATH, (C:\Rtools\bin and C:\Rtools\gcc-4.6.3\bin);
4. Ensure that the Git binary directory is not on the PATH. Otherwise the utilities included can (and often will) conflict with the utilities desired by Rtools.

I also vaguely recall that there are some really subtle things source packages can do that will implicitly require Rtools (e.g. creating a .Rbuildignore), so it really is best to ensure it's installed.

[petermeissner]

I never had problems with simply installing

• R
• Rtools
• git
• Rstudio

... and using the default directories. I never used make on Windows though.

Why would you use make on Windowes / for R developement - I am curious whats the use case?

[smach]

Agree with @petermeissner I've got R, Rtools, git and RStudio on my home Windows machine. I'm not a power developer, but for basic package creation with devtools and roxygen2 it's been working fine for me.

[jennybc]

Thanks to all who have weighed in!

@kevinushey You words are especially helpful. At least now I know we are not crazy that having two separate unix tool chains has the potential to create problems. I'm puzzled by this piece of advice:

Ensure that the Git binary directory is not on the PATH

where you're referring to the Git binary from Git for Windows, ideally found at C:\Program Files\Git. Won't this mean that their day in, day out use of Git would NOT be the current version from Git for Windows? Big picture, wouldn't it mean that RStudio, for example, would not necessarily find Git for Windows?

@petermeissner We use make to automate data analysis pipelines. So to run a series of scripts or commands, not necessarily all R code or scripts, in a coordinated fashion. In our course, we don't use make for R package development per se.

[daattali]

I just tried installing RTools on Windows 7 and it was fairly straight forward, and it also results in using make easier because RTools ships with make.exe and some other unix commands that are often used in makefiles (rm echo cp etc). RTools knows how to add itself to the user's PATH, and it does it at the beginning of the PATH so that it will definitely be reached before Git (as @kevinushey suggested).

This is my suggested workflow that worked for me, hopefully it'll just out out-of-the-box for students:

• install.packages("devtools")
• run devtools::find_rtools()
• Windows users should see a message telling you that you need to install Rtools. Go to the RTools website and install the latest version (currently v3.3)

• important During Rtools installation when it asks for additional tasks, make sure to select the box for "Edit the system PATH". It looks like this

  

  The option was NOT checked for me by default, I had to select it myself. Then continue with the installation and accept all the default settings.

• When RTools installation completes, run devtools::find_rtools() again, this time it should return TRUE
• Restart RStudio so that the new PATH variable will go into effect

Now users should be able to build packages and use makefiles.

[gmbecker]

@jennybc I hear what you're saying about taking an R-centric view of your unix life (though if we're talking about pretending Windows is unix we're already in somewhat strange territory), but the bottom line is that R's package build system doesn't support non-Rtools ways of simulating posix on Windows, and barring someone very technically proficient donating a bunch of time to tackle an honestly not very interesting problem, that isn't likely to change. Maybe the R Consortium will throw money at this problem to fix it, but I haven't heard of any current plans in that arena.

Is there a reason that the make that Rtools provides isn't sufficient for the non-R things you have students doing?

[jennybc]

@gmbecker

We didn't have trouble with stuff being too old per se. Rtools-using students would get different behaviour than what they see on my machine. Makefiles I or a Ubuntu-using TA write don't necessarily work as advertised for Rtools users. There were ugly surprises with PATH, which shell was being used from Make, depending on whether Make was being called directly or via Studio, trickiness re: which other unix commands could be used in make rules, which direction the slashes needed to go in paths, blah, blah blah.

Huge doses of classic command line bullshittery that ate valuable time and psychic energy.

Is there a manifest for exactly what Rtools installs and at what version? I can't find one in an obvious place. The screenshots people are sharing here and on Twitter are a glimpse into a secret Rtools world!

What I've learned here so far:

• Last year we did things in a different order. We did make and automation before package building + Rtools. This year it's other way `round and I think this will work better. Less gratuitous re-installation.
• This discussion and twitter has already given me much better info about optimal install location for Rtools and PATH settings. I am optimistic.

[jennybc]

https://github.com/rwinlib/r-base

[jennybc]

https://stat.ethz.ch/pipermail/r-sig-windows/

[jennybc]

https://cran.r-project.org/doc/manuals/R-admin.html#The-Windows-toolset

https://cran.r-project.org/doc/manuals/R-admin.html#The-command-line-tools

I gather we can count on these being provided by Rtools:

basename, cat, cmp, comm, cp, cut, date, diff, du, echo, expr, gzip, ls, make, makeinfo, mkdir, mv, rm, rsync, sed, sh, sort, tar, texindex, touch and uniq ... zip and unzip

[kevinushey]

@jennybc, the main reason why you probably don't want Git on the PATH is because it comes with its own set of command line utilities which can mask / interfere with the tools R expects to find from Rtools.

This might happen e.g. when you call install.packages(..., type = "source") -- I believe R actually launches a child R process to handle the package build + install, and if that child process ends up finding the wrong command line tools, bad things happen.

RStudio actually does some gymnastics to find a Git installation without relying on the PATH, e.g..

• Looking in the system registry,
• Looking for a shortcut in the start menu,

so, strictly speaking, it shouldn't be necessary for Git to be on the PATH for Windows for an RStudio user. That said, some changes were required to robustly look up the newest version of Git for Windows, so users without an up to date version of RStudio might have headaches.

And, I think if you launch Git Bash it also ensures that its own command line utilities will come first on the PATH, so you don't need to worry about anything there.

Net result: headaches are minimized if Rtools comes first on the PATH, and the git binary directory is not on the PATH.

(aside: I find Rapid Environment Editor a much better interface to Windows environment variables than its own crappy built-in utility)

[daattali]

I can confirm that Git Bash indeed finds Git's utils first regardless of where Git is on the PATH. But from personal experience, RStudio did not automatically find the git executable when I went to Global Options --> Git/SVN. At least not the last time I tried, maybe in the past few months it changed.

---

http://deanattali.com

On 9 November 2015 at 15:49, Kevin Ushey notifications@github.com wrote:

@jennybc https://github.com/jennybc, the main reason why you probably don't want Git on the PATH is because it comes with its own set of command line utilities which can mask / interfere with the tools R expects to find from Rtools.

This might happen e.g. when you call install.packages(..., type = "source") -- I believe R actually launches a child R process to handle the package build + install, and if that child process ends up finding the wrong command line tools, bad things happen.

RStudio actually does some gymnastics to find a Git installation without relying on the PATH, e.g..

• Looking in the system registry,
• Looking for a shortcut in the start menu,

so, strictly speaking, it shouldn't be necessary for Git to be on the PATH for Windows for an RStudio user.

And, I think if you launch Git Bash it also ensures that its own command line utilities will come first on the PATH, so you don't need to worry about anything there.

— Reply to this email directly or view it on GitHub https://github.com/STAT545-UBC/Discussion/issues/230#issuecomment-155236031 .

[jennybc]

Happy to report that our early use of make seems to have gone well. This year's order (R packages and therefore, Rtools .... then make) is vastly superior to last year's order. There were only 2 people with very minor issues when we tested the use of make from RStudio and from the shell. Thursday will be a bigger test but I'm still so relieved. Thanks for all of the discussion.

[daattali]

The 2 people with minor issues could very well have been related to #242 , which means that really nobody had any issues.

I'm not 100% sure students will be able to use unix tools (cd, echo, rm, ...) from within a Makefile, that we'll have to wait and see, but yes it's nice that at least a basic Makefile that calls Rscript works flawlessly