Add EPEL with Redhat Developer Toolkit compilers

[jeroen]

This is a special variant of CentOS6 which enables the "redhat developer toolkit". This includes a copy of gcc 5 which is binary compatible with the gcc 4.4 that is included with centos 6 itself. CentOS users need this to compile C++11 code.

In order to use these compilers, the user has to source a script which set the PATH and LD_LIBRARY_PATH and other stuff to this new set of compilers. I am not sure what is the best way to ensure this script gets executed before R in rhub. I am not using:

ENV RBINARY source /opt/rh/devtoolset-4/enable && R

[gaborcsardi]

Let me think a minute about RBINARY.

Btw. could you please also add a platform here? https://github.com/r-hub/rhub-frontend/blob/master/public/data/platforms.json

Thanks!

[jeroen]

I could also change the docker image to create a script which executes the enable script and then calls R, and then invoke this wrapper script in RBINARY. But if the current solution works that would be more transparent I think.

[jeroen]

re: frontend @ https://github.com/r-hub/rhub-frontend/pull/10/files

[gaborcsardi]

Thanks!

TBH I don't even like the RD solution, because if you want to start R from within R like many packages do, or even just install.packages does, it will start the wrong executable.

[gaborcsardi]

So I think it would be best just to have an R executable. I will roll our own image for this at some point....

[gaborcsardi]

Although it is also handy to have regular R around.... anyway, I'll leave it as it is for now.

[jeroen]

That's not entirely true. Packages that start an R process should use R.home() to find the current binary. The RD symlink is only a convenience shortcut to start the UBSAN R build from the shell.

I'm quite sure that install.packages won't have any problems with multiple R versions installed on a system. In fact I believe that Kurt uses a single Debian machine to run CRAN checks on the various r-devel-linux-x86_64-debian-gcc variants.

[gaborcsardi]

The problem is that the executable has a different name. R.home() only gives a directory:

❯ R.home("bin")
[1] "/Library/Frameworks/R.framework/Resources/bin"

How do you find out how the executable is called within that?

[gaborcsardi]

Oh, I see, RD is just a symlink.... good then.

[jeroen]

Yes they are entirely different builds of R in different directories. The RD symlink is only a convenience shortcut to start the "other" R on the system.

[gaborcsardi]

But then we can also just set the PATH to include the sanitizing version of R in it.....

[jeroen]

Wait we are discussing the UBSAN image in a thread of the EPEL image now. Are we still talking about the same thing?

Yes you could use PATH as well, but the rocker image already renames R and RScript to prevent the masking of executables so we would have to rename it back, or roll our own image...

[gaborcsardi]

Wait we are discussing the UBSAN image in a thread of the EPEL image now. Are we still talking about the same thing?

Yeah, both, sorry. I am just thinking about whether we need RBINARY at all.

Btw. if you set ENV RBINARY source /opt/rh/devtoolset-4/enable && R, then how do you run this without starting a subshell? I am sure there is a way, but this is surprisingly complicated....

[gaborcsardi]

Btw. if you set ENV RBINARY source /opt/rh/devtoolset-4/enable && R, then how do you run this without starting a subshell? I am sure there is a way, but this is surprisingly complicated....

OK, looks like eval will be OK.

[jeroen]

Hmm i assumed you were just calling $RBINARY CMD check or so, so it would naturally become:

source /opt/rh/devtoolset-4/enable && R CMD check

But maybe creating a separate script is more elegant.

[gaborcsardi]

Hmm i assumed you were just calling $RBINARY CMD check or so, so it would naturally become:

Yes, but that does not work with &&. It is fragile in general...

[gaborcsardi]

Also, I need to run R within xvfb-run, so this will not work, because the source should be run by the shell, and R by xvfb-run, obviously....

[gaborcsardi]

Also, if R starts up another R process, then the source part is not running, so the config will be wrong.

So how about putting it in some bash profile? Maybe the profile of the docker user?

[jeroen]

I don't think bash runs at all if you just execute R directly from docker.

[gaborcsardi]

I start bash. I need to do a bunch of stuff before the check, e.g. install dependent packages. I specifically run bash, so a .bash_profile / .bashrc is good I think.

And then RBINARY can contain the name of the executable, but not a generic bash command....

[jeroen]

OK i updated the pr. let me build this locally to see if this works.

[jeroen]

Updated again, I think it should work now. Also changed the order of the commands so that it will automatically share the intermediate container with the other epel image.

[jeroen]

OK i removed it.

[gaborcsardi]

Thanks!

[gaborcsardi]

Seems to work fine, thanks again!

[jeroen]

A good package to test is feather because it relies heavily on C++11 and does not build with the default compiler from CentOS6. So it really needs the RDT compilers.

[jeroen]

Hmm it doesn't seem to be working :( https://builder.r-hub.io/status/feather_0.3.0.tar.gz-dc14ec13d2448ba781abbb2f03135eb2

[gaborcsardi]

What's wrong with it? Is it using the right compilers?

On 5 Sep 2016 14:49, "Jeroen Ooms" notifications@github.com wrote:

Hmm it doesn't seem to be working :( https://builder.r-hub.io/ status/feather_0.3.0.tar.gz-dc14ec13d2448ba781abbb2f03135eb2

— You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub https://github.com/r-hub/rhub-linux-builders/pull/9#issuecomment-244751484, or mute the thread https://github.com/notifications/unsubscribe-auth/AAoTQKNQFNK_H-M7EnTXgRVDTLj6o7B9ks5qnB4DgaJpZM4J03ZX .

[jeroen]

I don't know, I can't see, but I suspect not. You should add a line like this to the output:

CC=`${RBINARY} CMD config CC`
${CC} --version

[gaborcsardi]

It uses g++ 4.4.7 which is not good I suppose.

But if R was not built with the RDT compilers, then it will not use them I guess. Don't you need to build R with RDT?

On Mon, Sep 5, 2016 at 3:16 PM, Jeroen Ooms notifications@github.com wrote:

I don't know, I can't see, but I suspect not. You should add a line like this to the output:

CC=R CMD config CC${CC} --version``

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/r-hub/rhub-linux-builders/pull/9#issuecomment-244757047, or mute the thread https://github.com/notifications/unsubscribe-auth/AAoTQF7uhoa8aHRudWnpBOSxKqYffHEZks5qnCQqgaJpZM4J03ZX .

[jeroen]

It will use them, the compilers are designed to be compatible. It just needs the correct g++ in the PATH which is what the enable script does. It works locally. Are you super sure you are starting a new bash so that the rc scripts run before calling R?

[gaborcsardi]

This is what I run: https://github.com/r-hub/rhub-backend/blob/master/templates/jenkins.sh#L47-L87

AFAIR R remembers the path of the compilers it was built from, and uses them for installing packages.

[jeroen]

It does not remember the path, only the name of the compiler, in this case g++. You can test this by running:

R CMD config CXX

From there it finds it from the PATH. The enable script puts g++ from RDT in front of the path, so it gets preference.

[jeroen]

Maybe you can also print $PATH in addition to CC --version to the console? That seems useful anyway.

[gaborcsardi]

Yeah, there should be a section that prints all config.

On Mon, Sep 5, 2016 at 4:18 PM, Jeroen Ooms notifications@github.com wrote:

Maybe you can also print $PATH in addition to CC --version to the console? That seems useful anyway.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/r-hub/rhub-linux-builders/pull/9#issuecomment-244770046, or mute the thread https://github.com/notifications/unsubscribe-auth/AAoTQHKtAuc4rtac2KCXVd7shTIqn_Fnks5qnDLhgaJpZM4J03ZX .

[gaborcsardi]

it's messed up, I have both a centos-epel-rdt and centos6-epel-rdt image. One of them is wrong....

On Mon, Sep 5, 2016 at 4:29 PM, Gábor Csárdi csardi.gabor@gmail.com wrote:

Yeah, there should be a section that prints all config.

On Mon, Sep 5, 2016 at 4:18 PM, Jeroen Ooms notifications@github.com wrote:

Maybe you can also print $PATH in addition to CC --version to the console? That seems useful anyway.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/r-hub/rhub-linux-builders/pull/9#issuecomment-244770046, or mute the thread https://github.com/notifications/unsubscribe-auth/AAoTQHKtAuc4rtac2KCXVd7shTIqn_Fnks5qnDLhgaJpZM4J03ZX .

[gaborcsardi]

OK, I got it. It is not a login shell, and /etc/bashrc is not used. I can make it a login shell, or we can put it in a different bash rc file.

[jeroen]

That's why I don't like relying in bash rc files. It becomes ambiguous if and when things get executed. It would be much better to explicitly trigger this script somewhere. That way it would also work if we invoke R directly from docker, without going via bash.

[jeroen]

Anyway for now maybe you can just modify it to echo that line to wherever it does get executed.

[gaborcsardi]

I made it to a login shell, and it is fine now.

I agree that bash rc files are messed up, but I am not sure what is a good alternative here.

[gaborcsardi]

https://builder.r-hub.io/status/parsedate_1.1.2.tar.gz-de3aa8c3cf249b237147533bfd02fd7c#L246

[jeroen]

OK retrying feather: https://builder.r-hub.io/status/feather_0.3.0.tar.gz-417f1b1e680e2cf6f34b3d0927902feb

[gaborcsardi]

Yeah, it is good now. :)