Can not install doAzureParallel on a node

grayskripko commented 7 years ago

When I'm trying to install doAzureParallel on a remote node it faces 2 problems

Infinite execution

Job Preparation Status: Package(s) being installed.......................

stderr.txt contains the reason related to 'xml2' package:


Downloading GitHub repo Azure/doAzureParallel@master
from URL https://api.github.com/repos/Azure/doAzureParallel/zipball/master
Installing doAzureParallel
Using GitHub PAT from envvar GITHUB_PAT
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed

0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   98k  100   98k    0     0  76603      0  0:00:01  0:00:01 --:--:-- 76805
Installing xml2
'/usr/lib64/microsoft-r/3.3/lib64/R/bin/R' --no-site-file --no-environ  \
--no-save --no-restore --quiet CMD INSTALL  \
'/tmp/RtmpxviY7O/devtoolsde039cababe/xml2'  \
--library='/usr/lib64/microsoft-r/3.3/lib64/R/library' --install-tests

installing source package 'xml2' ... package 'xml2' successfully unpacked and MD5 sums checked libs installing to /usr/lib64/microsoft-r/3.3/lib64/R/library/xml2/libs R inst tests preparing package for lazy loading ** help * installing help indices building package indices installing vignettes testing if installed package can be loaded ERROR: loading failed
removing '/usr/lib64/microsoft-r/3.3/lib64/R/library/xml2' Error: Command failed (1) Execution halted
```
Looks like a problem with rights
```

paselem commented 7 years ago

That is interesting. Is there any additional info in the stdout.txt file?

Also, can you give us the list (or code) of packages you're installing so we can try to repro?

grayskripko commented 7 years ago

It is not, actually. The full stdout.txt

Collecting blobxfer
  Downloading blobxfer-0.12.1-py2.py3-none-any.whl (41kB)
Installing collected packages: blobxfer
Successfully installed blobxfer-0.12.1
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c AAA_check_cpp11.cpp -o AAA_check_cpp11.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c Data.cpp -o Data.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c DataChar.cpp -o DataChar.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c DataDouble.cpp -o DataDouble.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c DataFloat.cpp -o DataFloat.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c Forest.cpp -o Forest.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c ForestClassification.cpp -o ForestClassification.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c ForestProbability.cpp -o ForestProbability.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c ForestRegression.cpp -o ForestRegression.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c ForestSurvival.cpp -o ForestSurvival.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c RcppExports.cpp -o RcppExports.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c Tree.cpp -o Tree.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c TreeClassification.cpp -o TreeClassification.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c TreeProbability.cpp -o TreeProbability.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c TreeRegression.cpp -o TreeRegression.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c TreeSurvival.cpp -o TreeSurvival.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c rangerCpp.cpp -o rangerCpp.o
gcc -std=c++11  -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -DR_BUILD -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include"     -fpic -c utility.cpp -o utility.o
gcc -std=c++11 -shared -L/usr/lib64/microsoft-r/3.3/lib64/R/lib -o ranger.so AAA_check_cpp11.o Data.o DataChar.o DataDouble.o DataFloat.o Forest.o ForestClassification.o ForestProbability.o ForestRegression.o ForestSurvival.o RcppExports.o Tree.o TreeClassification.o TreeProbability.o TreeRegression.o TreeSurvival.o rangerCpp.o utility.o -L/usr/lib64/microsoft-r/3.3/lib64/R/lib -lR
gcc -std=gnu99 -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG  -DU_STATIC_IMPLEMENTATION    -fpic  -DU_STATIC_IMPLEMENTATION -O2 -g  -c dm.c -o dm.o
gcc -std=gnu99 -shared -L/usr/lib64/microsoft-r/3.3/lib64/R/lib -o kknn.so dm.o -L/usr/lib64/microsoft-r/3.3/lib64/R/lib -lR
Found pkg-config cflags and libs!
Using PKG_CFLAGS=-I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include
Using PKG_LIBS=-L/anaconda/envs/py35/lib -lxml2 -L/anaconda/envs/py35/lib -lz -L/anaconda/envs/py35/lib -liconv -lm -ldl
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c RcppExports.cpp -o RcppExports.o
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c connection.cpp -o connection.o
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c xml2_doc.cpp -o xml2_doc.o
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c xml2_init.cpp -o xml2_init.o
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c xml2_namespace.cpp -o xml2_namespace.o
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c xml2_node.cpp -o xml2_node.o
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c xml2_url.cpp -o xml2_url.o
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c xml2_xpath.cpp -o xml2_xpath.o
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c xml_push.cpp -o xml_push.o
g++ -shared -L/usr/lib64/microsoft-r/3.3/lib64/R/lib -o xml2.so RcppExports.o connection.o xml2_doc.o xml2_init.o xml2_namespace.o xml2_node.o xml2_url.o xml2_xpath.o xml_push.o -L/anaconda/envs/py35/lib -lxml2 -L/anaconda/envs/py35/lib -lz -L/anaconda/envs/py35/lib -liconv -lm -ldl -L/usr/lib64/microsoft-r/3.3/lib64/R/lib -lR
Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared object '/usr/lib64/microsoft-r/3.3/lib64/R/library/xml2/libs/xml2.so':
  libiconv.so.2: cannot open shared object file: No such file or directory
Error: loading failed
Execution halted

The full stderr.txt

The directory '/mnt/resource/batch/tasks/startup/wd/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/mnt/resource/batch/tasks/startup/wd/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 98606  100 98606    0     0  86108      0  0:00:01  0:00:01 --:--:-- 86420
* installing *source* package 'ranger' ...
** package 'ranger' successfully unpacked and MD5 sums checked
** libs
installing to /usr/lib64/microsoft-r/3.3/lib64/R/library/ranger/libs
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (ranger)

The downloaded source packages are in
    '/tmp/RtmpNOOhoy/downloaded_packages'
Updating HTML index of packages in '.Library'
Making 'packages.html' ... done
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  379k  100  379k    0     0   257k      0  0:00:01  0:00:01 --:--:--  258k
* installing *source* package 'kknn' ...
** package 'kknn' successfully unpacked and MD5 sums checked
** libs
installing to /usr/lib64/microsoft-r/3.3/lib64/R/library/kknn/libs
** R
** data
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (kknn)

The downloaded source packages are in
    '/tmp/RtmpsbbGYS/downloaded_packages'
Updating HTML index of packages in '.Library'
Making 'packages.html' ... done
Using GitHub PAT from envvar GITHUB_PAT
Downloading GitHub repo cargomoose/stackatto@master
from URL https://api.github.com/repos/cargomoose/stackatto/zipball/master
Installing stackatto
Using GitHub PAT from envvar GITHUB_PAT
Downloading GitHub repo Azure/doAzureParallel@master
from URL https://api.github.com/repos/Azure/doAzureParallel/zipball/master
Installing doAzureParallel
Using GitHub PAT from envvar GITHUB_PAT
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 16   98k   16 16117    0     0  15788      0  0:00:06  0:00:01  0:00:05 16429
100   98k  100   98k    0     0  75653      0  0:00:01  0:00:01 --:--:-- 78059
Installing xml2
'/usr/lib64/microsoft-r/3.3/lib64/R/bin/R' --no-site-file --no-environ  \
  --no-save --no-restore --quiet CMD INSTALL  \
  '/tmp/Rtmp49Flb7/devtoolsdd7bcd68a5/xml2'  \
  --library='/usr/lib64/microsoft-r/3.3/lib64/R/library' --install-tests 

* installing *source* package 'xml2' ...
** package 'xml2' successfully unpacked and MD5 sums checked
** libs
installing to /usr/lib64/microsoft-r/3.3/lib64/R/library/xml2/libs
** R
** inst
** tests
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
ERROR: loading failed
* removing '/usr/lib64/microsoft-r/3.3/lib64/R/library/xml2'
Error: Command failed (1)
Execution halted

paselem commented 7 years ago

@grayskripko - are you installing your package at the cluster level or inside the foreach loop .packages setting?

grayskripko commented 7 years ago

  "rPackages": {
    "cran": ["ranger", "kknn"],
    "github": ["cargomoose/stackatto"]

The last one is a private repo. Its DESCRIPTION file contains

Imports: caret,
    doParallel,
    doAzureParallel
Remotes: Azure/doAzureParallel

grayskripko commented 7 years ago

@paselem at the cluster level. The only thing my foreach loop contains is .packages = c('stackatto'). Furthermore, the problem does not finish makeCluster() function

grayskripko commented 7 years ago

When I try running a similar command using node ssh access

[gray@f0067192...0002 ~]$ /bin/bash -c "set -e; set -o pipefail; export PATH=/anaconda/envs/py35/
bin:$PATH; Rscript -e 'args <- commandArgs(TRUE)' -e 'options(warn=2)' -e 'install.packages(args[1])' ranger"
Error in install.packages(args[1]) :
  (converted from warning) 'lib = "/usr/lib64/microsoft-r/3.3/lib64/R/library"' is not writable
Execution halted

I'm not sure that it is relevant but it may be

grayskripko commented 7 years ago

@paselem I can not even install

"rPackages": {
    "cran": ["xml2"]

Start task command line

/bin/bash -c "set -e; set -o pipefail; export PATH=/anaconda/envs/py35/bin:$PATH; env PATH=$PATH pip install --no-dependencies blobxfer; Rscript -e 'args <- commandArgs(TRUE)' -e 'options(warn=2)' -e 'install.packages(args[1])' xml2; wait"

startup/stdout.txt

Collecting blobxfer
  Downloading blobxfer-0.12.1-py2.py3-none-any.whl (41kB)
Installing collected packages: blobxfer
Successfully installed blobxfer-0.12.1
Found pkg-config cflags and libs!
Using PKG_CFLAGS=-I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include
Using PKG_LIBS=-L/anaconda/envs/py35/lib -lxml2 -L/anaconda/envs/py35/lib -lz -L/anaconda/envs/py35/lib -liconv -lm -ldl
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c RcppExports.cpp -o RcppExports.o
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c connection.cpp -o connection.o
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c xml2_doc.cpp -o xml2_doc.o
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c xml2_init.cpp -o xml2_init.o
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c xml2_namespace.cpp -o xml2_namespace.o
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c xml2_node.cpp -o xml2_node.o
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c xml2_url.cpp -o xml2_url.o
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c xml2_xpath.cpp -o xml2_xpath.o
g++ -I/usr/lib64/microsoft-r/3.3/lib64/R/include -DNDEBUG -I/anaconda/envs/py35/include/libxml2 -I/anaconda/envs/py35/include -DU_STATIC_IMPLEMENTATION -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/Rcpp/include" -I"/usr/lib64/microsoft-r/3.3/lib64/R/library/BH/include"   -fpic  -DU_STATIC_IMPLEMENTATIN -O2 -g  -c xml_push.cpp -o xml_push.o
g++ -shared -L/usr/lib64/microsoft-r/3.3/lib64/R/lib -o xml2.so RcppExports.o connection.o xml2_doc.o xml2_init.o xml2_namespace.o xml2_node.o xml2_url.o xml2_xpath.o xml_push.o -L/anaconda/envs/py35/lib -lxml2 -L/anaconda/envs/py35/lib -lz -L/anaconda/envs/py35/lib -liconv -lm -ldl -L/usr/lib64/microsoft-r/3.3/lib64/R/lib -lR
Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared object '/usr/lib64/microsoft-r/3.3/lib64/R/library/xml2/libs/xml2.so':
  libiconv.so.2: cannot open shared object file: No such file or directory
Error: loading failed
Execution halted

startup/stderr.txt

The directory '/mnt/resource/batch/tasks/startup/wd/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/mnt/resource/batch/tasks/startup/wd/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   98k  100   98k    0     0  77281      0  0:00:01  0:00:01 --:--:-- 77457
* installing *source* package 'xml2' ...
** package 'xml2' successfully unpacked and MD5 sums checked
** libs
installing to /usr/lib64/microsoft-r/3.3/lib64/R/library/xml2/libs
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
ERROR: loading failed
* removing '/usr/lib64/microsoft-r/3.3/lib64/R/library/xml2'
Error in install.packages(args[1]) : 
  (converted from warning) installation of package 'xml2' had non-zero exit status
Execution halted

brnleehng commented 7 years ago

@grayskripko @paselem

I think the issue is due to anaconda installation. xml2 package depends on libxml2 but we already installed it via anaconda. However, this libxml2 is incompatible with the R libxml2. http://www.perfectlyrandom.org/2015/12/13/install-xml2-r-package-on-macos/

The core functionality of the doAzureParallel doesn't depend on the xml2 package. The xml2 package is used for listing storage containers and files, extracting XML responses from Azure Storage HTTP requests. So I think we should put it as a Suggests package in the description (http://r-pkgs.had.co.nz/description.html).

Thoughts?

grayskripko commented 7 years ago

What if user wants use some package that requires xml2?

brnleehng commented 7 years ago

Yeah, I'll be seeing if we can remove our dependency with anaconda.

Azure / doAzureParallel

Can not install doAzureParallel on a node #114