Git repo failing when hashing via http://git2nixsha.dev

[eliocamp]

I'm getting a 500 error for a github package.

library(rix)

metR <- list(package_name = "metR", 
             repo_url = "https://github.com/eliocamp/metR", 
             branch_name = "master", 
             commit = "1dd5d391d5da6a80fde03301671aea5582643914")

rix(r_ver = "4.3.2",
    git_pkgs = metR,
    ide = "rstudio",
    project_path = ".",
    overwrite = TRUE,
    print = FALSE)
#> Error: Request `curl::curl_fetch_memory(url = 'http://git2nixsha.dev:1506/hash?repo_url=https://github.com/eliocamp/metR&branchName=master&commit=1dd5d391d5da6a80fde03301671aea5582643914')` failed:
#>  The requested URL returned error: 500
#> If it's a Github repo, check the url, branch name and commit.
#> Are these correct? If it's an archived CRAN package, check the name
#> of the package and the version number.

The information of repo, branch and commit sha is correct as far as I can tell: https://github.com/eliocamp/metR/commit/1dd5d391d5da6a80fde03301671aea5582643914

Other similar remotes seem to work correctly.

ggperiodic <- list(package_name = "ggperiodic", 
                   repo_url = "https://github.com/eliocamp/ggperiodic", 
                   branch_name = "master",
                   commit = "74f41571b0f82cf8a46e4f782a0dad148d09df29")
rix(r_ver = "4.3.2",
    git_pkgs = ggperiodic,
    ide = "rstudio",
    project_path = ".",
    overwrite = TRUE,
    print = FALSE)

This seems to be an issue on the git2nisha thing, though. For some reason it fails with the first repo.

curl::curl_fetch_memory(url = 'http://git2nixsha.dev:1506/hash?repo_url=https://github.com/eliocamp/metR&branchName=master&commit=1dd5d391d5da6a80fde03301671aea5582643914')$status_code
#> [1] 500

Any commit seems to fail, too.

curl::curl_fetch_memory(url = 'http://git2nixsha.dev:1506/hash?repo_url=https://github.com/eliocamp/metR&branchName=master&commit=ff01f509513706facef846887f17a506c0df3563')$status_code
#> [1] 500

Am I missing something in the specification of the remote?

[b-rodrigues]

This is probably related: https://github.com/b-rodrigues/rix/issues/159

I don’t know what’s going on exactly, there seems to be an issue with certain repos. In my tests (for that other issue), I ran the api locally and was able to get the results. So I don’t really know what’s going on yet...

We are working on making this work without using the remote server if Nix is available. So this would be done locally and should work more consistently.

[philipp-baumann]

Let's also investigate while implementing linked whishlist for 0.8.0: https://github.com/b-rodrigues/rix/issues/143

[philipp-baumann]

Hey @b-rodrigues and @eliocamp This should be solved when hashing locally (the new feature added: ).

library(rix)

metR <- list(package_name = "metR", 
             repo_url = "https://github.com/eliocamp/metR", 
             branch_name = "master", 
             commit = "1dd5d391d5da6a80fde03301671aea5582643914")

rix(r_ver = "4.3.2",
    git_pkgs = metR,
    ide = "rstudio",
    project_path = "_test",
    overwrite = TRUE,
    print = FALSE)

This generates the following expression in default.nix:

# This file was generated by the {rix} R package v0.8.0 on 2024-07-04
# with following call:
# >rix(r_ver = "219f896bdf192721446db4fedc338eebf732057d",
#  > git_pkgs = metR,
#  > ide = "rstudio",
#  > project_path = ".",
#  > overwrite = TRUE)
# It uses nixpkgs' revision 219f896bdf192721446db4fedc338eebf732057d for reproducibility purposes
# which will install R version 4.3.2.
# Report any issues to https://github.com/b-rodrigues/rix
let
 pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/219f896bdf192721446db4fedc338eebf732057d.tar.gz") {};

  git_archive_pkgs = [
    (pkgs.rPackages.buildRPackage {
      name = "metR";
      src = pkgs.fetchgit {
        url = "https://github.com/eliocamp/metR";
        branchName = "master";
        rev = "1dd5d391d5da6a80fde03301671aea5582643914";
        sha256 = "sha256-CLTX347KwwsNyuX84hw2/n/9HwQHBYQrGDu7jFctGO4=";
      };
      propagatedBuildInputs = builtins.attrValues {
        inherit (pkgs.rPackages) 
          checkmate
          data_table
          digest
          Formula
          formula_tools
          ggplot2
          gtable
          memoise
          plyr
          scales
          sf
          stringr
          purrr
          isoband
          lubridate;
      };
    })
   ];

  system_packages = builtins.attrValues {
    inherit (pkgs) 
      R
      glibcLocales
      nix;
  };

in

pkgs.mkShell {
  LOCALE_ARCHIVE = if pkgs.system == "x86_64-linux" then  "${pkgs.glibcLocales}/lib/locale/locale-archive" else "";
  LANG = "en_US.UTF-8";
   LC_ALL = "en_US.UTF-8";
   LC_TIME = "en_US.UTF-8";
   LC_MONETARY = "en_US.UTF-8";
   LC_PAPER = "en_US.UTF-8";
   LC_MEASUREMENT = "en_US.UTF-8";

  buildInputs = [ git_archive_pkgs   system_packages   ];

}

nix_build(project_path = "_test")
# ...takes a while to build since we cannot get it from the official nix cachix cache ;-)
[100%] Generating gdal.jar, build/maven/gdal-3.8.4.jar, build/maven/gdal-3.8.4-sources.jar, build/maven/gdal-3.8.4.pom, build/maven/gdal-3.8.4-javadoc.jar
Buildfile: /tmp/nix-build-gdal-3.8.4.drv-0/source/swig/java/build.xml

init:
     [echo] version = 3.8.4

compile:
    [mkdir] Created dir: /tmp/nix-build-gdal-3.8.4.drv-0/source/build/swig/java/build/classes
    [javac] Compiling 88 source files to /tmp/nix-build-gdal-3.8.4.drv-0/source/build/swig/java/build/classes
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 7
    [javac] error: Source option 7 is no longer supported. Use 8 or later.
    [javac] error: Target option 7 is no longer supported. Use 8 or later.

BUILD FAILED
/tmp/nix-build-gdal-3.8.4.drv-0/source/swig/java/build.xml:25: Compile failed; see the compiler error output for details.

Total time: 0 seconds
make[2]: *** [swig/java/CMakeFiles/java_binding.dir/build.make:107: swig/java/gdal.jar] Error 1
make[1]: *** [CMakeFiles/Makefile2:9841: swig/java/CMakeFiles/java_binding.dir/all] Error 2
make: *** [Makefile:146: all] Error 2
error: builder for '/nix/store/ilcpzhq2jpva5pkwvk5r5fwd9gnx7zcj-gdal-3.8.4.drv' failed with exit code 2;
       last 10 log lines:
       >     [javac] error: Source option 7 is no longer supported. Use 8 or later.
       >     [javac] error: Target option 7 is no longer supported. Use 8 or later.
       >
       > BUILD FAILED
       > /tmp/nix-build-gdal-3.8.4.drv-0/source/swig/java/build.xml:25: Compile failed; see the compiler error output for details.
       >
       > Total time: 0 seconds
       > make[2]: *** [swig/java/CMakeFiles/java_binding.dir/build.make:107: swig/java/gdal.jar] Error 1
       > make[1]: *** [CMakeFiles/Makefile2:9841: swig/java/CMakeFiles/java_binding.dir/all] Error 2
       > make: *** [Makefile:146: all] Error 2
       For full logs, run 'nix log /nix/store/ilcpzhq2jpva5pkwvk5r5fwd9gnx7zcj-gdal-3.8.4.drv'.
error: 1 dependencies of derivation '/nix/store/58s1d8wqnl901qma286clvwyxk8m0fri-r-sf-1.0-15.drv' failed to build
error: 1 dependencies of derivation '/nix/store/c35djnxrbn1apdndmv6jf7bmy1bary88-r-metR.drv' failed to build
error: 1 dependencies of derivation '/nix/store/37bgkq0akrcppybv8cgi7wl2gmwpg9d2-nix-shell.drv' failed to build

@b-rodrigues The question I have is why are online hashing vs. local hashing not the same? Can you experimentally change the newly locally implementednix-hash vs. nix hash approach to compute the NAR SRI SHA-256 hashes of the guntared downloaded package directories? I really don't know why could be the case the API request on git2nixsha.dev on port 1506 is failing.

To sum up, the Git repo is not failing now anymore with the newest version (see changelog), however the issue with building seems to be a darwin-aarch64 related Nix issue (currently tested on Mac); guess linux does work. @eliocamp can you briefly confirm? Thanks a lot for reporting and generally your detailed feedback on Nix :-)

[philipp-baumann]

We get an internal server error when doing it online; my gut feeling tells me it takes quite long to download (what we do locally) and process the hashing in this case. Maybe quite a bit of (reverse) deps involved. So it maybe something with response timing?

> req <- curl::curl_fetch_memory(url = 'http://git2nixsha.dev:1506/hash?repo_url=https://github.com/eliocamp/metR&branchName=master&commit=1dd5d391d5da6a80fde03301671aea5582643914')
> jsonlite::fromJSON(rawToChar(req$content))
$error
[1] "500 - Internal server error"

[eliocamp]

I'm getting

Error in sys::exec_internal(cmd = cmd, args = args) : 
  Executing 'nix-hash' failed with status 1

[b-rodrigues]

That’s very surprising, could you do which nix-hash in a terminal?

[eliocamp]

/nix/var/nix/profiles/default/bin/nix-hash

[b-rodrigues]

@b-rodrigues The question I have is why are online hashing vs. local hashing not the same? Can you experimentally change the newly locally implementednix-hash vs. nix hash approach to compute the NAR SRI SHA-256 hashes of the guntared downloaded package directories? I really don't know why could be the case the API request on git2nixsha.dev on port 1506 is failing.

@philipp-baumann I’ll work on it this weekend; also I think I’ll get completely rid of the branchName argument, as per our Matrix discussion. After some thinking, I think that what you propose makes the most sense: since the commit correctly identifies the code at a point in time, no need for it, and we might as well get rid of it before submitting to CRAN.

/nix/var/nix/profiles/default/bin/nix-hash

@eliocamp Are you trying to build the same expression as Philipp, above? From an R session running from your system, or a Nix R session that contains rix? I wonder if perhaps (but I don’t know why/how) you local library of packages might interfere?

library(rix)

metR <- list(package_name = "metR", 
             repo_url = "https://github.com/eliocamp/metR", 
             branch_name = "master", 
             commit = "1dd5d391d5da6a80fde03301671aea5582643914")

rix(r_ver = "4.3.2",
    git_pkgs = metR,
    ide = "rstudio",
    project_path = "_test",
    overwrite = TRUE,
    print = FALSE)

[b-rodrigues]

@eliocamp could you try with latest master? You can now remove the 'branchName':

library(rix)

metR <- list(package_name = "metR", 
             repo_url = "https://github.com/eliocamp/metR", 
             commit = "1dd5d391d5da6a80fde03301671aea5582643914")

rix(r_ver = "4.3.2",
    git_pkgs = metR,
    ide = "rstudio",
    project_path = ".",
    overwrite = FALSE,
    print = TRUE)

[philipp-baumann]

I'm getting

Error in sys::exec_internal(cmd = cmd, args = args) : 
  Executing 'nix-hash' failed with status 1

@eliocamp that should be solved with https://github.com/b-rodrigues/rix/pull/244 for more detailed diagnostics why i implemented this approach, see https://github.com/b-rodrigues/rix/issues/243 does it work for you linux too?

[eliocamp]

Can confirm that

library(rix)

metR <- list(package_name = "metR", 
             repo_url = "https://github.com/eliocamp/metR", 
             commit = "1dd5d391d5da6a80fde03301671aea5582643914")

rix(r_ver = "4.3.2",
    git_pkgs = metR,
    ide = "rstudio",
    project_path = ".",
    overwrite = FALSE,
    print = TRUE)

creates the default.nix file correctly.

nix-build fails with

[100%] Built target python_binding
make: *** [Makefile:146: all] Error 2
error: builder for '/nix/store/vss6vy5px8fw1wh8rq5zrbpj9xx563vk-gdal-3.8.4.drv' failed with exit code 2;
       last 25 log lines:
       >
       > compile:
       >     [mkdir] Created dir: /build/source/build/swig/java/build/classes
       >     [javac] Compiling 88 source files to /build/source/build/swig/java/build/classes
       >     [javac] warning: [options] bootstrap class path not set in conjunction with -source 7
       >     [javac] error: Source option 7 is no longer supported. Use 8 or later.
       >     [javac] error: Target option 7 is no longer supported. Use 8 or later.
       >
       > BUILD FAILED
       > /build/source/swig/java/build.xml:25: Compile failed; see the compiler error output for details.
       >
       > Total time: 0 seconds
       > make[2]: *** [swig/java/CMakeFiles/java_binding.dir/build.make:107: swig/java/gdal.jar] Error 1
       > make[1]: *** [CMakeFiles/Makefile2:10036: swig/java/CMakeFiles/java_binding.dir/all] Error 2
       > make[1]: *** Waiting for unfinished jobs....
       > g++ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/iygsjhqpc4s47y0xar4pi5pi6pmfigad-libxcrypt-4.4.36/include -shared build/temp.linux-x86_64-cpython-311/extensions/ogr_wrap.o -L/build/source/build -L/nix/store/3v2ch16fkl50i85n05h5ckss8pxx6836-python3-3.11.8/lib -lgdal -o build/lib.linux-x86_64-cpython-311/osgeo/_ogr.cpython-311-x86_64-linux-gnu.so
       > g++ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/iygsjhqpc4s47y0xar4pi5pi6pmfigad-libxcrypt-4.4.36/include -shared build/temp.linux-x86_64-cpython-311/extensions/gdal_wrap.o -L/build/source/build -L/nix/store/3v2ch16fkl50i85n05h5ckss8pxx6836-python3-3.11.8/lib -lgdal -o build/lib.linux-x86_64-cpython-311/osgeo/_gdal.cpython-311-x86_64-linux-gnu.so
       > copying build/lib.linux-x86_64-cpython-311/osgeo/_gdal.cpython-311-x86_64-linux-gnu.so -> osgeo
       > copying build/lib.linux-x86_64-cpython-311/osgeo/_gdalconst.cpython-311-x86_64-linux-gnu.so -> osgeo
       > copying build/lib.linux-x86_64-cpython-311/osgeo/_osr.cpython-311-x86_64-linux-gnu.so -> osgeo
       > copying build/lib.linux-x86_64-cpython-311/osgeo/_ogr.cpython-311-x86_64-linux-gnu.so -> osgeo
       > copying build/lib.linux-x86_64-cpython-311/osgeo/_gnm.cpython-311-x86_64-linux-gnu.so -> osgeo
       > copying build/lib.linux-x86_64-cpython-311/osgeo/_gdal_array.cpython-311-x86_64-linux-gnu.so -> osgeo
       > [100%] Built target python_binding
       > make: *** [Makefile:146: all] Error 2
       For full logs, run 'nix log /nix/store/vss6vy5px8fw1wh8rq5zrbpj9xx563vk-gdal-3.8.4.drv'.
error: 1 dependencies of derivation '/nix/store/2rdsad5bw8sx94h5g0cz2wvpa1c8m9yz-r-sf-1.0-15.drv' failed to build
error: 1 dependencies of derivation '/nix/store/kkdyb5g529dac6ysng7mfn9q4jbndidy-r-metR.drv' failed to build
error: 1 dependencies of derivation '/nix/store/b31nmwqrwavgb2b2rcka5sa468fylm8n-nix-shell.drv' failed to build

Not sure if a nix/rix bug or just that that particular commit doesn't work. It seems to be a compilation error and metR doesn't have any source code that needs compiling, though.

[b-rodrigues]

it's sf that fails, in particular gdal apparently, what happens with a more recent commit ?

[eliocamp]

Not even

rix::rix(r_ver = "4.3.2",
    r_pkgs = "metR",
    ide = "rstudio",
    project_path = ".",
    overwrite = TRUE,
    print = TRUE)

works. 😱

I don't know if something changed with my system or with rix, but before environments would build in a few seconds and now it's taking ages.

[MGousseff]

Hello, still trying to use NIX, and I had the same error message. I used the code from your tutorial :

rix(
  r_ver = "latest",
  r_pkgs = c("dplyr", "janitor"),
  git_pkgs = list(
    list(
      package_name = "housing",
      repo_url = "https://github.com/rap4all/housing/",
      commit = "1c860959310b80e67c41f7bbdc3e84cef00df18e"
    ),
    list(
      package_name = "fusen",
      repo_url = "https://github.com/ThinkR-open/fusen",
      commit = "d617172447d2947efb20ad6a4463742b8a5d79dc"
    )
  ),
  ide = "code",
  project_path = path_default_nix,
  overwrite = TRUE
)

And get Error in sys::exec_internal(cmd = cmd, args = args) : Executing 'nix-hash' failed with status 1

I'm afraid that for an applied statistician quite non devops savvy, the use of NIX / RIX ; the learning curve is too steep.

[b-rodrigues]

that's weird, are you the latest version of rix?

[MGousseff]

I installed it like this :

utils::remove.packages("rix")
install.packages("rix", repos = c(
  "https://b-rodrigues.r-universe.dev",
  "https://cloud.r-project.org"
))
require(rix)

Then ran this :

path_default_nix <- tempdir()

rix(
  r_ver = "latest",
  r_pkgs = c("dplyr", "janitor"),
  git_pkgs = list(
    list(
      package_name = "housing",
      repo_url = "https://github.com/rap4all/housing/",
      commit = "1c860959310b80e67c41f7bbdc3e84cef00df18e"
    ),
    list(
      package_name = "fusen",
      repo_url = "https://github.com/ThinkR-open/fusen",
      commit = "d617172447d2947efb20ad6a4463742b8a5d79dc"
    )
  ),
  ide = "code",
  project_path = path_default_nix,
  overwrite = TRUE
)

And got this error message :

Error in sys::exec_internal(cmd = cmd, args = args) : 
  Executing 'nix-hash' failed with status 1

For information : I use r installed with rig, so if some lib paths are hardcoded it could be a problem.

[b-rodrigues]

instead of tempdir() could you try any other valid path on your computer ? tempdir() is weird sometimes

[MGousseff]

I got the same result with a "normal" directory. The error only occurs when I set some remote installations with git_pkgs.

[b-rodrigues]

can you then successfully build the generated expression using nix-build?

[b-rodrigues]

Could you try options(rix.sri_hash="api_server") and then try to generate the expression with Github packages again ?

[MGousseff]

I did, and the error message is :

Error: Request `curl::curl_fetch_memory(url = 'http://git2nixsha.dev:1506/hash?repo_url=https://github.com/rap4all/housing/&commit=1c860959310b80e67c41f7bbdc3e84cef00df18e')` failed:
 Failed to connect to git2nixsha.dev port 1506 after 4193 ms: No route to host
If it's a Github repo, check the url and commit.
Are these correct? If it's an archived CRAN package, check the name
of the package and the version number.

I checked that a normal remotes::install_github works (it does). From what I try to understand git2nixsha means there could be different ways to create the hash and it would cause this to fail ?

[b-rodrigues]

yes there are two ways to compute the hash and both are failing for you. What operating system are you using? does your installation of Nix work? can you run nix-shell --expr "$(curl -sl https://raw.githubusercontent.com/b-rodrigues/rix/master/inst/extdata/default.nix)" successfully ?

[MGousseff]

So I typed

nix-shell --expr --show-trace "$(curl -sl https://raw.githubusercontent.com/b-rodrigues/rix/master/inst/extdata/default.nix)"

(I added --show-trace to have a better error message since the expression alone give few informations).

I get the following error

error: cannot connect to socket at '/nix/var/nix/daemon-socket/socket': Connection refused

       … while realising the context of a path

       at «string»:14:9:

           13| let
           14|  pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/976fa3369d722e76f37c77493d99829540d43845.tar.gz") {};
             |         ^
           15|

The cannot connect to socket made me think of a permission problem so I ran the same command preceded by sudo. Only then did it downloaded the default.nix and started to build things (it still is as I'm writing this). Do you think it could be something tied to R not being launched as root ? I'll follow up as soon as the nix-shell command is done executing.

[b-rodrigues]

there's something wrong with your Nix installation, you don't need root to use Nix. How did you install Nix? did you use the determinate systems installer?

[b-rodrigues]

@philipp-baumann this convo makes me think we need a rix::check_install() that checks if everything is alright and provide some guidance if not

[MGousseff]

My current version is Ubuntu 22.04.4 LTS.

The determinate systems installer never worked, I didn't save all the error messages, but it did not install seamlessly. I had to :

nix-channel --add https://nixos.org/channels/nixpkgs-unstable
sudo nix-channel --update

and then an apt install nix. I f the determinate systems is the only Nix installation that works with Rix, I think I'll have to let this aside for a while, because I spent quite some time trying to install Nix, and I'm alas getting late on what I need to deliver. I appreciate your work but I lack system skills to use it for now.

[MGousseff]

The result of sudo nix-shell --expr --show-trace "$(curl -sl https://raw.githubusercontent.com/b-rodrigues/rix/master/inst/extdata/default.nix)"

was the following error message

building '/nix/store/hi03px6v99iri5invmkqmi47caz82bdb-r-curl-5.0.1.drv'...
unpacking sources
unpacking source archive /nix/store/wxldyqldqvs02jafv6pgp31z7qy2a9v2-curl_5.0.1.tar.gz
source root is curl
setting SOURCE_DATE_EPOCH to timestamp 1686179412 of file curl/MD5
patching sources
updateAutotoolsGnuConfigScriptsPhase
configuring
patching script interpreter paths in configure
configure: interpreter directive changed from "#!/bin/sh" to "/nix/store/ir0j7zqlw9dc49grmwplppc7gh0s40yf-bash-5.2-p15/bin/sh"
building
running tests
installing
During startup - Warning messages:
1: package 'utils' in options("defaultPackages") was not found 
2: package 'stats' in options("defaultPackages") was not found 
Error: .onLoad failed in loadNamespace() for 'utils', details:
  call: system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE)
  error: cannot popen '/nix/store/m1kcpc0n70sam6c1x8fdx6c5vwxjacia-which-2.21/bin/which 'uname' 2>/dev/null', probable reason 'Cannot allocate memory'
* removing '/nix/store/8yq377d5hgg0hjk93w5na89fnfbq4wdg-r-curl-5.0.1/library/curl'
error: builder for '/nix/store/hi03px6v99iri5invmkqmi47caz82bdb-r-curl-5.0.1.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/v0b2663c9s4kwz0hibqshq94x6z9dwch-r-rix.drv' failed to build

Is this message useful in anyway ?

[b-rodrigues]

yeah I think that something is not quite right with your installation. It's possible to install Nix using the determinate systems installer but also the official Nix installer but it requires more work. I don't think using the ubuntu repos to install Nix will work, because it'll be missing configuration steps that other installers perform. The best approach really is to the determinate systems installer and ideally not have a system wide install of R either

[MGousseff]

My installation of R is through rig, and it should not be a problem. Would you like me to remove my nix install and try again the determinate systems nix install and update you with errors I encounter ?

[b-rodrigues]

that'd be helpful !

[philipp-baumann]

@philipp-baumann this convo makes me think we need a rix::check_install() that checks if everything is alright and provide some guidance if not

very much agree! I´m not yet convinced it's a nix installer problem. I've seen too much suboptimal behavior of RStudio like this before: https://support.posit.co/hc/en-us/articles/115014830827-Why-is-libPaths-different-in-the-RStudio-IDE-vs-R . It might really well be rig does some more side effects. @MGousseff could you report Sys.getenv("PATH") in your rig R(Studio?) sesssion. What does Sys.getenv("R_LIBS_USER") and .libPaths() report?

[philipp-baumann]

i'd propose to use a --pure shell. I think the "which" problem seems familiar to me, but i have no exact reference. like

nix-shell --pure --expr --show-trace "$(curl -sl https://raw.githubusercontent.com/b-rodrigues/rix/master/inst/extdata/default.nix)"

@MGousseff any reason you used sudo before? ==> edit: sorry just saw you sudo comment above.

[b-rodrigues]

rig could also be doing some stuff on the side but in this case there's also something wrong with the Nix installation as it doesn't successfully build an expression

[MGousseff]

I do not use RStudio. In fact, the main reason I tried to use NIX / RIX is to make environments that are compatible with different IDE. Namely, I used intelliJ and its R plugin until it stopped working and we realized no one was maintaining the R plugin anymore (already 4 months or so with unusable R plugin for R 4.3.*). I donwgraded R with rig, and used previous versions of intelliJ, until I encountered compatibility problems between system libraries and R packages that use them (like GDAL/GEOS/PROJ used by sf, tmap, terra and so on). So then I tried again the RIX solution.

[MGousseff]

@philipp-baumann this convo makes me think we need a rix::check_install() that checks if everything is alright and provide some guidance if not

very much agree! I´m not yet convinced it's a nix installer problem. I've seen too much suboptimal behavior of RStudio like this before: https://support.posit.co/hc/en-us/articles/115014830827-Why-is-libPaths-different-in-the-RStudio-IDE-vs-R . It might really well be rig does some more side effects. @MGousseff could you report Sys.getenv("PATH") in your rig R(Studio?) sesssion. What does Sys.getenv("R_LIBS_USER") and .libPaths() report?

Wow, it's dirtier than I thought ;-) :

"/home/gousseff/.local/bin:/home/gousseff/bin:/home/gousseff/.sdkman/candidates/groovy/current/bin:/home/gousseff/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/home/gousseff/anaconda3/bin:/usr/local/texlive/2023/bin/x86_64-linux:/home/gousseff/.local/share/JetBrains/Toolbox/scripts"

I have no ideas where the games path come from, for instance.

[philipp-baumann]

@philipp-baumann this convo makes me think we need a rix::check_install() that checks if everything is alright and provide some guidance if not

very much agree! I´m not yet convinced it's a nix installer problem. I've seen too much suboptimal behavior of RStudio like this before: https://support.posit.co/hc/en-us/articles/115014830827-Why-is-libPaths-different-in-the-RStudio-IDE-vs-R . It might really well be rig does some more side effects. @MGousseff could you report Sys.getenv("PATH") in your rig R(Studio?) sesssion. What does Sys.getenv("R_LIBS_USER") and .libPaths() report?

Wow, it's dirtier than I thought ;-) :

"/home/gousseff/.local/bin:/home/gousseff/bin:/home/gousseff/.sdkman/candidates/groovy/current/bin:/home/gousseff/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/home/gousseff/anaconda3/bin:/usr/local/texlive/2023/bin/x86_64-linux:/home/gousseff/.local/share/JetBrains/Toolbox/scripts"

I have no ideas where the games path come from, for instance.

i'm just comparing on my WSL2 Ubuntu 22.04 with nix installed at work. my list is much longer. But what's missing is nix stuff. i have also "/nix/var/nix/profiles/default/bin" in the list. I *think i have an idea. what is

# we have a helper inside rix, which adds `/nix/var/nix/profiles/default/bin` to the path
rix:::set_nix_path()

now maybe you can call nix_build() successfully inside your rix rsession?

[MGousseff]

So I uninstalled all nix installation I found. By the way, I think that until nix provides a simple .deb package I can install AND UNINSTALL with dpkg or apt, I will find it hard to trust it. Having to go and erase manually any mention of it in bashsrc like fill is tedious.

I then ran curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install

and had this message :

 INFO nix-installer v0.24.0
`nix-installer` needs to run as `root`, attempting to escalate now via `sudo`...
 INFO nix-installer v0.24.0
warning: the group 'nixbld' specified in 'build-users-group' does not exist
nix-env (Nix) 2.6.0
`nix` is already a valid command, so it is installed

But does it mean it was already installed and nothing was done or that it is now installed in a valid way ?

[b-rodrigues]

I think nothing was done, what does which nix do? is the nix aptitude package still installed?

[MGousseff]

```r
# we have a helper inside rix, which adds `/nix/var/nix/profiles/default/bin` to the path
rix:::set_nix_path()

now maybe you can call nix_build() successfully inside your rix rsession?

I ran rix:::set_nix_path() which returned nothing, so I don't know if it did what it was supposed to do.

Then I ran :

rix(
  r_ver = "4.2.1",
  r_pkgs = c("dplyr", "janitor"),
  git_pkgs = list(
    list(
      package_name = "housing",
      repo_url = "https://github.com/rap4all/housing/",
      commit = "1c860959310b80e67c41f7bbdc3e84cef00df18e"
    ),
    list(
      package_name = "fusen",
      repo_url = "https://github.com/ThinkR-open/fusen",
      commit = "d617172447d2947efb20ad6a4463742b8a5d79dc"
    )
  ),
  ide = "other",
  project_path = path_default_nix,
  overwrite = TRUE
)

and got... the exact same error message :

Error in sys::exec_internal(cmd = cmd, args = args) : 
  Executing 'nix-hash' failed with status 1

[MGousseff]

I think nothing was done, what does which nix do? is the nix aptitude package still installed?

which nix returns : /usr/bin/nix

Is nix aptitude package still installed ? There was a nix-bin that apt accepted to remove. Then I retried

curl --proto '=https' --tlsv1.2 -sSf \
    -L https://install.determinate.systems/nix | \
     sh -s -- install

and got the initial errors that made me try other installations : INFO nix-installer v0.24.0 nix-installer needs to run as root, attempting to escalate now via sudo... INFO nix-installer v0.24.0 Error: 0: Planner error 1: Error executing action 2: Action create_upstream_init_service errored 3: Action configure_init_service errored 4: /etc/systemd/system/nix-daemon.service already exists, consider removing it with rm /etc/systemd/system/nix-daemon.service

Location: src/cli/subcommand/install.rs:200

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it. Run with RUST_BACKTRACE=full to include source snippets.

Consider reporting this error using this URL: https://github.com/DeterminateSystems/nix-installer/issues/new?title=%3Cautogenerated-issue%3E&body=%23%23+Error%0A%60%60%60%0AError%3A+%0A+++0%3A+Planner+error%0A+++1%3A+Error+executing+action%0A+++2%3A+Action+%60create_upstream_init_service%60+errored%0A+++3%3A+Action+%60configure_init_service%60+errored%0A+++4%3A+%60%2Fetc%2Fsystemd%2Fsystem%2Fnix-daemon.service%60+already+exists%2C+consider+removing+it+with+%60rm+%2Fetc%2Fsystemd%2Fsystem%2Fnix-daemon.service%60%0A%60%60%60%0A%0A%23%23+Metadata%0A%7Ckey%7Cvalue%7C%0A%7C--%7C--%7C%0A%7C**version**%7C0.24.0%7C%0A%7C**os**%7Clinux%7C%0A%7C**arch**%7Cx86_64%7C%0A```

So I removed said `/etc/systemd/system/nix-daemon.service` and got `   4: `/etc/systemd/system/nix-daemon.socket` already exists, consider removing it with `rm /etc/systemd/system/nix-daemon.socket` `
So I removed said nix-daemon.socket and ran the install again and... didn't get any error. 

Now I got back in my R session and tried and... It's a new error. But I'm confident the nix install is OK as the error seems linked to the use of vscode. I ran : 

rix( r_ver = "latest", r_pkgs = c("dplyr", "janitor"), git_pkgs = list(
package_name = "lczexplore", repo_url = "https://github.com/orbisgis/lczexplore", commit = "33a71ef9ad3b8acc03226c42ce99364be3b65d3a" ), ide = "code", project_path = path_default_nix, overwrite = TRUE )

And the error message is : ``` cannot open the connectionwarning messages from top-level task callback 'vsc.workspace'
Warning message:
In file(con, "w") :
  cannot open file '/tmp/Rtmphz5efQ/vscode-R/workspace.json': No such file or directory

Does it mean rix can only be used with no IDE ? I feel too old to go back to learning emacs for R. If I can't use intellij, nor RStudio nor VSCode, it will be too much work for me to use all this. Once again, I do not belittle your good work on the RIX package, I just wonder if I lack too much skills to use it.

[philipp-baumann]

```r
# we have a helper inside rix, which adds `/nix/var/nix/profiles/default/bin` to the path
rix:::set_nix_path()

now maybe you can call nix_build() successfully inside your rix rsession?

I ran rix:::set_nix_path() which returned nothing, so I don't know if it did what it was supposed to do.

Then I ran :

rix(
  r_ver = "4.2.1",
  r_pkgs = c("dplyr", "janitor"),
  git_pkgs = list(
    list(
      package_name = "housing",
      repo_url = "https://github.com/rap4all/housing/",
      commit = "1c860959310b80e67c41f7bbdc3e84cef00df18e"
    ),
    list(
      package_name = "fusen",
      repo_url = "https://github.com/ThinkR-open/fusen",
      commit = "d617172447d2947efb20ad6a4463742b8a5d79dc"
    )
  ),
  ide = "other",
  project_path = path_default_nix,
  overwrite = TRUE
)

and got... the exact same error message :

Error in sys::exec_internal(cmd = cmd, args = args) : 
  Executing 'nix-hash' failed with status 1

To me this shows

I think nothing was done, what does which nix do? is the nix aptitude package still installed?

which nix returns : /usr/bin/nix

Is nix aptitude package still installed ? There was a nix-bin that apt accepted to remove. Then I retried

curl --proto '=https' --tlsv1.2 -sSf \
    -L https://install.determinate.systems/nix | \
     sh -s -- install

and got the initial errors that made me try other installations : INFO nix-installer v0.24.0 nix-installer needs to run as root, attempting to escalate now via sudo... INFO nix-installer v0.24.0 Error: 0: Planner error 1: Error executing action 2: Action create_upstream_init_service errored 3: Action configure_init_service errored 4: /etc/systemd/system/nix-daemon.service already exists, consider removing it with rm /etc/systemd/system/nix-daemon.service

Location: src/cli/subcommand/install.rs:200

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it. Run with RUST_BACKTRACE=full to include source snippets.

Consider reporting this error using this URL: https://github.com/DeterminateSystems/nix-installer/issues/new?title=%3Cautogenerated-issue%3E&body=%23%23+Error%0A%60%60%60%0AError%3A+%0A+++0%3A+Planner+error%0A+++1%3A+Error+executing+action%0A+++2%3A+Action+%60create_upstream_init_service%60+errored%0A+++3%3A+Action+%60configure_init_service%60+errored%0A+++4%3A+%60%2Fetc%2Fsystemd%2Fsystem%2Fnix-daemon.service%60+already+exists%2C+consider+removing+it+with+%60rm+%2Fetc%2Fsystemd%2Fsystem%2Fnix-daemon.service%60%0A%60%60%60%0A%0A%23%23+Metadata%0A%7Ckey%7Cvalue%7C%0A%7C--%7C--%7C%0A%7Cversion%7C0.24.0%7C%0A%7Cos%7Clinux%7C%0A%7Carch%7Cx86_64%7C%0A```

So I removed said `/etc/systemd/system/nix-daemon.service` and got `   4: `/etc/systemd/system/nix-daemon.socket` already exists, consider removing it with `rm /etc/systemd/system/nix-daemon.socket` `
So I removed said nix-daemon.socket and ran the install again and... didn't get any error. 

Now I got back in my R session and tried and... It's a new error. But I'm confident the nix install is OK as the error seems linked to the use of vscode. I ran : 

rix( r_ver = "latest", r_pkgs = c("dplyr", "janitor"), git_pkgs = list( package_name = "lczexplore", repo_url = "https://github.com/orbisgis/lczexplore", commit = "33a71ef9ad3b8acc03226c42ce99364be3b65d3a" ), ide = "code", project_path = path_default_nix, overwrite = TRUE )

And the error message is : ``` cannot open the connectionwarning messages from top-level task callback 'vsc.workspace'
Warning message:
In file(con, "w") :
  cannot open file '/tmp/Rtmphz5efQ/vscode-R/workspace.json': No such file or directory

Does it mean rix can only be used with no IDE ? I feel too old to go back to learning emacs for R. If I can't use intellij, nor RStudio nor VSCode, it will be too much work for me to use all this. Once again, I do not belittle your good work on the TIX package, I just wonder if I lack too much skills to use it.

Thanks! It's super annoying, i get you! i work in VSCode via WSL2 in nix via rix and that does play well. You absolutely do not lack skills, it's the convoluted setup. From all the good diagnostics you provided and also what we have done so far to fix some of the annoying stuff and behavior, I am fairly confident your issue can be fixed too. set_nix_path() returns invisibly. see https://github.com/b-rodrigues/rix/blob/5a95641f03bb9fc4941e7d420adcdcde754adfc2/R/rix_init.R#L368-L387

So it should really add Nix default path of the system-wide profile to PATH environment, which from your output i read you haven't had in your default ubuntu shell.

[philipp-baumann]

@MGousseff this is what i have:

which nix
/nix/var/nix/profiles/default/bin/nix

[MGousseff]

Sys.getenv("R_LIBS_USER") returns "/home/gousseff/R/x86_64-pc-linux-gnu-library/4.4"

and .libPaths() returns "/home/gousseff/R/x86_64-pc-linux-gnu-library/4.4" "/opt/R/4.4.1/lib/R/library"

And every command now also returns a warning :

cannot open the connectionwarning messages from top-level task callback 'vsc.workspace' Warning message: In file(con, "w") : cannot open file '/tmp/Rtmphz5efQ/vscode-R/workspace.json': No such file or directory

which nix now returns... nothing (even though I ran set_nix_path())

[MGousseff]

I checked : nix does exist in /nix/var/nix/profiles/default/bin so I don't get why I can't add it to paths. Shall I do it directly and not using the set_nix_path() function ? And if so what would be the safest way to do it ?

[b-rodrigues]

Does it mean rix can only be used with no IDE ? I feel too old to go back to learning emacs for R. If I can't use intellij, nor RStudio nor VSCode, it will be too much work for me to use all this.

it can, see https://b-rodrigues.github.io/rix/articles/e-interactive-use.html#scenario-3-you-use-some-ide-other-than-rstudio

[MGousseff]

Does it mean rix can only be used with no IDE ? I feel too old to go back to learning emacs for R. If I can't use intellij, nor RStudio nor VSCode, it will be too much work for me to use all this.

it can, see https://b-rodrigues.github.io/rix/articles/e-interactive-use.html#scenario-3-you-use-some-ide-other-than-rstudio

I read your article, and that is why I chose ide="code" in the rix() call.

But as for now, I didn't manage to make a single NIX environment following these instructions.

I do not see myself as even remotely skilled in "devops", but I feel some of my esteemed statisticians colleagues are even less prone to persist on those kind of installations. I'm afraid this could somehow slow down the adoption of rix. But once again, I may underestimate my inability to install things. The way I see things: it is thanks to disabled people that we started to improve our towns in terms of boardwalks and so on. Maybe my case can help make the installation of nix and rix more straightforward fo all ubuntu users. For instance, I can't understand how the nix package in the software installer of ubuntu is not compatible, or why nix won't provide a proper deb package, easy to install and uninstall. Please notice these remarks do not concern rix but nix, and I don't berate any one here, just providing feedback.

[philipp-baumann]

I checked : nix does exist in /nix/var/nix/profiles/default/bin so I don't get why I can't add it to paths. Shall I do it directly and not using the set_nix_path() function ? And if so what would be the safest way to do it ?

I checked : nix does exist in /nix/var/nix/profiles/default/bin so I don't get why I can't add it to paths. Shall I do it directly and not using the set_nix_path() function ? And if so what would be the safest way to do it ?

I *think this nix profile path should be added to the path when your default shell starts (via $HOME/.profile or $HOME/.bashrc), and the nix installation process should have done so. In your case i think the nix installer is causing the troubles here. I should have noted what i did but as said this sounds too familiar because i once did a tweak to add the nix profile. I would really for now go with a plain bash terminal, start R from there. https://github.com/NixOS/nix/issues/1396 and stuff like this is annoying.

First, maybe you can do.

1. open bash terminal.
2. append path manually:

   export PATH=$PATH:/home/<your-username>/.nix-profile/bin:/nix/var/nix/profiles/default/bin

3. then run expression in pure environment so that no programs are leaking from host system.

   sudo nix-shell --pure --expr --show-trace "$(curl -sl https://raw.githubusercontent.com/b-rodrigues/rix/master/inst/extdata/default.nix)"

edit: added --pure i forgot before

[MGousseff]

@philipp-baumann Thank you for your patience and perseverance. I added the path manually. I had to run nix-shell without sudo as it would not find nix-shell if "sudoed".
It is currently copying. I keep you updated once it's done.

[philipp-baumann]

@philipp-baumann Thank you for your patience and perseverance. I added the path manually. I had to run nix-shell without sudo as it would not find nix-shell if "sudoed". It is currently copying. I keep you updated once it's done.

that's a good sign, happy to hear :-)

[MGousseff]

It seems it worked ? Here is the end of the message :

post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/929fx96hhwpbqwcr75c4jwn73kgam79f-r-rix
checking for references to /build/ in /nix/store/929fx96hhwpbqwcr75c4jwn73kgam79f-r-rix...
patching script interpreter paths in /nix/store/929fx96hhwpbqwcr75c4jwn73kgam79f-r-rix
rewriting symlink /nix/store/929fx96hhwpbqwcr75c4jwn73kgam79f-r-rix/nix-support/propagated-user-env-packages to be relative to /nix/store/929fx96hhwpbqwcr75c4jwn73kgam79f-r-rix

Now, what would be the safest next steps to try and buil a rix environment using VSCode (or RStudio if you feel it's more stable). Do you think I have to kill my rig R installation and go for a new R installation with the default directory I don't like so much ? Or can I try and launch VSCode with the R plugin and specify the R interpreter I want and launch rix from there ?