Add install_julia function to download Julia

[christopher-dG]

Description

Adds the install_julia function which will install Julia for you. The user can choose where the installation will go, otherwise it will be put in the user's "data dir" e.g. ~/.local/share/JuliaCall on Linux. The installed location is remembered by adding a file $JULIA_DEPOT_PATH/prefs/JuliaCall which contains the install path, and that file is checked in julia_locate.

Related Issue

116

---

Probably the most finicky part of this is building up the download URL, so some eyes/suggestions on that bit would be appreciated. Still needs tests and docs, I would just like to see if the approach is okay.

Also please point out any really awful bits of code, and suggest the "right" way to do it (this is my first contribution in R).

[Non-Contradiction]

Looks good to me! And I also tested this on my windows laptop.

I notice that the julia version is hard coded. I try a little for urls corresponding to different versions. It seems the url only works for recent versions. I guess we should at least to have a mechanism to grab the latest julia release information.

Also, should the installation folder contain the version number?

Maybe we could further test it on travis CI? So we could try to use install_julia to install julia instead of manual installation in travis.yml.

[christopher-dG]

I notice that the julia version is hard coded. I try a little for urls corresponding to different versions. It seems the url only works for recent versions. I guess we should at least to have a mechanism to grab the latest julia release information.

Since this uses Julia_jll instead of the normal Julia download page (to get tarballs instead of installers or .dmg files), we only have access to a couple of versions. We could try some automatic highest-version lookup, but that would involve either HTML or JSON parsing which requires another dependency to do properly. Or maybe a good idea would be to parse the Project.toml in the master branch to get what the Julia version is? That wouldn't be too hard to do I think.

Also, should the installation folder contain the version number?

I considered this, but then I thought that maybe it would be annoying for users if a new Julia version came out, we updated the version that JuliaCall looks for, and suddenly it no longer finds Julia installed when it did yesterday. We could still do this though, and check the install dir for any vX.Y.Z dir and consider that a Julia installation.

Maybe we could further test it on travis CI? So we could try to use install_julia to install julia instead of manual installation in travis.yml.

Good idea! Will do.

---

For 1 and 2, I'll try to implement them tomorrow and see how it goes.

[Non-Contradiction]

I considered this, but then I thought that maybe it would be annoying for users if a new Julia version came out, we updated the version that JuliaCall looks for, and suddenly it no longer finds Julia installed when it did yesterday. We could still do this though, and check the install dir for any vX.Y.Z dir and consider that a Julia installation.

Good point! The problem can be alleviated since JuliaCall can remember the location of the installed julia in the prefs file. Another thing to consider is that the current without version number behavior somewhat excludes possibility of having different versions of julia simultaneously. So I think folder name with version number should be better. (In the future, JuliaCall may also remembers a list of different julia locations, and gives options for users to use different versions. Further, we can provide users functions to manage the julia installations. But this can be implemented later.)

[Non-Contradiction]

Another miscellaneous thing I just noticed is that on my laptop, the JuliaCall prefs and the .julia prefs are not in the same location because of the difference in determining the home directory: homedir() in julia gives "C:\Users\lch34" but path.expand("~") in R gives "C:/Users/lch34/OneDrive/Documents". I'm not sure whether this is due to my OneDrive setting.

[christopher-dG]

Ah, I was wondering how that would work on WIndows... Do you know how to get home directory in R?

[Non-Contradiction]

This works on my windows laptop, but I have no idea whether this is a robust way for windows system. shell("echo %USERPROFILE%", intern = TRUE) gives me the correct folder in R

[christopher-dG]

Just tried myself and you can do Sys.getenv("USERPROFILE") which seems a bit less hacky.

Also I've deleted the AppVeyor config and replaced it with Windows on Travis, does that seem like a good idea?

[Non-Contradiction]

Thanks! That should work. Yes, I think it is totally okay to replace appveyor windows by travis windows.

[christopher-dG]

So turns out Travis doesn't support R on Windows yet 🙁

[christopher-dG]

We could look at migrating to GitHub Actions which supports all three major OS options, but maybe that can be for a separate PR.

[Non-Contradiction]

Yes, I have always considering migrating to GitHub action but don't have the time. And manually installation of julia really complicates the migration of CI. Hopefully the process can be made easy with the new install_julia function.

[christopher-dG]

I keep getting this error:

  Error: 'install_julia' is not an exported object from 'namespace:JuliaCall'

But I have the @export bit above the function and I don't know what else to do. Do you know what the problem is?

[Non-Contradiction]

@christopher-dG Beyond the @export, you should also run devtools::document() in R when your working directory is the root of the package, so devtools will use Roxygen to update the NAMESPACE, where export really happens.

[Non-Contradiction]

I take a further look at the travis testing log, and it is okay in general except a few notes. I think we should also add utils into the suggestion field in DESCRIPTION, use requireNamespace(quiet = TRUE) instead of require for rappdirs, and use utils:: and rappdirs:: to quantify the usage of download.file, untar, and user_data_dir. Also, currrently julia is not installed when checking the examples. So there is an error about julia not installed.

[ChrisRackauckas]

I assume the translation of this function can be used in pyjulia as well @tkf?

[tkf]

I do not want to get in the way of JuliaCall devs so this is just me talking to myself. But, frankly, I'm against the approach that each X-Julia bridge library for language X implements Julia installer/manager. I tried to summarize my thoughts in https://github.com/JuliaPy/pyjulia/issues/118#issuecomment-687683910

[Non-Contradiction]

@tkf Thanks for your insightful thought! I agree that it is not a perfect approach for each X-Julia bridge library to have its own Julia installer/manager in the long run. And a general solution as proposed in JuliaPy/pyjulia#118 would be really nice.

This PR gives a direct solution for the julia automatic installation problem based on current utilities, verified the feasibility of the solution, and builds up the basic corresponding logic in the JuliaCall package. Also, the interface/functions in this PR can definitely use a more mature and general julia installation/management solution as the "backend" once it comes up. So the endeavor in the PR won't be wasted.

[tkf]

Yeah, it's conceivable that you'll find something important by shipping this feature. Then, it might give us some insights for designing a more general solution. I'd certainly appreciate if JuliaCall goes ahead and accumulate and share such experience.

[christopher-dG]

Looks like tests finally pass. I'm wanting to add checksum/GPG verification now, is there a stdlib way to calculate an MD5 or SHA256? It looks like openssl and digest have the functionality but are both third party dependencies. There's a gpg package but same deal.

Maybe we can make digest and gpg optional dependencies and if they're not installed, try to use CLI programs (sha256sum, gpg) if they're installed, otherwise just warn that verification isn't happening?

[Non-Contradiction]

For this purpose, I find tools::md5sum, which is in R standard library.

The appveyor failure seems to be related to compilation. And I guess it is due to that the Rtools (compilation toolchain used by R) version does not match the new R version. There is some guidance on how to install and setup new Rtools at https://cran.r-project.org/bin/windows/Rtools/

[christopher-dG]

Actually, now that I think about it, verifying SHA + GPG is not really doable for this since we're using the Julia_jll tarballs instead of the files from julialang.org/downloads. It'll work for Linux, but that's it (since the Windows/Mac files on the Julia downloads page are not tarballs). IMO it's not really worth it in that case. So we end up trusting GitHub to serve us the right files, which we already do a fair bit of. Thoughts?

[Non-Contradiction]

It seems that we could dig out the checksum from the julia_jll repo. But I guess it does not worth the effort to do this. Then I think we don't need to check the checksum currently in this version. We can add the functionality later once the utility is there or some general solution emerges.

[christopher-dG]

Ah yeah, the hashes are indeed in Artifacts.toml. But then we need a TOML parser, and I agree it is likely not worth the effort. In that case, everything here is done minus AppVeyor, which I'm not quite sure how to fix.

[Non-Contradiction]

I still suspect that the error is from the compilation part. After investigating into https://cran.r-project.org/bin/windows/Rtools/, I think the problem is that the Rtools changes its directory structure so the new Rtools is not correctly on path.
Previously, we have https://github.com/Non-Contradiction/JuliaCall/blob/cec33df007e53b6421933466399aa820abd725f3/appveyor.yml#L72 But new https://cran.r-project.org/bin/windows/Rtools/ suggests that we do the following: PATH="${RTOOLS40_HOME}\usr\bin;${PATH}" So I guess this difference in \usr\bin vs bin maybe the cause for appveyor failure. Beyond that, we also don't have automatic installation of julia on appveyor. After change of this, I expect the failure to go away. But if it still persists, one way to debug the issue is to uncomment the last line of the appveyor.yml and use Remote Desktop.

[christopher-dG]

Plenty of progress has been made... but still no success. Julia does get installed correctly as the last step of the build_script (I've checked via RDP), but it seems like the test_script never gets run and a confusing error message is left:

Command executed with exception: 
downloaded 115.3 MB

Which comes from downloading the Julia tarball.

[christopher-dG]

Since I couldn't really get AppVeyor figured out, I've tried to set up GitHub Actions according to this example. But unfortunately it seems like it needs to be merged before it will run... 😬

[ChrisRackauckas]

@Non-Contradiction what's left to get this merged?

[Non-Contradiction]

Thanks for the reminder! I think it is good to go and we can make further changes if needed after merging.