Running inside AWS Glue endpoint machine w/o internet

[yelled1]

Question

I am trying to get Metal inside AWS Glue endpoint machine w/o internet by sftp or rsync files from a WSL (Ubuntu 20.04) instance.

• Operating system: Linux
• Editor: Visual Studio Code
• Metals version: Latest

Could I copy ~/project_dir/.metal from WSL to ~/project_dir/.metal in AWS or is there more directories & files to copy over?

N.B.:

Just found out that ~/.cache directory has

drwxrwxrwx 3 glue glue 4096 May 29 00:44 coursier drwxrwxrwx 2 glue glue 4096 May 29 18:44 metals Do believe this is the key files as .metals directories has just: -rw-rw-r-- 1 glue glue 118784 Jun 1 20:59 metals.h2.db -rw-rw-r-- 1 glue glue 3997 Jun 1 20:59 metals.log

Or can I use WSL for developement by copying those jar files java path as well? $ ls -l jars/ total 585984 -rw-r--r-- 1 hyoon hyoon 107815 Jun 1 14:38 aws-glue-datacatalog-spark-client-1.8.0-SNAPSHOT.jar -rw-r--r-- 1 hyoon hyoon 546757895 Jun 1 14:43 glue-assembly.jar -rw-rw-r-- 1 hyoon hyoon 53178063 Jun 1 14:38 glueml-assembly.jar

Additional context AWS & company admin policy do NOT allow internet access AWS Endpoint says $ uname -a Linux ip-172-32-49-52 4.14.133-88.112.amzn1.x86_64 #1 SMP Tue Jul 30 21:21:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux WSL says $ uname -a Linux NY11559-MSW10E 4.4.0-18362-Microsoft #836-Microsoft Mon May 05 16:04:00 PST 2020 x86_64 x86_64 x86_64 GNU/Linux

Search terms AWS, glue, gluecontext, no internet

[tgodzik]

This is a good question. We do not currently have a valid solution for no internet use cases. There is some discussion about adding an offline mode here: https://github.com/scalameta/metals-feature-requests/issues/9

But currently you would need to copy things from .cache directory. In particular .cache/bloop and .cache/coursier. You could possibly also need to copy .ivy2 directory, but it should not be needed.

Or can I use WSL for developement by copying those jar files java path as well?

I think this should be possible, but you would also need to copy the vscode extension from .vscode/extensions/scalameta.metals-1.9.0/ and .vscode/extensions/scala-lang.scala-0.3.9/. Not sure about the glue-assembly files.

[yelled1]

@tgodzik Thank you for pointing that feature request out. I add those jars as symlink to:

/opt/spark/jars/
lrwxrwxrwx /opt/spark/jars/glueml-assembly.jar -> /opt/jars/glueml-assembly.jar
lrwxrwxrwx /opt/spark/jars/glue-assembly.jar -> /opt/jars/glue-assembly.jar
lrwxrwxrwx /opt/spark/jars/aws-glue-datacatalog-spark-client-1.8.0-SNAPSHOT.jar -> /opt/jars/aws-glue-datacatalog-spark-client-1.8.0-SNAPSHOT.jar

Which seems to have set up for WSL & fill the "metals" directories in ~/.cache. I am getting autocomplete at both WSL & AWS Glue endpoint.

drwxrwxrwx 1 hyoon hyoon 4096 Jun  1 15:33 coursier
drwxrwxrwx 1 hyoon hyoon 4096 Jun  1 15:34 metals 

The ~/.vscode-server/extension/soursier not changed though: -rwxr-xr-x 1 hyoon hyoon 24825 May 31 14:29 coursier

I wonder for if metal could use something like below for VS Code only: SSH with VSCode without internet I realize that this is a bit off-topic as this would be editor specific rather than in general; however, there might be a potential gateway that can be used for VSCode at least, which likely covers a big chunk of the users, especially very few editors could directly ssh into a project (one can always resort to text editors [like vi or emacs] or pay for it via JetBrains). Potentially text based editors could use ssh/mosh tunnel as well?

As everyone is well aware, InfoSec demands this type of offline usage all the time. The reason I switched to VSCode (from nvim) is because it's one of few editors which can do this without setting up rsync or scp getup as in JetBrains or Atom.

Well, at least this works for VSCode now. A bit of pain & adventure, which was still worth it.

Here's an exchange with someone who uses his own github repo for vim https://github.com/VundleVim/Vundle.vim/issues/135

[tgodzik]

I wonder for if metal could use something like below for VS Code only:

Metals does work with SSH, though you would still need to copy the coursier and Bloop directories. This could for sure be optimized.

We could something like this with coursier:

coursier launch org.scalameta:metals_2.12:0.8.3 -J-Dcoursier.cache=./output -M scala.meta.metals.DownloadDependencies
tar -zcvf metals-deps.tar.gz output

Then copy that tar via ssh, unpack to .cache/coursier and this should work for the server dependencies.

However, there might still be some additional stuff that we might need to do depending on the environment. Unfortunately, I don't think we will be able to implement a full solution inside Metals, but I imagine someone could create and extension that would do all the steps automatically for ssh.

[yelled1]

May be more efficient to do rsync type of extension as that will only copy changed files: Something like coursier then rsync...

coursier launch org.scalameta:metals_2.12:0.8.3 -J-Dcoursier.cache=./output -M scala.meta.metals.DownloadDependencies 
(I'm assuming above will update the ~/.cache/coursier directory on source WSL)

rsync -auv --rsh="ssh -i /home/WSLuser/.ssh/id_rsa_private_key" /home/WSLuser/.cache/coursier/* glue@glue_machine:/home/glue/.cache/coursier/ 

A bit of issue is that I have Windows machine that goes through WSL via ssh to glue & with VS Code in Windows, but others might want to do OwnLinux or Mac vs glue. Ultimately, would this be more efficient than just a cron type of job on WSL? And WSL internet access does likely get restricted & proxies need to be setup on WSL side, if that can be done at all. I am not quite sure. Definitely good for a show & tell write up, as the road will begin with. BTW: Setting up open-ssh on Windows is NOT easy as pie, but whole thing only works when source & client are both Linux on Glue. I guess one can do Mac -> Linux. But as u pointed out we can encounter Windows -> Linux , Linux -> Windows, or Win -> Win as well.

Also, I just encountered follow error: $ coursier Command 'coursier' not found, did you mean: command 'courier' from deb courier-mta (1.0.6-1build2) Try: sudo apt install Which means it can become quite a detailed write up.

I guess given I am on: Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_252) & latest Metals is 0.9.0 above should be revised to:

coursier launch org.scalameta:metals_2.11:0.9.0  -J-Dcoursier.cache=./output -M scala.meta.metals.DownloadDependencies 

Hmm, I wonder if the jars in /opt/scala/jars would be reflected with above command...? I wonder if coursier could add jar directory w/o sbt or other means. Sorry, not much of java person. @tgodzik can you advise? This likely only needs to done when jar files are added or once in a while refresh due to metals changes.

[tgodzik]

-J-Dcoursier.cache=./output will cause the cache to actually be downloaded into .output, which should be a smaller size and quicker to copy.

Also, I just encountered follow error: $ coursier Command 'coursier' not found, did you mean:

You need to fist download coursier here: https://get-coursier.io/ Forgot to add that, sorry! It might be coursier jar or the cs native assembly.

Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_252) & latest Metals is 0.9.0 above should be revised to:

Server is only compiled against 2.12 and DownloadDependencies does the job of downloading everything that is needed (also deps for 2.11.12). We do it also via coursier, so everything will just go to the specified coursier cache.

Hmm, I wonder if the jars in /opt/scala/jars would be reflected with above command...?

Those jars are not used by sbt, coursier or Metals, so they are not needed in this scenario.

[tgodzik]

Closing the issue, since basically what we want is a new feature as defined in scalameta/metals-feature-requests#9

You can also ask us questions on gitter or discord.

[yelled1]

Closing the issue, since basically what we want is a new feature as defined in scalameta/metals-feature-requests#9

You can also ask us questions on gitter or discord.

Thanks much. Will ask Q there. I just had to redo this over again. On Python side, it was really nice to see MS making sure everything it provided worked. I was sorry to see Kite go, but not worth the troubles. Metal is a bit involved, but nothing as difficult as Kite, which was a standalone app that linked with editors. Just copied over the coursier from WSL:

./.cache/coursier/v1/https/repo1.maven.org/maven2/org/scalameta/metals_2.12
./.cache/metals

But VsCode is hanging on installs, even though I see:

./.vscode-server/extensions/alssys.interactive-scala-1.2.1/.metals
./.vscode-server/extensions/scalameta.metals-1.9.0
./.vscode-server/extensions/scalameta.metals-1.9.0/node_modules/metals-languageclient

Installed already. (Well, this took about 2 hrs to get done, but it did on its own after I rsynced .cache/coursier & .cache/metals; however, without w/o copying or rysncing. Very nice.)

rsync -auv --rsh="ssh -i /home/me/.ssh/id_rsa_mine" /home/me/.cache glue@x0.17x.xx.xxx:/home/glue/

I will just keep on writing here until I get a full pic & hopefully give people bread crumb trails to get this done.

[yelled1]

@tgodzik Trying this finally but getting strange errors. Can you tell me what I am doing wrong? Thank you,

~$ ~/cs launch org.scalameta:metals_2.12:0.9.0 -J -Dcoursier.cache=./output -M scala.meta.metals.DownloadDependencies tar -zcvf metals-deps.tar.gz output
Unrecognized argument: -zcvf
(so, I figured I will just download & tar.gz later)
~$ ~/cs launch org.scalameta:metals_2.12:0.9.0 -J -Dcoursier.cache=./output -M scala.meta.metals.DownloadDependencies tar zcvf metals-deps.tar.gz output
Error: only at most one app can be passed as dependency
(cdc_daily) ny11559@ec2-ny11559:~$ ~/cs launch org.scalameta:metals_2.12:0.9.0 -J -Dcoursier.cache=./output -M scala.meta.metals.DownloadDependencies
2020.06.10 17:40:09 INFO  Downloading mdoc
Exception in thread "main" coursierapi.error.SimpleResolutionError$1: Error downloading org.scalameta:mdoc_0.24:2.1.5
  not found: /home/ny11559/.ivy2/local/org.scalameta/mdoc_0.24/2.1.5/ivys/ivy.xml
  not found: https://repo1.maven.org/maven2/org/scalameta/mdoc_0.24/2.1.5/mdoc_0.24-2.1.5.pom
  not found: https://oss.sonatype.org/content/repositories/public/org/scalameta/mdoc_0.24/2.1.5/mdoc_0.24-2.1.5.pom
  not found: https://oss.sonatype.org/content/repositories/snapshots/org/scalameta/mdoc_0.24/2.1.5/mdoc_0.24-2.1.5.pom
        at coursierapi.error.SimpleResolutionError.of(SimpleResolutionError.java:11)
        at coursierapi.shaded.coursier.internal.api.ApiHelper$.simpleResError(ApiHelper.scala:326)
        at coursierapi.shaded.coursier.internal.api.ApiHelper$.doFetch(ApiHelper.scala:359)
        at coursierapi.shaded.coursier.internal.api.ApiHelper.doFetch(ApiHelper.scala)
        at coursierapi.Fetch.fetchResult(Fetch.java:244)
        at coursierapi.Fetch.fetch(Fetch.java:239)
        at scala.meta.internal.metals.Embedded$.downloadDependency(Embedded.scala:172)
        at scala.meta.internal.metals.Embedded$.downloadMdoc(Embedded.scala:193)
        at scala.meta.metals.DownloadDependencies$.$anonfun$downloadMdoc$2(DownloadDependencies.scala:47)
        at scala.collection.immutable.List.foreach(List.scala:392)
        at scala.meta.metals.DownloadDependencies$.downloadMdoc(DownloadDependencies.scala:44)
        at scala.meta.metals.DownloadDependencies$.main(DownloadDependencies.scala:26)
        at scala.meta.metals.DownloadDependencies.main(DownloadDependencies.scala)

[tgodzik]

@tgodzik Trying this finally but getting strange errors. Can you tell me what I am doing wrong? Thank you,

~$ ~/cs launch org.scalameta:metals_2.12:0.9.0 -J -Dcoursier.cache=./output -M scala.meta.metals.DownloadDependencies tar -zcvf metals-deps.tar.gz output
Unrecognized argument: -zcvf
(so, I figured I will just download & tar.gz later)
~$ ~/cs launch org.scalameta:metals_2.12:0.9.0 -J -Dcoursier.cache=./output -M scala.meta.metals.DownloadDependencies tar zcvf metals-deps.tar.gz output
Error: only at most one app can be passed as dependency
(cdc_daily) ny11559@ec2-ny11559:~$ ~/cs launch org.scalameta:metals_2.12:0.9.0 -J -Dcoursier.cache=./output -M scala.meta.metals.DownloadDependencies
2020.06.10 17:40:09 INFO  Downloading mdoc
Exception in thread "main" coursierapi.error.SimpleResolutionError$1: Error downloading org.scalameta:mdoc_0.24:2.1.5
  not found: /home/ny11559/.ivy2/local/org.scalameta/mdoc_0.24/2.1.5/ivys/ivy.xml
  not found: https://repo1.maven.org/maven2/org/scalameta/mdoc_0.24/2.1.5/mdoc_0.24-2.1.5.pom
  not found: https://oss.sonatype.org/content/repositories/public/org/scalameta/mdoc_0.24/2.1.5/mdoc_0.24-2.1.5.pom
  not found: https://oss.sonatype.org/content/repositories/snapshots/org/scalameta/mdoc_0.24/2.1.5/mdoc_0.24-2.1.5.pom
        at coursierapi.error.SimpleResolutionError.of(SimpleResolutionError.java:11)
        at coursierapi.shaded.coursier.internal.api.ApiHelper$.simpleResError(ApiHelper.scala:326)
        at coursierapi.shaded.coursier.internal.api.ApiHelper$.doFetch(ApiHelper.scala:359)
        at coursierapi.shaded.coursier.internal.api.ApiHelper.doFetch(ApiHelper.scala)
        at coursierapi.Fetch.fetchResult(Fetch.java:244)
        at coursierapi.Fetch.fetch(Fetch.java:239)
        at scala.meta.internal.metals.Embedded$.downloadDependency(Embedded.scala:172)
        at scala.meta.internal.metals.Embedded$.downloadMdoc(Embedded.scala:193)
        at scala.meta.metals.DownloadDependencies$.$anonfun$downloadMdoc$2(DownloadDependencies.scala:47)
        at scala.collection.immutable.List.foreach(List.scala:392)
        at scala.meta.metals.DownloadDependencies$.downloadMdoc(DownloadDependencies.scala:44)
        at scala.meta.metals.DownloadDependencies$.main(DownloadDependencies.scala:26)
        at scala.meta.metals.DownloadDependencies.main(DownloadDependencies.scala)

We fixed that in the current master so best to use something like: ~/cs launch -r sonatype:public org.scalameta:metals_2.12:0.9.0+201-7d0ecb66-SNAPSHOT -- -J -Dcoursier.cache=./output -M scala.meta.metals.DownloadDependencies