Network unavailable

[psvensson]

I've donlnoaded and started the latest version (TruffleSqueak-22.1.0.image) running under GraalVm for OpenJDk 17 on Linux Mint 20.2.

It starts fine and I can run som eexamples, but as soon as I try to do anything involving the network (like starting a monitcello installer), the image hangs and eventually produces a debugger where it's obvious that the result was nil.

[fniephaus]

Hi @psvensson, thanks for raising this. Unfortunately, TruffleSqueak's implementation of the SocketPlugin is currently incomplete and still has a few issues, so it is not surprising that some networking tasks fail. The problem is that we're emulating what the SocketPlugin does using Java NIO and that's not always working.

Ultimately, it would be nice to implement the SocketPlugin based on a POSIX backend, similar to GraalPython (https://github.com/hpi-swa/trufflesqueak/issues/39#issuecomment-1134566709).

[psvensson]

Hi Fabio, thanks for the quick reply. OK, but then it is a known issue and you also know where it can be fixed. I know a bit of Smalltalk and a bit of Java, but not much about GraalVM or indeed truffleSqueak. I will probably fail but will see what I can do :)

I do not know Python so would start to see if I could find a good enough solution fixing existing bugs / implementing what needs to implemented.

I assume that I start looking here; https://github.com/hpi-swa/trufflesqueak/tree/main/src/de.hpi.swa.trufflesqueak/src/de/hpi/swa/trufflesqueak/nodes/plugins/network

Do you have any more hints or ideas on how to go about this, obvious shortlist of what is not working or something like that?

[fniephaus]

Yes, it is a known issue. It just never was pressing enough to fix because, well, you can do networking via interop with Ruby or other programming languages. :wink:

I don't think you need to know much about Python. GraalPython is, just like TruffleSqueak, implemented in Java using Truffle. What you could do is copy over the PosixSupportLibrary and its implementations (e.g., NFIPosixSupport and/or EmulatedPosixSupport) and re-implement the SocketPlugin primitives.

An example: primitiveResolverGetAddressInfo is currently not implemented, I think, and uses getaddrinfo under the hood in the OpenSmalltalkVM (here's primitiveResolverGetAddressInfo and here's the getaddrinfo() call in the Unix impl.). So in TruffleSqueak, the primitive could be implemented using getaddrinfo from the POSIX backend. If you have troubles doing that, you could copy over just the getaddrinfo impl from the EmulatedPosixBackend and use that to implement primitiveResolverGetAddressInfo.

One more tip: If you select the TruffleSqueak test image the first time you start TruffleSqueak, you can also look at the original Slang sources of the SocketPlugin, which may be helpful as well.

I don't want to discourage you in any way but I think it's fair to warn you that this is not really a beginner project. Please feel free to give it a shot and asked questions!

[psvensson]

Yes, of course. But it would be more easy for first-time users to come to what otherwise works as a normal Squeak environment, and then learn a bit at a time from there.

Thank you! This is much more info that I had hoped for. I am not a beginner, really. I rage-quite Java in 2005 and reacquainted myself with Smalltalk since 2015 or so, and have done quite a bit of network programming (although not often this low-level). I did teach Networking Technologies in most of the 90s as well. I might survive. We'll see :) .

I will definitely ask. By chance my vacation start tomorrow also, so good winds ahead..

[psvensson]

Just thought I could post my proceedings here, for reference.

I have been bale to compile from command-line both GraalVM and truffleSqueak according to instructions, using the proper JVM downloaded by mx.

It then did prove possible to connect eclipse as a remote debugger (using an already defined target for this - thanks) to connect to a truffleSqueak instance using according to the development instructions. I then set a lot of breakpoints in SqueakSocket.java and SqueakTCPSocket.java.

It turns out that name resolution works well, I'm able to set breakpoints in SqueackTCPSocket::receiveDataFrom where an address is present.

But there things are stuck and I'm trying to figure out why. It feels like receive never returns, and then eventually it times out and truffleSqueak gets back nil and throws an exception. The odd hting is that I can see that data is received, and follow it a couple of times. I'm still trying to get to grips with Eclipse deugging again, but I'm fairly sure that something comes across.

[psvensson]

Hmm, I might juts as well have started properly inside truffleSqueak, though, as when I look more closely, it does actually receive data, the main problem seems to be that the connectiontimes out before data is fully read, or that it thinks so.

[fniephaus]

Great that you're able to dive into networking in TruffleSqueak!

I remember that there are some issues with certain websites. I just had a quick look and the non-SSL version of the website seems to work fine (WebClient httpGet: 'http://dn.se'). Maybe this has also something to do with the SqueakSSL plugin?

In case you only need to send some simple http requests, you can also implement a subclass of WebClient using Java, Python, Ruby, or some other GraalVM language.

[psvensson]

Ahaa, good catch! yes, this seems to be just about https indeed. It might be SqueakSSL, but I also saw some comments in the SecureSocketStream, which says:

"Note: The loop here is necessary to catch cases where a TLS packet is split among TCP packets. In this case we would pull the first portion of the TLS packet here but receiveAvailableData would return nothing since the contents of the packet can't be decoded until the rest has come in."

So I'll check that out first. It feels like that for SSL, something thinks there is data to be read when it really isn't any, so I hpoe to narrow down where soon.

And yes, I could call out to node.js for example, but my goal is to see if it would be possible to lift truffleSqueak to a level where it's more or less as usable as the standard distribution, but with ultra-mega-superpowers. So I'll continue to weld here in the cellar for a while longer :)

[psvensson]

Silly question, but what is the easiest way to get some logging output from the Java/vm code? I tried something like this;

protected final boolean isDataAvailable() throws IOException {
        selector.selectNow();
        final Set<SelectionKey> keys = selector.selectedKeys();
        System.out.println("isDataAvailable keys length = " + keys.size());
        for (final SelectionKey key : keys) {
            if (key.isReadable()) {
                LogUtils.SOCKET.finer(() -> this + " data available");
                return true;
            }
        }

        LogUtils.SOCKET.finer(() -> this + " no data available");
        return false;
    }

And checked bout stdout and the console tab in Eclipse so of course nothing showed (but it does get called, I can set a breakpoint there, for example).

Since data transport function get called many times, debugging doesn't give the whole picture I need to see where things go awry, so I do need a way to print things over time. In Squeak land Transcript works well as usual, of course.

There are also logging options to set for the vm when starting it, but I don't understand what to write here; From the help:

--log.[logger].level= Set language log level to OFF, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST or ALL

What are possible values of [logger] here?

[fniephaus]

Not a silly question at all, looks like we haven't documented this yet. You've seen the list of loggers, and the way you use enable is like this:

# enable only the socket logger on the FINER level
trufflesqueak --log.smalltalk.socket.level=FINER
# enable all loggers at the same time on the ALL level (this is very verbose)
trufflesqueak --log.smalltalk.level=ALL

[psvensson]

Thanks! OK, hmm. I do not see any output anywhere when I do that (even with the ALL option).

Also, I tried to do --log.java.level=ALL but nothing happened with that either. It sounded like it should work and I am 100% for this, but more magic is needed, apparently

[psvensson]

For the ongoing research I'm now fairly convinced that the culprit is the SqueakSocket::isDataAvailable method in teh plugin, which I think does not flag that data is available when it is.

[fniephaus]

Can you check that --log.smalltalk.primitives.level=ALL or some other logger works for you? Are you sure you're running the code that you are seeing? It's sometimes easy to forget to rebuild with mx.

BTW, I recommend something like this:

cd trufflesqueak
mx --dy /compiler build # build TruffleSqueak and Graal compiler
../graal/sdk/latest_graalvm_home/bin/trufflesqueak # always points to the latest TruffleSqueak build
# change TruffleSqueak code...
mx --dy /compiler build # should rebuild only TruffleSqueak
../graal/sdk/latest_graalvm_home/bin/trufflesqueak --log.smalltalk.socket.level=FINER

Bonus points for setting $GRAALVM_HOME to your /path/to/graal/sdk/latest_graalvm_home, which allows you to use this instead:

$GRAALVM_HOME/bin/trufflesqueak --log.smalltalk.socket.level=FINER

[fniephaus]

and this would let you run with the Java debugger:

$GRAALVM_HOME/bin/trufflesqueak --vm.Xrunjdwp:transport=dt_socket,server=y,address=8000,suspend=y

[psvensson]

Aha! Wow, of course. I had all the time happily just started trufflesqueak without specifying which one, so of course I ran the earlier installation of trufflesqueak in my classpath from when I downloaded it while trying things out.

Now, when I correctly start using the following line;

$GRAALVM_HOME/bin/trufflesqueak --vm.Xrunjdwp:transport=dt_socket,server=y,address=8000,suspend=y ../../../Installs/graalvm-ce-java17-22.1.0/languages/smalltalk/resources/TruffleSqueakTestImage-6.0alpha-20288b-64bit.image --log.smalltalk.socket.level=FINER

I do see a lot of logging, including my System.out.println. Thank you, the car is moving again :)

[psvensson]

However, now that am using the correct and locally built truffleSqueak, I am starting to see a lot of DNUs for getLoopAndCount .

They appear after I've get the usual exception on not being able to go to the https site, and after that appear every time I try to do anything with a calltarget browser (like looking at any code, so I then need to restart again. Any ideas what it might be?

[fniephaus]

It seems like you're using the CallTargetBrowser without opening the right modules for allow for introspection. Note that this also only works when running with the Graal compiler.

If you don't need to use the CallTargetBrowser, I suggest you switch back to the standard Browser via, for example, SystemBrowser askForDefault and then pick "Browser".

[psvensson]

Thanks a lot for the help

[psvensson]

I have now run into another problem which is very confusing. I merely made the suggested changes (and changed browser), saved the image and then restarted. Now, after I start truffleSqueak and then tries to attach the Eclipse debugger, I get this;

$ $GRAALVM_HOME/bin/trufflesqueak --vm.Xrunjdwp:transport=dt_socket,server=y,address=8000,suspend=y ../../../Installs/graalvm-ce-java17-22.1.0/languages/smalltalk/resources/TruffleSqueakTestImage-6.0alpha-20288b-64bit.image --log.smalltalk.level=FINER
WARNING: Unknown module: jdk.internal.vm.compiler specified to --add-opens
Listening for transport dt_socket at address: 8000
[To redirect Truffle log output to a file use one of the following options:
* '--log.file=<path>' if the option is passed using a guest language launcher.
* '-Dpolyglot.log.file=<path>' if the option is passed using the host Java launcher.
* Configure logging using the polyglot embedding API.]
[engine] WARNING: The polyglot context is using an implementation that does not support runtime compilation.
The guest application code will therefore be executed in interpreted mode only.
Execution only in interpreted mode will strongly impact the guest application performance.
For more information on using GraalVM see https://www.graalvm.org/java/quickstart/.
To disable this warning the '--engine.WarnInterpreterOnly=false' option or use the '-Dpolyglot.engine.WarnInterpreterOnly=false' system property.
[trufflesqueak] Running TruffleSqueakTestImage-6.0alpha-20288b-64bit.image on Interpreted...
ERROR: java.lang.NullPointerException: Cannot invoke "de.hpi.swa.trufflesqueak.image.SqueakImageChunk.setObject(Object)" because "specialChunk" is null
org.graalvm.polyglot.PolyglotException: java.lang.NullPointerException: Cannot invoke "de.hpi.swa.trufflesqueak.image.SqueakImageChunk.setObject(Object)" because "specialChunk" is null
    at de.hpi.swa.trufflesqueak.image.SqueakImageReader.initPrebuiltConstant(SqueakImageReader.java:303)
    at de.hpi.swa.trufflesqueak.image.SqueakImageReader.initObjects(SqueakImageReader.java:354)
    at de.hpi.swa.trufflesqueak.image.SqueakImageReader.run(SqueakImageReader.java:90)
    at de.hpi.swa.trufflesqueak.image.SqueakImageReader.load(SqueakImageReader.java:75)
    at de.hpi.swa.trufflesqueak.SqueakImage$SqueakImageNode.execute(SqueakImage.java:40)
    at <smalltalk> null(Unknown)
    at org.graalvm.sdk/org.graalvm.polyglot.Context.eval(Context.java:399)
    at de.hpi.swa.trufflesqueak.launcher.TruffleSqueakLauncher.execute(TruffleSqueakLauncher.java:126)
    at de.hpi.swa.trufflesqueak.launcher.TruffleSqueakLauncher.launch(TruffleSqueakLauncher.java:82)
    at org.graalvm.launcher.AbstractLanguageLauncher.launch(AbstractLanguageLauncher.java:296)
    at org.graalvm.launcher.AbstractLanguageLauncher.launch(AbstractLanguageLauncher.java:121)
    at org.graalvm.launcher.AbstractLanguageLauncher.runLauncher(AbstractLanguageLauncher.java:168)
Original Internal Error: 
java.lang.NullPointerException: Cannot invoke "de.hpi.swa.trufflesqueak.image.SqueakImageChunk.setObject(Object)" because "specialChunk" is null
    at de.hpi.swa.trufflesqueak.image.SqueakImageReader.initPrebuiltConstant(SqueakImageReader.java:303)
    at de.hpi.swa.trufflesqueak.image.SqueakImageReader.initObjects(SqueakImageReader.java:354)
    at de.hpi.swa.trufflesqueak.image.SqueakImageReader.run(SqueakImageReader.java:90)
    at de.hpi.swa.trufflesqueak.image.SqueakImageReader.load(SqueakImageReader.java:75)
    at de.hpi.swa.trufflesqueak.SqueakImage$SqueakImageNode.execute(SqueakImage.java:40)
    at org.graalvm.truffle/com.oracle.truffle.api.impl.DefaultCallTarget.callDirectOrIndirect(DefaultCallTarget.java:85)
    at org.graalvm.truffle/com.oracle.truffle.api.impl.DefaultCallTarget.call(DefaultCallTarget.java:102)
    at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotContextImpl.eval(PolyglotContextImpl.java:1326)
    at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotContextDispatch.eval(PolyglotContextDispatch.java:63)
    at org.graalvm.sdk/org.graalvm.polyglot.Context.eval(Context.java:399)
    at de.hpi.swa.trufflesqueak.launcher.TruffleSqueakLauncher.execute(TruffleSqueakLauncher.java:126)
    at de.hpi.swa.trufflesqueak.launcher.TruffleSqueakLauncher.launch(TruffleSqueakLauncher.java:82)
    at org.graalvm.launcher.AbstractLanguageLauncher.launch(AbstractLanguageLauncher.java:296)
    at org.graalvm.launcher.AbstractLanguageLauncher.launch(AbstractLanguageLauncher.java:121)
    at org.graalvm.launcher.AbstractLanguageLauncher.runLauncher(AbstractLanguageLauncher.java:168)
Caused by: Attached Guest Language Frames (1)

I have tried to rebuild the truffleSqueak vm and then all of it, but to no avail. I then thought that the changes file of the image might have been damaged (maybe I terminated it too soon?) so I removed it and tried again, but got the same thing again.

It is very strange, I have not had time to do anything in particular (what I can remember). Is this something you have come across before?

As a last resort I'll download everything again and begin from scratch.

[fniephaus]

It seems the image has been damanged somehow. I believe the first positional argument is the image and all following are considered image arguments. So to avoid confusion, I'd either move the log option before the image file or drop the image argument. If no image is provided, TruffleSqueak will automatically use the one in the resources directory.

To get a fresh test image, simply empty "Installs/graalvm-ce-java17-22.1.0/languages/smalltalk/resources/" and run TruffleSqueak again. It should asked you again to select an image.

[psvensson]

Thank you, that did the trick.

[psvensson]

Some observations while spelunking the code;

1. When rebuilding the truffleSqueak VM, the downloaded image gets (understandable) deleted and must be downloaded anew.
2. It is only the 'test' image (option 2) which has logging enabled, so trying to set logging to anything at all while using (for example) the recommended image does not work.
3. Copying a downloaded squeak image to a safe location and using the when starting truffleSqueak for debugging is not quite enough, since it then complains that it cannot find its sources

This just in case anyone else stumbles on the same things in the future :)

The Graal socket python code is not like the smalltalk one, so they don't compare directly. But it's very little code in both cases. The problem arises when in the SecureSocketStream tries to read further from the socket but is blocked until timeout. It reads initial data but then in a subsequent step just sits there.

It has to be an assumption on what the underlying layer should do, or not, but it is not very obvious (yet) what it is. I'm reading up on java.nio and getting my bearings.

Will be travelling for a couple of days now, so getting back to this later in the week.

[psvensson]

So, while waiting for packing to be done, I noticed something interesting. When using the WebClient to go directly to 'https://dn.se' it reads a bit then times out, but when going to the basic vanilla http url 'http://dn.se', I notice that it first reads from port 80 but then apparently gets redirected to https --- and that works.

Squeak SecureSocketsStream socket vm plugin log:
---------------------------------------------------

isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:51928 remote=/52.29.209.1:443], selector=sun.nio.ch.EPollSelectorImpl@57b75756, interestOps=13, readyOps=5 isReadable = true
PrimSocketReceiveDataBufCountNode::receiveData vanilla called
receiveData key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:51928 remote=/52.29.209.1:443], selector=sun.nio.ch.EPollSelectorImpl@57b75756, interestOps=13, readyOps=5 isReadable = true
receiveData received 4096 bytes
PrimSocketReceiveDataBufCountNode::doCount finalCount = 4096
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:51928 remote=/52.29.209.1:443], selector=sun.nio.ch.EPollSelectorImpl@57b75756, interestOps=13, readyOps=5 isReadable = true
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:51928 remote=/52.29.209.1:443], selector=sun.nio.ch.EPollSelectorImpl@57b75756, interestOps=13, readyOps=5 isReadable = true
PrimSocketReceiveDataBufCountNode::receiveData vanilla called
receiveData key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:51928 remote=/52.29.209.1:443], selector=sun.nio.ch.EPollSelectorImpl@57b75756, interestOps=13, readyOps=5 isReadable = true
receiveData received 300 bytes
PrimSocketReceiveDataBufCountNode::doCount finalCount = 300
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:51928 remote=/52.29.209.1:443], selector=sun.nio.ch.EPollSelectorImpl@57b75756, interestOps=13, readyOps=4 isReadable = false
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:51928 remote=/52.29.209.1:443], selector=sun.nio.ch.EPollSelectorImpl@57b75756, interestOps=13, readyOps=4 isReadable = false
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:51928 remote=/52.29.209.1:443], selector=sun.nio.ch.EPollSelectorImpl@57b75756, interestOps=13, readyOps=4 isReadable = false
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:51928 remote=/52.29.209.1:443], selector=sun.nio.ch.EPollSelectorImpl@57b75756, interestOps=13, readyOps=4 isReadable = false
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:51928 remote=/52.29.209.1:443], selector=sun.nio.ch.EPollSelectorImpl@57b75756, interestOps=13, readyOps=4 isReadable = false

SocketStream sock vm plugin log:
--------------------------------
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:60030 remote=/52.29.209.1:80], selector=sun.nio.ch.EPollSelectorImpl@58a63629, interestOps=13, readyOps=4 isReadable = false
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:60030 remote=/52.29.209.1:80], selector=sun.nio.ch.EPollSelectorImpl@58a63629, interestOps=13, readyOps=5 isReadable = true
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:60030 remote=/52.29.209.1:80], selector=sun.nio.ch.EPollSelectorImpl@58a63629, interestOps=13, readyOps=5 isReadable = true
PrimSocketReceiveDataBufCountNode::receiveData vanilla called
receiveData key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:60030 remote=/52.29.209.1:80], selector=sun.nio.ch.EPollSelectorImpl@58a63629, interestOps=13, readyOps=5 isReadable = true
receiveData received 112 bytes
PrimSocketReceiveDataBufCountNode::doCount finalCount = 112
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:60030 remote=/52.29.209.1:80], selector=sun.nio.ch.EPollSelectorImpl@58a63629, interestOps=13, readyOps=4 isReadable = false
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:40666 remote=/2.19.113.133:443], selector=sun.nio.ch.EPollSelectorImpl@680d4a6a, interestOps=13, readyOps=5 isReadable = true
PrimSocketReceiveDataBufCountNode::receiveData vanilla called
receiveData key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:40666 remote=/2.19.113.133:443], selector=sun.nio.ch.EPollSelectorImpl@680d4a6a, interestOps=13, readyOps=5 isReadable = true
receiveData received 3608 bytes
PrimSocketReceiveDataBufCountNode::doCount finalCount = 3608
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:40666 remote=/2.19.113.133:443], selector=sun.nio.ch.EPollSelectorImpl@680d4a6a, interestOps=13, readyOps=4 isReadable = false
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:40666 remote=/2.19.113.133:443], selector=sun.nio.ch.EPollSelectorImpl@680d4a6a, interestOps=13, readyOps=5 isReadable = true
PrimSocketReceiveDataBufCountNode::receiveData vanilla called
receiveData key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:40666 remote=/2.19.113.133:443], selector=sun.nio.ch.EPollSelectorImpl@680d4a6a, interestOps=13, readyOps=5 isReadable = true
receiveData received 242 bytes
PrimSocketReceiveDataBufCountNode::doCount finalCount = 242
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:40666 remote=/2.19.113.133:443], selector=sun.nio.ch.EPollSelectorImpl@680d4a6a, interestOps=13, readyOps=4 isReadable = false
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:40666 remote=/2.19.113.133:443], selector=sun.nio.ch.EPollSelectorImpl@680d4a6a, interestOps=13, readyOps=4 isReadable = false
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:40666 remote=/2.19.113.133:443], selector=sun.nio.ch.EPollSelectorImpl@680d4a6a, interestOps=13, readyOps=5 isReadable = true
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:40666 remote=/2.19.113.133:443], selector=sun.nio.ch.EPollSelectorImpl@680d4a6a, interestOps=13, readyOps=5 isReadable = true
PrimSocketReceiveDataBufCountNode::receiveData vanilla called
receiveData key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:40666 remote=/2.19.113.133:443], selector=sun.nio.ch.EPollSelectorImpl@680d4a6a, interestOps=13, readyOps=5 isReadable = true
receiveData received 4096 bytes
PrimSocketReceiveDataBufCountNode::doCount finalCount = 4096
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:40666 remote=/2.19.113.133:443], selector=sun.nio.ch.EPollSelectorImpl@680d4a6a, interestOps=13, readyOps=5 isReadable = true
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:40666 remote=/2.19.113.133:443], selector=sun.nio.ch.EPollSelectorImpl@680d4a6a, interestOps=13, readyOps=5 isReadable = true
PrimSocketReceiveDataBufCountNode::receiveData vanilla called
receiveData key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:40666 remote=/2.19.113.133:443], selector=sun.nio.ch.EPollSelectorImpl@680d4a6a, interestOps=13, readyOps=5 isReadable = true
receiveData received 4096 bytes
PrimSocketReceiveDataBufCountNode::doCount finalCount = 4096
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:40666 remote=/2.19.113.133:443], selector=sun.nio.ch.EPollSelectorImpl@680d4a6a, interestOps=13, readyOps=5 isReadable = true
isDataAvailable key = channel=java.nio.channels.SocketChannel[connected local=/192.168.86.207:40666 remote=/2.19.113.133:443], selector=sun.nio.ch.EPollSelectorImpl@680d4a6a, interestOps=13, readyOps=5 isReadable = true
.... etc

That is super-weird though, so it means that something special happens when connecting initially using TLS. Oh well, vacation it is, then.

[fniephaus]

Good find, that is indeed interesting! And enjoy your vacation! I'm currently occupied with some other things anyway until the end of next week, but it'd be great to fix this at some point and happy to help as much as my time allows.

[psvensson]

Since the truffleSqueak SockePlugin obviously works in some cases, I have been trying to find the difference in handling from the SqueakSSL side between going directly to https and being redirected to https. I had several great theories, but none stuck unfortunately.

I now tried other sites than the one I've been testing, and interesting things happens.

vanilla http request WebClient httpGet: 'http://google.com' fails silently with a Transcript complaint of 'Unknown cookie field: SameSite' after redirecting from google.com to www.google.com.

On the other had, going directly to https to github; WebClient httpGet: 'https://github.com' works!

The only reason I can think of is that it is the WebClient which is wonky.

Trying to update Squeak fails happily as usual, trying to access http://source.squeak.org/trunk which is just http:// and accessible through a browser.

it might be that it is the WebClient which is not correctly hotted-up to dance the latest dances? However, that can't be true since it does actually work in a vanilla VM squeak 6.0.

Very X-files right now :)

[fniephaus]

TruffleSqueak's image and test image are based on a fixed Squeak trunk image that we haven't updated in a while. So maybe worth doing these experiments in a Squeak 6.0 release image since WebClient might have seen some updates?

[psvensson]

Yes, that is a good idea. Actually, I was wondering how to create new truffleSqueak images and a bit about how one would go about to write the Smalltalk parts (if needed) to get truffleSqueak to work with another Smalltalk (I was thinking Pharo, for example). If you could add some notes on that to the docs that would be great. And then I could also help out more

[psvensson]

Hmm, I have also been trying to find the 'gu' utility in the graalvm directory in the trufflesqueak-build folder I created, but I can't seem to find it.

In the pre-built graalvm installation it can be found in bin/gu which is a link to lib/installer/bin/gu but nothing can be found in the mx built graalvm folder in the same places.

I have then tried to understand if there are any specific mx build command that can be used to build and 'install' the gu command, but I don't see ny there either. To be fair, I've never used mx before so I'm mostly plodding along semi-randomly.

If there is an easy way to to this, it would be great, because then I can start installing node.js and see if I can install packages (which was my next step anyway)

[fniephaus]

Yes, that is a good idea. Actually, I was wondering how to create new truffleSqueak images and a bit about how one would go about to write the Smalltalk parts (if needed) to get truffleSqueak to work with another Smalltalk (I was thinking Pharo, for example). If you could add some notes on that to the docs that would be great. And then I could also help out more

Here's how the TruffleSqueak images are created. Maybe it works without any problems with the Squeak 6.0 release, I don't know but it is something we need to look into now that there's a new release.

Regarding Pharo support, it's blocked by at least two problems:

• They have changed object layouts, so loading the image doesn't work atm. TruffleSqueak's object model needs to be extended to accommodate that.
• Pharo does drawing and windowing through FFI. TruffleSqueak's FFI needs to be finished, only very few things are working. I'm afraid that alone may be more work than fixing the SocketPlugin.

[psvensson]

Thank you, of course i was already documented, sry :) I'll give it a try and see if anything have changed in WebClient to the better. Also, I will of course ask in the GraalVM channels on how to install gu - still mixing projects, it seems

And I can't say I'm surprised that Pharo is a tougher nut to crack. Sometimes it feels like the whole point of something is to be incompatible, when possible. But it is much easier to work with out of the box (code completion, searching for anything, et.c.) Maybe I can find something done for Squeak too, I'll look around.

[psvensson]

The first parts of the setup image instructions went well, but when executing the last 'setupImage' part,

(Smalltalk at: #TruffleSqueakUtilities) setUpImage.

I get this exception

More specifically it is this that fials;

[fniephaus]

Hmm, I have also been trying to find the 'gu' utility in the graalvm directory in the trufflesqueak-build folder I created, but I can't seem to find it.

You're just building TruffleSqueak plus the base GraalVM JDK, so gu isn't there. mx is a dedicated build tool for GraalVM, so there's not really a way you could've used it in the past. Have a look at its README.md for some general questions like setting up IDE configs. For building GraalVMs, take a look at this. The gu impl actually lives inside this directory of the graal repo, so you could do this:

cd ../graal/vm
mx --dy trufflesqueak,/compiler build

which builds a GraalVM + compiler + TruffleSqueak + gu. Or this:

cd ../graal/vm
mx --dy trufflesqueak,/graal-nodejs,/compiler build

to build a GraalVM + compiler + TruffleSqueak + GraalVM's Node.js runtime + gu. (note that you need to check out the graaljs repo next to your trufflesqueak/ and graal/ checkouts for this to work).

However, I'd suggest you do this instead:

• Download a recent GraalVM dev build
• Build TruffleSqueak via mx --env trufflesqueak-jvm build
• Run mx --env trufflesqueak-jvm paths SMALLTALK_INSTALLABLE_JAVA11 to find the path to the installable
• Install it via $GRAALVM_HOME/bin/gu install -f -L path/to/the/installable.jar
• Run $GRAALVM_HOME/bin/trufflesqueak ...

This way, you can install any other language (e.g., gu install nodejs) without having to build any of this from source, while trying out a TruffleSqueak dev build.

[fniephaus]

Thank you, of course i was already documented, sry :) I'll give it a try and see if anything have changed in WebClient to the better. Also, I will of course ask in the GraalVM channels on how to install gu - still mixing projects, it seems

There's also a #trufflesqueak channel in the GraalVM Slack where you can reach me.

The first parts of the setup image instructions went well, but when executing the last 'setupImage' part, (Smalltalk at: #TruffleSqueakUtilities) setUpImage. I get this exception

I'm not surprised, but this is not too much of a problem because it's setting up some workspace or something. Try running the TruffleSqueak tests, that'd be interesting.

[psvensson]

This is excellent stuff! Could you please add this to the existing documentation, I think it would be really helpful to have it all in one place (and so one can search for it also).

OK, now I need to make a choice hmmm :)

[fniephaus]

I hope it's not too much to ask, but it'd be nice if you could contribute some extensions for the docs via a PR. That way, I can also learn and comment on how you use things. Otherwise, maybe open an issue for tracking this so I don't forget and do it when I have some more time?

[psvensson]

NP, I'll make a small PR of an edited version of your text, to the development documentation

[psvensson]

So, hmm. I have had a bout of Covid so been under the ice for most of the week. But today I managed to fix the trufflesqueak setup utilities to work with the actual squeak 6.0 release. They have made several changes that are probably nice but completely ignores backward compatibility. It almost feels like a Pharo core dev have been working with this :) .

This fix might not be ideal, but I crated a PR for the 'image' branch and we can crate a new issue if/when you want to have any changes done to it.

Anyhow, after I got the setup working, I again tried to do the very first WebClient test that is the reason for this issue in the first place -- and it worked. Of course it now worked.

So the problem was indeed some weird bug in the WebClient handling of SqueakSSL which was fixed between the release used in the TruffleSqueak documentation and the official release - I don't even want to go back and find out what it was, but now it runs and purrs.

[fniephaus]

So, hmm. I have had a bout of Covid so been under the ice for most of the week.

Oh, I hope you are better now! Thanks for the PR!

Of course it now worked.

That's great news, thanks for checking! According to the Squeak/Smalltalk 6.0 release notes, WebClient has received some bug fixes and improvements... always good if the problem turned out to be fixed somewhere else. :)

We'll upgrade to the 6.0 release image very soon (after the upcoming 22.2 release), but that will mean that the new 6.0-based image will ship with TruffleSqueak 22.3 in a couple of months (we'll need some time to make sure everything's working ok).

[psvensson]

I am, thanks! Yes, this has been a bit of a surprising ride, but it all ended well. Please let me know if there is anything else you need from me to close this ticket. And also looking forward to any changes need to the image PR. I haven't looked at everything so I don't feel I have the full picture

[fniephaus]

Good! I'm keeping this open until we've adopted the 6.0 release. The most work is going to be updating the test map in test.properties. testTestMapConsistency is the test for this. I'm sure there are many new tests and some test removals. And then, we need to figure out which ones work and which ones don't.

[psvensson]

OK, sounds good! I'll lick my wounds a bit until the next release then. I also had some problems using a recent dev graalVm , this dev truffleSqueak and adding node.js. Also I could not figure out a way to list the installed languages in truffleSqueak in a printable format. All I got was just remote objects. I suspect this is just a bit of missing docs (or me not finding it) but we can take that later then. Very glad we got this far.

[fniephaus]

Sure, let me know in case you run into problems and don't know how to proceed. For language names, you can, for example, do this: Polyglot availableLanguages collect: [ :ea | ea getName asString ].

[psvensson]

OK, I will try this, thx

[psvensson]

So, this is embarrassing. I have had fevers on and off and it's sometimes easy to mix things up.

The image I tested was indeed the latest 6.0 release - but it was running on a standard VM. I had just started it some days earlier trying to debug the trufflesqueak image setup process.

So o course the (now also fixed) WebClient worked when I tried it.

Now I saved the image and restarted it under my latest GraalVM/Trufflesqueak installation, and it sadly show the same problems as before - unable to read more than an initial amount from a https url, but works well when using http which redirects to https.

[fniephaus]

I have had fevers on and off and it's sometimes easy to mix things up.

I hope you are feeling better now!

it sadly show the same problems

Ok, so this issue is essential still a problem?

[psvensson]

A little bit, thank you.

Yes, it seems so, unfortunately.

I will try out Cuis as well (and Maybe Pharo) to see if I can get something out of that. Also, I'll start checking out how the open smalltalk VM implements things, in case something pops out.

[psvensson]

Cuis seems to have the same issue, but behaves a little bit differently. It doesn't time out after 45s but instead throws an exception when the remote side closes the connection.

I am trying to understand what needs to be installed in Pharo to make it work over truffleSqueak but I'm stuck trying to figure out how the Cuis image setup works and what it is doing; https://github.com/hpi-swa/trufflesqueak/blob/main/docs/images.md#trufflesqueak-image-creation

I did look quickly at the squeak baseline for trufflesqueak and that of course does not work with Pharo, but perhaps it would be possible to modify it a little bit so it can detect Pharo and do something else instead.

[fniephaus]

Interesting, maybe the SqueakSSL plugin in TruffleSqueak doesn't claim to be done early enough and leaves the connection open.

As mentioned in https://github.com/hpi-swa/trufflesqueak/issues/170#issuecomment-1184450457, Pharo isn't as easy to support as Cuis and probably requires significantly more work than needed to fix the SqueakSSL plugin.

[psvensson]

Yes, that could be the case.

Hmm, yes. You know that much more than I do of course, but it seems to be the need for filesystem access in the editor and should be not too difficult if so.

The reason Pharo is interesting is that is has its very own web client, as part of the Zinc Web server/client package by Sven von Caekenberg, so it would have provided an excellent triangulation vs. the Squeak and Cuis WebClient.

But OK, I'll start comparing the open smalltalk vm SSL plugin and see what I can find. Will probably be some time.

[fniephaus]

So I had some time to look into this myself. I logged a couple of things inside the image and found out that the connect primitive never returns -1, which signals that "More input is required". So I digged into the SqueakSSL code and found where it is supposed to ask for more input.

With https://github.com/hpi-swa/trufflesqueak/commit/f44be51cbe11df2451d8d94c579aeb93c84fd8a9, WebClient httpGet: 'https://dn.se' and WebClient httpGet: 'https://google.com' now work for me. However, I think some of the SqueakSSLTest cases are failing now, maybe they never really succeeded. I don't know yet...