Network Support

[konsumer]

This is related to #12

It looks like lovejs is trying to make a websocket connection to the same host/port as UDP in native code. It would be cool if I could use regular websockets in both, so I can use 1 server, and same love code in both (even if it's a wrapper.) Is there a way for me to inject a compatibility layer on wasm-build that uses real websockets if it's web, and love-ws otherwise? I know UDP won't work on web, but if I could swap out native vs wasm websockets, it seems like it should work.

Here is how to reproduce, currently:

I wrote a python UDP socket server:

#!/usr/bin/env python3

import socket
import sys
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server_address = ('localhost', 12345)
print('starting up on {} port {}'.format(*server_address))
sock.bind(server_address)

while True:
    print('\nwaiting to receive message')
    data, address = sock.recvfrom(4096)

    print('received {} bytes from {}'.format(len(data), address))
    print(data)

    if data:
        sent = sock.sendto(data, address)
        print('sent {} bytes back to {}'.format(sent, address))

If I do this in a love app:

local socket = require("socket")

local udp

function love.load()
  udp = socket.udp()
  udp:settimeout(0) -- 0 seems to not be working right to make it non-blocking
  udp:setpeername('127.0.0.1', 12345)
  udp:send("pakemon says sup")
end

function love.update(dt)
  repeat
    data, err = udp:receive()
    if data then
      print("Received from server: " .. data)
    end
  until not data
end

function love.draw()
  love.graphics.setColor( 1, 1, 1, 1 )
  love.graphics.printf("Check your socket-server.", 0,100, 320, "center")
end

On native, it works, on web I get love.js:9 WebSocket connection to 'ws://127.0.0.1:12345/' failed. which makes me think it's trying with websockets, at least.

[konsumer]

Reading here it seems like I can use similar to websockify and it might work fine, but I'd prefer to not have to setup & maintain a proxy, somewhere, if possible, and just connect directly. Even still it tries to use same host/address which would be wrong. Is there a way to detect I am in wasm or native? I could swap the address out, and that might be workable, and rather than websockify, I could give it another port and just run a websocket server there.

[konsumer]

Sorry for the noise. I can add Module[‘websocket’][‘url’] in the browser, and it will use that instead, so I just need 2 ports, one for real sockets, and one for websockets. I will close this.

[Davidobot]

So just to clarify, did you get it working with the Module[‘websocket’][‘url’] fix and what exactly did you do?

Sorry, this is just really useful for documentation and I can merge some changes into mainline too.

EDIT: To target WebSockets specifically, I really ought to link them at compilation time: https://emscripten.org/docs/porting/networking.html

Would be interested in hearing whether you got it working!

[konsumer]

I got it requesting my other port, as the docs say, but I will need to play with the websocket server part, I think, as I couldn't get that working.

[konsumer]

To target WebSockets specifically, I really ought to link them at compilation time:

I totally agree. I would way prefer real websockets in both, instead of cramming regular sockets into websockets, in a compat layer. It will be an extra thing for native love users to compile & install, but they would need to anyway for websockets, and the code would actually represent the real thing it's doing (websockets.)

I will make a few minimal demo projects:

• UDP as I have now, try to figure out why the websocket is erroring a bunch. I will change ports in web-version, so I can try to get the websocket-proxy thing working (maybe in docker)
• TCP socket could maybe improve it? I will also change ports in web version
• love-ws so we have a tester for working on what I think is the more optimal solution

All 3 could be good for the lovejs docs, if I get them working. the first 2 have the feature of not needing to install love-ws (just use love's luasocket) and so examples would be good, I think. 3rd solution has the bonus of being clearer all the way through (like in love native you know you are using websockets) and the server will be simpler (not 2 ports open, translating one to the other, just 1 regular websocket server.)

It seems to me that if it's possible to detect "they are trying to use love-ws here" and swap out the code that gets compiled, it should work best, but I really have no idea how to do it (not super-familiar with emscripten or reading C.)

[konsumer]

I started work over here and got the UDP local all setup and everything scaffolded for the rest. It goes kinda slow, and I still need to work out UDP ws proxy, getting TCP to work local (and via ws proxy) and also the ws local code. I will report back when I have more done. If I am reading the emscripten docs right, it should work as-is via websocket proxies (albeit a bit slow.) The native love-ws stuff is a pain to install on my local machine, but I will try to figure it out. Maybe it's not the best choice, but I'm not sure what is.

[konsumer]

Should we reopen this issue? I guess it's not really resolved, and then we can track progress & ideas on networking stuff better.

[konsumer]

Also, I am seeing errors I didn't see before, when using makelove. I'm not sure what the difference is, but before I was just getting websocket errros, now I get this, which looks related to threads:

[konsumer]

Looks like maybe Atomics.wake was renamed Nov 2020.

In the outputted html, I did this:

Atomics.wake = Atomics.notify

Which got me further, but still more errors:

Maybe I should be using makelove's version in these demos, instead of the npm-published?

[Davidobot]

Maybe I should be using makelove's version in these demos, instead of the npm-published?

I actually haven't pushed anything to npm. Tanner offered to give me rights to it, but I haven't done it yet. Sorry.

The latest version should be obtained from this repo only. Not sure what makelove is using. The node package is very outdated tho.

[konsumer]

I switched all my stuff to makelove, just to avoid building the whole thing, and get past the errors. I can setup a docker lovejs builder later that compiles it, so we have nice fresh builds, after I get the basic proxy demos working.

[konsumer]

Ok, I think I made progress, but it's still not working.

I got the websocket proxy running in docker.

UDP

websocket_to_posix_proxy_1  | Unknown WebSocket opcode received 5!
websocket_to_posix_proxy_1  | Established new proxy connection handler thread for incoming connection, at fd=4
websocket_to_posix_proxy_1  | hashing key: "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
websocket_to_posix_proxy_1  | Sent handshake:
websocket_to_posix_proxy_1  | HTTP/1.1 101 Switching Protocols
websocket_to_posix_proxy_1  | Upgrade: websocket
websocket_to_posix_proxy_1  | Connection: Upgrade
websocket_to_posix_proxy_1  | Sec-WebSocket-Accept: Kfh9QIsMVZcl6xEPYxPHzW8SZ8w=
websocket_to_posix_proxy_1  | 
websocket_to_posix_proxy_1  | 
websocket_to_posix_proxy_1  | Proxy connection closed
websocket_to_posix_proxy_1  | Closing WebSocket connection 4

TCP

websocket_to_posix_proxy_1  | Unknown WebSocket opcode received 5!

Not sure what websocket opcode 5 is, in this context, but that is error in both (on proxy-side.) The project currently demonstrates TCP/UDP attempts over a websocket using Module.websocket.url to set the proxy-address, so this should give you something to test the currently document way to work with sockets, but as I said a plain websocket-interface would really be better, anyway.

[Davidobot]

I switched all my stuff to makelove, just to avoid building the whole thing, and get past the errors. I can setup a docker lovejs builder later that compiles it, so we have nice fresh builds, after I get the basic proxy demos working.

You don't need to build to use the repo. It's precompiled. You'd install it via npm locally.

I doubt you'll get it working until I compile the love source with the websocket flag enabled.

[konsumer]

Oh, duh, yeh it needs to be built with -lwebsocket.js -s PROXY_POSIX_SOCKETS=1 -s USE_PTHREADS=1 -s PROXY_TO_PTHREAD=1.

So if we aren't going to setup some kind of swapping out the API on build (which seems complicated) this might be a reasonable default, and we can just add something to the README for people who need some kind of networking (like me.) I can more properly dockerize & publish the proxy, so it's easier to use, and that could just be the solution. It's not ideal (proxy + original socket-server needed) but it resolves the basic usecase, I think.

I think additionally, I could dockerize the env for building luajs, so it can be quickly used to make a fresh build (could also be used on github actions to create release-builds for easy download.)

As a quick test, to see what the default behavior is doing without that, I wrote this (4th programming language used on this project!):

const WebSocket = require('ws')

const wss = new WebSocket.Server({ port: 12346 })

wss.on('connection', function connection (ws) {
  console.log('connect')
  ws.on('message', function incoming (message) {
    console.log('received: %s', message)
    ws.send('message')
  })
})

I tested with TCP & UDP, and both connected but never registered a message. I tried with Module.websocket.subprotocol set to both text & binary.

[konsumer]

You don't need to build to use the repo. It's precompiled. You'd install it via npm locally.

Ah, yes, I see the src/ folder has built versions of it. I think I could easily build it in Docker, too, but I am happy to use those. I think I can even ADD the URL directly in docker, on build, for my webserver container. I don't think installing with npm, even locally, would do much, as I just need to copy the folder, right?

[Davidobot]

That's correct. Installing it locally gives you the nice command that you can use globally, but that's it.

[konsumer]

Ok, I did a clone of this repo in the dockerfile, and ran lovejs on my demo love apps, here. it seems to build, but then the resulting page freezes, with no error. It may just be because the flags aren't enabled in the compiled lovejs, and it's trying to hit the proxy, but I'm not sure. Hopefully that is helpful to play around with it, and I can do more testing tomorrow.

[konsumer]

This is sort of related, but I dockerized the build environment to make it easier to try things out & simplify things. Feel free to use it, if you like.

https://github.com/konsumer/lovejs-docker

Currently it builds with -lwebsocket.js -s PROXY_POSIX_SOCKETS=1 -s USE_PTHREADS=1 -s PROXY_TO_PTHREAD=1 but I still need to test it out.

[konsumer]

Looking here it seems like there are a lot of options for exposing the JS APIs to C/love.

Here is a simple example I think would work ok to add love.window.js(CODE) to lua:

// https://github.com/Davidobot/love/blob/emscripten/src/modules/window/wrap_Window.cpp

#ifdef __EMSCRIPTEN__ 
int w_js(lua_State *L)
{
  const char* code = luaL_checkstring (L, 1);
  emscripten_run_script(code);
  return 0;
}
#endif

static const luaL_Reg functions[] =
{
#ifdef __EMSCRIPTEN__ 
  { "js", w_js },
#endif
  // ...
};

But we probably want something a bit more purpose-fit, so we can register callbacks and stuff. This seems like a great way to do that, but I'm unsure of what is needed on the C/lua side to put it together.

[Davidobot]

Calling JS from love is provided by: https://github.com/Davidobot/love.js/issues/26

[konsumer]

Hmm. That is cool, but still no way to say "call my lua function with the argument of the message or error and make a client object available." I can just write some inline JS to achieve the same thing, but if the goal is to make a generic way to websocket in lua, it's not going to work. Also, as a sidenote, that lib looks like a stub to me, as the core function is essentially a noop, unless I am missing some other part. Have you tested it?

[Davidobot]

I built the newest love.js version with the linker flag -lwebsocket.js and pushed it to npm. Please let me know if it works. I don't have time to test the demos myself.

[konsumer]

I updated all my docker testing stuff to use npm version, again.

I get

npx love.js
sh: love.js: Permission denied

on latest. I installed globally, and it seemed to work ok, so it's probly not got the right permissions for a good npm-publish.

Additionally, when I run this (/app/src is where the love files are):

mkdir -p /app/build && love.js --title "Network Test" /app/src /app/build && cp /app/template.html /app/build/index.html && light-server -s /app/build -p 8000

I get Bad Magic Number for the wasm file, from the webserver. I tried a few different node-based webservers, so I think it's an issue with the wasm file in npm-published love.js.

I tried my (docker-based, using makelove) builder and I get this:

[konsumer]

Even with this very basic tester in love, and no separate html file (just using what comes with love.js in npm):

local h = love.graphics.getHeight()
local w = love.graphics.getWidth()

function love.draw()
  love.graphics.printf( "Hi. Love works", 0, (h/2) - 6, w, "center" )
end

I get errno: 28, message: "FS error"

I made like this:

npm i -g love.js light-server
love.js . ../build --title "Network Test" --memory 16777216
light-server -s  ../build -p 8000

NODE: v15.14.0
NPM: 7.7.6

[Davidobot]

Yikes! Thanks for altering me. Could you try the simple demo using a love file? I don't think folder support isn't working at the moment.

[konsumer]

That worked with basic test love.

Here is whole testing procedure:

docker run -it --rm -p 8000:8000 node:alpine sh

# inside container

apk add zip
npm i -g love.js light-server
mkdir /tmp/test

cat << EOF > /tmp/test/main.lua
local h = love.graphics.getHeight()
local w = love.graphics.getWidth()

function love.draw()
 love.graphics.printf( "Hi. Love works", 0, (h/2) - 6, w, "center" )
  end
EOF
cd /tmp/test
zip ../test.love main.lua
love.js ../test.love ../build --title "Network Test" --memory 16777216
light-server -s  ../build -p 8000

[konsumer]

I will update my docker-based tests, so I can see if TCP/UDP (over the emscripten proxy-thing) is still failing.

[konsumer]

Ok, I updated the tester project to use npm version of lovejs, and build from a love file.

I inject this into the index.html:

Module.websocket = {
  url: "ws://localhost:12346"
}

I run a simple socket-server liek this:

import socketserver

class MyUDPHandler(socketserver.BaseRequestHandler):
    def handle(self):
        data = self.request[0].strip()
        socket = self.request[1]
        print(f"{self.client_address[0]} wrote: {data}")
        socket.sendto(data.upper(), self.client_address)

print("Starting simple UDP socket server on port 12345")
with socketserver.UDPServer(('0.0.0.0', 12345), MyUDPHandler) as server:
    server.serve_forever()

(tested working locally, in regular love) and I start the websocket_to_posix_proxy provided by emscripten on port 12346. I still get this:

My docker logs say this, which looks like all is ok:

websocket_to_posix_proxy_1  | websocket_to_posix_proxy server is now listening for WebSocket connections to ws://localhost:12346/
web_1                       |   adding: conf.lua (deflated 73%)
web_1                       |   adding: main.lua (deflated 60%)
web_1                       | light-server is listening at http://0.0.0.0:8000
web_1                       |   serving static dir: /app/pub

My instincts tell me emscripten is still not building the wasm right.

[konsumer]

Local, I get this with same python server running (in docker) & same love-code

[konsumer]

Again, if you want to test, it's very fast if you have docker & make installed. https://github.com/konsumer/lovejs-networking

Just run make for help. I have been testing with make udp-web

[ParticleG]

Any update for this?