Server not responding to network requests

[benloh]

Joshua reports having trouble getting GEMSTEP to run as a server on his new M2 laptop.

It runs fine on localhost: Main, Viewers, and Character Controllers all work.

But on a client machine, it seems that no connection is being made to the server. On the client, if you go to http://(serverip) you get no response at all — the network console reports no response, and the terminal on the server does not show any kind of connection request.

On the client, we CAN see the outgoing request, we just don’t see a response from the server. There’s no browser console output. I would think that even if the socket / server communication is broken, you would still at least be able to serve the HTML files...

It’s probably not a browser security issue because he used two different clients: an old MPB and an iPhone, and neither could connect to the server.

The firewall has been turned off. He's using the right node version, and we've tried a fresh install. MEME is on node v10.20.1, and GEMSTEP on v14.18.1.

The run log looks like this:

bl-educ-0007991:gsgo jdanish$ nvm use
Found '/Users/jdanish/gitcode/gsgo/.nvmrc' with version <v14.18.1>
Now using node v14.18.1 (npm v9.6.4)
bl-educ-0007991:gsgo jdanish$ npm run gem
> gem
> cd gs_packages/gem-srv/ && npm run local
> @gemstep/gem-srv@0.1.0-alpha.0 local
> npm run dev:gem
> @gemstep/gem-srv@0.1.0-alpha.0 dev:gem
> nodemon --trace-warnings gem_run.js dev
[nodemon] 2.0.19
[nodemon] reading config ./package.json
[nodemon] to restart at any time, enter `rs`
[nodemon] or send SIGHUP to 14642 to restart
[nodemon] ignoring: ./node_modules/**/*
[nodemon] watching path(s): ../ursys/server/**/* gem_run.js server/**/* config/**/*
[nodemon] watching extensions: js,json,graphql
[nodemon] starting `node --trace-warnings gem_run.js dev`
[nodemon] spawning
[nodemon] child pid: 14644
[nodemon] watching 14 files
GEMRUN        Starting Development Server...
GEMRUN        using repo branch: dev-jd/spring-2023-extending
GEM_SRV       START HTTP server (Express) 1682122061419
 WPACK        ... wp.base.loaders setting default rules
 WPACK        ... wp.pack.webapp setting appsrv bundle rules (dev)
GEM_SRV       ... starting hot devserver (this may take a while)
URDB          Starting Database Server [async]
URDB          ... loading runtime/db.loki
URDB          ... lokidb loaded w/ 1 collections
URDB          ... loaded schema config/graphql/schema.graphql
URDB          ... bound GraphQL Endpoint /urnet/urdb
URDB          Database Server ready
GEM_SRV       ... transpiling complete!
 URLOG        logging to /Users/jdanish/gitcode/gsgo/gs_packages/gem-srv/runtime/logs
GEM_SRV       webapp bundle: '/Users/jdanish/gitcode/gsgo/gs_packages/gem-srv/built/web'
GEM_SRV       webapp server listening 192.168.50.186 on port 80
GEM_SRV       LIVE RELOAD ENABLED
PTRACK        LISTENING FOR PTRACK ON PORT 21234
 URNET        socket server listening on port 2930
webpack built 8ce8e3c16f7d8456840d in 3372ms
NETDVC        devicedir is: <empty>
NETDVC        devicedir is: UDEV_01:0 
NETDVC        devicedir is: UDEV_01:0 
NETDVC        devicedir is: UDEV_01:0 UDEV_02:0 
PTRACK        MQTT Offline
PTRACK        MQTT Connection Closed
PTRACK        MQTT reconnect
NETDVC        ... UADDR_01 dropping 1 registered device(s)
NETDVC        devicedir is: UDEV_02:0 
NETDVC        devicedir is: UDEV_02:0 
NETDVC        devicedir is: UDEV_02:0 UDEV_03:0 

So everything appears normal.

Wireshark log for GEMSTEP looks like this: NOTE that the first 4 lines are probably MEME/NetCreate calls to port 3000. GEMSTEP starts with Number 518, the [SYN] on port 80.

In contrast, here's Wireshark output with MEME, note all the ACK messages from the server:

Tests

Other Mac

Ben is able to get it to run on his M1 MacBook Air with a clean install.

Use port 8888 instead of 80

We tried changing GS_APP_PORT to 8888, and are able to run Main on port 8888 on the server, but the client had the same connection problem, e.g. http://192.168.50.186:8888 from the client was unable to connect to the server. (edited)

Use another app

NetCreate does run, so we also tried changing GS_APP_PORT to 3000 but still the same problem. So it doesn't seem like it's an issue with port 80. I was at first thinking it was some kind of Ventura security/firewall/permissions issue, but given that NetCreate runs, it seems odd that GEMSTEP wouldn't. The compiled MEME Electron app also runs.

Other Ideas We Haven't Tried Yet

• Try installing and running MAMP or MAMP Pro and try to serve a PHP site on port 80. Pay particular attention to security permissions that are being requested as you run it. And then see if Terminal (or VSCode, if you're running the server from VSCode) has similiar permissions.

• Try running GEMSTEP on the other mac

• Try running NetCreate or MEME with Node v14 (may not work).

• Try adding more verbose server logging to monitor connection status and any failures

• Review IU security / restrictions that might still be enabled on the machine

[benloh]

unassigned @jdanish

[benloh]

In GitLab by @daveseah on Apr 25, 2023, 13:15

Noting a difference in the two wireshark traces in the very beginning.

• In the working connection (MEME) there are [SYN] and [ACK] packets (sometimes combined as [SYN, ACK]) going back and forth. This is the TCP connection handshake. SYN is a request to sync, and ACK is the returned acknowledgement that this is OK. It happens in both directions.
• In the GEMSTEP one, there are [SYN] and [RST,ACK] instead. RST means that the connection was refused (reset).

This suggests that the M2 is receiving the connection request, but is actively refusing the attempt to connect and sending back an [RST,ACK] packet. The client continues to send the connection request over and over again.

There is a list of causes for RST on this page. One of them is that there is no open port on the server. So I think we have to check when the server is actually listening on the right port and the right interface. It could be something like an extra space in local-settings.json in the specified port or a malformed JSON file that is silently failing.

OTHER CHECKS

• [ ] M2: Does the application asset server work? http://<ip-addr>/assets
• [ ] M2: Does mcp-srv work? This is a separate express server for centralized asset hosting. To test, stop the GEMSTEP server and then do npm run mcp:multi to start the resource host server and browse to http://<ip-addr>:8080.
• [ ] Request a copy of Joshua's gs_packages/gem-srv/config/local-settings.json file to check for weirdness.
• [X] Also check that the local settings import handles malformed JSON and reports an error.
• • Webpack is converting it into OVERRIDES for import by mcp-settings and gem-settings.js, so if there was an error it would choke (?) during build.
• • A malformed port set in GS_APP_PORT works for strings that are numbers, and malformed numbers will throw an error, so this probably isn't the issue...
• [ ] Add some detailed express server startup debug statements.

[benloh]

In GitLab by @daveseah on Apr 25, 2023, 13:29

• [X] I was able to run GEMSTEP with Joshua's spring2023 branch and art-assets on my M1.
• [X] testing use of strings for PORT doesn't seem to break the server in an obvious way.

[benloh]

In GitLab by @daveseah on Apr 25, 2023, 14:10

ANOTHER THING TO TRY

The particular node binary for v14.18.1 being blocked despite the firewall being off due to MacOS security. Each version of node that is installed is a different binary, so this may be why NetCreate works and GEMSTEP doesn't IF they are using different versions of NodeJS as determined by nvm use and the .nvmrc file.

• [x] nvm uninstall v14.18.1 (if this is the version being used by GEMSTEP via nvm use)
• [x] rm -fr ~/.nvm/versions/node/v14.18.1
• [x] nvm install v14.18.1 - this should be a new binary presumably distinct from the previous one, even though the path is the same...

NOTE: I don't think nvm use permanently sets the node version in a folder. If you change it to something in another app project that required a different version, and then switch back, you have to run nvm use each time.

NOTE: Each version of node switched-in by nvm use has its own set of global node_modules, so you will have to reinstall lerna. Also, you should nuke node_modules in the GEMSTEP directory after changing the node version because they may be incompatible with the version you're using.

[benloh]

In GitLab by @jdanish on Apr 25, 2023, 15:18

Tried this, and this is what the helper script asked for so this is what I ran:

nvm install v14.18.1 nvm alias default v14.18.1 nvm use npm install -g lerna npm ci npm run bootstrap

[benloh]

In GitLab by @jdanish on Apr 25, 2023, 15:21

This did it! With the new node it asked me to sign off on node accepting connections and I did (I have an admin password now) and it worked. So it was somehow caching that I couldn't before. Anyhow, thanks!! Apologies for the hassle!!!