Experiment crashes but no error message; admin console reset

[aliciamchen]

Is there an existing issue for this?

• [X] I have searched the existing issues

What happened?

In the middle of running my experiment, players are kicked off and the experiment crashes. Unlike with previous versions of empirica, this time I don't get an error message in the console. But when I reload empirica admin, it says no players and no games, and I stop getting the logs from the experiment in the console.

Perhaps this is also related to #466 ?

When I empirica serve the experiment again, I am not able to (unless I delete the tajriba file, like with previous errors I got that crashed the experiment) and it gives me the error below.

Steps To Reproduce

No response

Empirica Version

Version: v1.8.11
SHA:     fce31f9
Build:   154
Branch:  main
Time:    2023-12-20T10:26:18Z

What OS are you seeing the problem on?

macOS

What browser are you seeing the problem on?

Does not apply

Relevant log output

Error: Unhandled error. (connect ECONNREFUSED 127.0.0.1:3000)
    at Ia.Ot.emit (/tmp/ba2f298c74fd691b/node_modules/@empirica/tajriba/src/events.js:146:14)
    at error (/tmp/ba2f298c74fd691b/node_modules/@empirica/tajriba/src/tajriba.ts:332:9)
    at Object.emit (/tmp/ba2f298c74fd691b/node_modules/graphql-ws/lib/client.mjs:104:21)
    at ze.de.onerror (/tmp/ba2f298c74fd691b/node_modules/graphql-ws/lib/client.mjs:162:47)
    at Nf (/tmp/ba2f298c74fd691b/node_modules/ws/lib/event-target.js:290:14)
    at ze.n (/tmp/ba2f298c74fd691b/node_modules/ws/lib/event-target.js:230:9)
    at ze.emit (node:events:513:28)
    at xf (/tmp/ba2f298c74fd691b/node_modules/ws/lib/websocket.js:1027:13)
    at ClientRequest.<anonymous> (/tmp/ba2f298c74fd691b/node_modules/ws/lib/websocket.js:874:5)
    at ClientRequest.emit (node:events:513:28) {
  context: L1 {
    [Symbol(kTarget)]: ze {
      _events: [Object: null prototype] {
        error: [Function: n] {
          [Symbol(kIsForOnEventAttribute)]: true,
          [Symbol(kListener)]: [Function (anonymous)]
        },
        close: [Function: n] {
          [Symbol(kIsForOnEventAttribute)]: true,
          [Symbol(kListener)]: [Function (anonymous)]
        },
        open: [Function: n] {
          [Symbol(kIsForOnEventAttribute)]: true,
          [Symbol(kListener)]: [AsyncFunction (anonymous)]
        },
        message: [Function: n] {
          [Symbol(kIsForOnEventAttribute)]: true,
          [Symbol(kListener)]: [Function (anonymous)]
        }
      },
      _eventsCount: 4,
      _maxListeners: undefined,
      _binaryType: 'nodebuffer',
      _closeCode: 1006,
      _closeFrameReceived: false,
      _closeFrameSent: false,
      _closeMessage: Buffer(0) [Uint8Array] [],
      _closeTimer: null,
      _extensions: {},
      _paused: false,
      _protocol: '',
      _readyState: 2,
      _receiver: null,
      _sender: null,
      _socket: null,
      _bufferedAmount: 0,
      _isServer: false,
      _redirects: 0,
      _url: 'ws://localhost:3000/query',
      _req: null,
      [Symbol(kCapture)]: false
    },
    [Symbol(kType)]: 'error',
    [Symbol(kError)]: Error: connect ECONNREFUSED 127.0.0.1:3000
        at __node_internal_captureLargerStackTrace (node:internal/errors:477:5)
        at __node_internal_exceptionWithHostPort (node:internal/errors:655:12)
        at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1247:16) {
      errno: -111,
      code: 'ECONNREFUSED',
      syscall: 'connect',
      address: '127.0.0.1',
      port: 3000
    },
    [Symbol(kMessage)]: 'connect ECONNREFUSED 127.0.0.1:3000'
  }
}
20:51:54.837 FTL empirica: failed to start error="npm run --silent serve -- --token PFHdSUXYnatSDCzi --url http://localhost:3000/query --sessionTokenPath /root/.empirica/local/callBackSessionToken: exit status 1"
20:51:54.840 ERR serve: failed server command error="exit status 1"
20:52:09.733 INF tajriba: server started build={"branch":"main","build":"154","sha":"fce31f9","tag":"v1.8.11","time":"2023-12-20T10:26:18Z","version":"v1.8.11"}

Anything else?

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[npaton]

I think you might have ran out of memory. What size machine are you using at the moment?

[aliciamchen]

I'm using a DigitalOcean droplet with 8 GB Memory / 4 Intel vCPUs / 25 GB Disk. I've been monitoring the memory usage as I've been running experiments, and when I got this error the max memory usage was 15%.

[aliciamchen]

I made my droplet bigger to be safe, but still have been getting this issue. All the batches and games disappear from the admin console, and after this happens participants cannot progress through the game.

Perhaps this is related to #425?

[npaton]

Thank you for the update. I have not been able to reproduce this yet, I am still looking into it. If it ever happens again, if you could grab the logs in admin browser console, that would be very helpful (or if you still have the incriminating tajriba.json file that you can send to me through Slack). 🙏

[aliciamchen]

Thank you so much; I've DMed you the tajriba file.

[npaton]

I don't know if this is still happening, but I have not had any new reports on this and there have been many changes and fixes since this was happening. I'm closing for now, please reopen if it's still an issue for you.