Resolve issue with etherpad that seems to not save contents

[jdotpz]

I'm betting when this is on mysql and production quality infra it will be fixed, but its worth noting here. It does a bunch of reconnecting, and does not seem to save a lot of data. cc @JohnMcLear @OpenMatt

[JohnMcLear]

hrm, is that caused by having 2 of the same browsers hit a pad? That's expected behaviour if that's the case.

We restrict pad connectivity to one per browser.

[davidascher]

That seems like a weird choice. Aren't users likely to be confused more than anything else?

[jdotpz]

I had an incognito Chrome window up with only that page, and it started misbehaving.

Any hints?

[jdotpz]

[JohnMcLear]

Check your server side logs, it will tell you what's happening..

The disconnects look weird.. That's unexpected behaviour for sure..

[jdotpz]

Here's what I see in error.log: http://pastebin.mozilla.org/8406526

Nothing really in error.log.

[JohnMcLear]

Jump on here: http://pad.webmaker.org/p/2FzaW203iz I'm not getting any disconnects..

[JohnMcLear]

Okay now I'm getting disconnects.. heh wtf

[JohnMcLear]

"Failed to load 'ep_markdownify/static/js/index' for 'ep_markdownify/main/client_hooks/aceEditorCSS': Error: The module at "ep_etherpad-lite/static/js/security.js" does not exist." ace2_common.js:12950

"Failed to load 'ep_markdownify/static/js/index:aceGetFilterStack' for 'ep_markdownify/main/client_hooks/aceGetFilterStack': Error: The module at "ep_etherpad-lite/static/js/security.js" does not exist." ace2_common.js:12950

"Failed to load 'ep_markdownify/static/js/index:aceCreateDomLine' for 'ep_markdownify/main/client_hooks/aceCreateDomLine': Error: The module at "ep_etherpad-lite/static/js/security.js" does not exist." ace2_common.js:12950

"Failed to load 'ep_linkify/static/js/index:aceGetFilterStack' for 'ep_linkify/main/client_hooks/aceGetFilterStack': Error: The module at "ep_etherpad-lite/static/js/security.js" does not exist." ace2_common.js:12950

"Failed to load 'ep_linkify/static/js/index:aceCreateDomLine' for 'ep_linkify/main/client_hooks/aceCreateDomLine': Error: The module at "ep_etherpad-lite/static/js/security.js" does not exist." ace2_common.js:12950

Error: Couldn't find translations for en-GB
Stack trace:
window.html10n</Loader.prototype.parse@http://pad.webmaker.org/static/js/html10n.js:193:31
window.html10n</Loader.prototype.fetch/xhr.onreadystatechange@http://pad.webmaker.org/static/js/html10n.js:167:11
 html10n.js:139
Firefox can't establish a connection to the server at ws://pad.webmaker.org/socket.io/?EIO=3&transport=websocket&sid=2eZXUNxZe-KT_PccAAAF. socket.io.js:5258
Object { type: "CLIENT_VARS", data: Object } pad.js:302
Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help http://xhr.spec.whatwg.org/ 2FzaW203iz:295
Use of getPreventDefault() is deprecated.  Use defaultPrevented instead.

[JohnMcLear]

markdownify and linkify look like they both have errors..

Also Firefox can't establish a connection to the server at ws://pad.webmaker.org/socket.io/?EIO=3&transport=websocket&sid=2eZXUNxZe-KT_PccAAAF. socket.io.js:5258

Check your console log bro!

[jdotpz]

Hmm, this is using sockets? If that is the case, I think the ELB needs to change protocol.

[jdotpz]

Hi @JohnMcLear, many thanks for the help again!

We're still seeing a lot of stability issues, and I wondered if I might be able to please pick your brain as to what the problem might be. I've removed linkify and markdownify as well.

I see this in the log a couple times: http://pastebin.mozilla.org/8444787

I'm not seeing any crazy utilization that would lead me to believe it is sizing of the instance:

[JohnMcLear]

Hrm nothing obvious from that log.. Do this

cd node_modules grep -rni pad.collabClient

And see if any are in plugins

Stability in general should not be an issue put possibly caused by plugin!

----- Reply message ----- From: "JP Schneider" notifications@github.com To: "MozillaFoundation/mofo-lightsaber" mofo-lightsaber@noreply.github.com Cc: "John McLear" John@mclear.co Subject: [mofo-lightsaber] Resolve issue with etherpad that seems to not save contents (#18) Date: Fri, Jan 30, 2015 17:55

Hi @JohnMcLearhttps://github.com/JohnMcLear, many thanks for the help again!

We're still seeing a lot of stability issues, and I wondered if I might be able to please pick your brain as to what the problem might be. I've removed linkify and markdownify as well.

I see this in the log a couple times: http://pastebin.mozilla.org/8444787

I'm not seeing any crazy utilization that would lead me to believe it is sizing of the instance: [screen shot 2015-01-30 at 11 54 41 am]https://cloud.githubusercontent.com/assets/1373628/5980528/d6d37a44-a876-11e4-8aed-c6715d3e488e.JPG

— Reply to this email directly or view it on GitHubhttps://github.com/MozillaFoundation/mofo-lightsaber/issues/18#issuecomment-72241817.

[jdotpz]

here's a file list:

• ubuntu@i-0134faed:/opt/etherpad$ grep -ril collabClient ./ ./node_modules/ep_etherpad-lite/static/js/pad.js ./node_modules/ep_etherpad-lite/static/js/collab_client.js ./node_modules/ep_etherpad-lite/static/js/pad_savedrevs.js ./node_modules/ep_etherpad-lite/static/js/ace2_inner.js ./node_modules/ep_etherpad-lite/static/js/chat.js ./src/static/js/pad.js ./src/static/js/collab_client.js ./src/static/js/pad_savedrevs.js ./src/static/js/ace2_inner.js ./src/static/js/chat.js ./var/minified_L2phdmFzY3JpcHRzL2xpYi9lcF9ldGhlcnBhZC1saXRlL3N0YXRpYy9qcy9hY2UyX2lubmVyLmpzP2NhbGxiYWNrPXJlcXVpcmUuZGVmaW5l ./var/minified_L2phdmFzY3JpcHRzL2xpYi9lcF9ldGhlcnBhZC1saXRlL3N0YXRpYy9qcy90aW1lc2xpZGVyLmpzP2NhbGxiYWNrPXJlcXVpcmUuZGVmaW5l ./var/minified_L2phdmFzY3JpcHRzL2xpYi9lcF9ldGhlcnBhZC1saXRlL3N0YXRpYy9qcy9wYWQuanMY2FsbGJhY2s9cmVxdWlyZS5kZWZpbmU

./node_modules/ep_etherpad-lite/static/js/pad.js:var getCollabClient = require('./collab_client').getCollabClient; ./node_modules/ep_etherpad-lite/static/js/pad.js: msg.client_rev=pad.collabClient.getCurrentRevisionNumber(); ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.setChannelState("CONNECTED"); ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.setChannelState("DISCONNECTED"); ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.setChannelState("DISCONNECTED", "reconnect_timeout"); ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.setChannelState("RECONNECTING"); ./node_modules/ep_etherpad-lite/static/js/pad.js: // Add a 'globalUserColor' property to myUserInfo, so collabClient knows we have a query parameter. ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.handleMessageFromServer(obj); ./node_modules/ep_etherpad-lite/static/js/pad.js: collabClient: null, ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.sendClientMessage(msg); ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient = getCollabClient(padeditor.ace, clientVars.collab_client_vars, pad.myUserInfo, { ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.setOnUserJoin(pad.handleUserJoin); ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.setOnUpdateUserInfo(pad.handleUserUpdate); ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.setOnUserLeave(pad.handleUserLeave); ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.setOnClientMessage(pad.handleClientMessage); ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.setOnServerMessage(pad.handleServerMessage); ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.setOnChannelStateChange(pad.handleChannelStateChange); ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.setOnInternalAction(pad.handleCollabAction); ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.sendMessage({"type": "GET_CHAT_MESSAGES", "start": start, "end": chatHead}); ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.updateUserInfo(pad.myUserInfo); ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.updateUserInfo(pad.myUserInfo); ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.sendClientMessage( ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.sendClientMessage( ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.diagnosticInfo.collabDiagnosticInfo = pad.collabClient.getDiagnosticInfo(); ./node_modules/ep_etherpad-lite/static/js/pad.js: $('form#reconnectform input.missedChanges').val(JSON.stringify(pad.collabClient.getMissedChanges())); ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.callWhenNotCommitting(f); ./node_modules/ep_etherpad-lite/static/js/pad.js: return pad.collabClient.getCurrentRevisionNumber(); ./node_modules/ep_etherpad-lite/static/js/pad.js: if (!pad.collabClient) ./node_modules/ep_etherpad-lite/static/js/pad.js: pad.collabClient.addHistoricalAuthors(data); ./node_modules/ep_etherpad-lite/static/js/collabclient.js: "serverVars" are from calling doc.getCollabClientVars() on the server. / ./node_modules/ep_etherpad-lite/static/js/collab_client.js:function getCollabClient(ace2editor, serverVars, initialUserInfo, options, _pad) ./node_modules/ep_etherpad-lite/static/js/collab_client.js:exports.getCollabClient = getCollabClient; ./node_modules/ep_etherpad-lite/static/js/pad_savedrevs.js: pad.collabClient.sendMessage({"type": "SAVE_REVISION"}); ./node_modules/ep_etherpad-lite/static/js/ace2_inner.js: parent.parent.pad.collabClient.sendMessage({"type":"SAVEREVISION"}); / The parent.parent part of this is BAD and I feel bad.. It may break something */ ./node_modules/ep_etherpad-lite/static/js/chat.js: this._pad.collabClient.sendMessage({"type": "CHAT_MESSAGE", "text": text}); ./node_modules/ep_etherpad-lite/static/js/chat.js: pad.collabClient.sendMessage({"type": "GET_CHAT_MESSAGES", "start": start, "end": end}); ./src/static/js/pad.js:var getCollabClient = require('./collab_client').getCollabClient; ./src/static/js/pad.js: msg.client_rev=pad.collabClient.getCurrentRevisionNumber(); ./src/static/js/pad.js: pad.collabClient.setChannelState("CONNECTED"); ./src/static/js/pad.js: pad.collabClient.setChannelState("DISCONNECTED"); ./src/static/js/pad.js: pad.collabClient.setChannelState("DISCONNECTED", "reconnect_timeout"); ./src/static/js/pad.js: pad.collabClient.setChannelState("RECONNECTING"); ./src/static/js/pad.js: // Add a 'globalUserColor' property to myUserInfo, so collabClient knows we have a query parameter. ./src/static/js/pad.js: pad.collabClient.handleMessageFromServer(obj); ./src/static/js/pad.js: collabClient: null, ./src/static/js/pad.js: pad.collabClient.sendClientMessage(msg); ./src/static/js/pad.js: pad.collabClient = getCollabClient(padeditor.ace, clientVars.collab_client_vars, pad.myUserInfo, { ./src/static/js/pad.js: pad.collabClient.setOnUserJoin(pad.handleUserJoin); ./src/static/js/pad.js: pad.collabClient.setOnUpdateUserInfo(pad.handleUserUpdate);

[JohnMcLear]

No plugins there.. Very weird..

[jdotpz]

Interesting indeed.

I'm seeing this quite a bit as well:

An error occured The error was reported with the following id: 'niFl3a2pnIcLyokCTzGD'

Please press and hold Ctrl and press F5 to reload this page, if the problem persists please send this error message to your webmaster: 'ErrorId: niFl3a2pnIcLyokCTzGD URL: https://pad.webmaker.org/p/noshizzle UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.94 Safari/537.36 Uncaught TypeError: Cannot read property 'setChannelState' of null in https://pad.webmaker.org/javascripts/lib/ep_etherpad-lite/static/js/pad.js?callback=require.define at line 254' Chat 0

[JohnMcLear]

Are you rewriting or reverse proxying or something? It sounds like an issue with your configs...

[jdotpz]

Actually, at the moment I am not reverse proxying at all! Because I had /plugins broken in my vhost/conf, I turned it off until we got our plugins in a stable place.

[JohnMcLear]

hrm, do you get the same if you don't use https?

[JohnMcLear]

hrm this could be the issue: Firefox can't establish a connection to the server at wss://pad.webmaker.org/socket.io/?EIO=3&transport=websocket&sid=oBLts_3-MopE7lCeAADL.

[JohnMcLear]

Just for what it's worth, I can support this project with general software etc. stuff but specific implementation into environments it's tricky for me to find the resources to support things for free. Not to sound like I'm not willing to help or anything just want you to understand if I'm less forthcoming in resolving these issues for you.

[jdotpz]

Understood, and I appreciate the help you are able to give!

I've updated the load balancer to use tcp instead of http so it should now support ws protocol.

[jdotpz]

FWIW, the socket.io error we see now is actually likely in the code.

I see it on beta as well:

[JohnMcLear]

beta.etherpad.org is stable tho right?

[jdotpz]

Yup, so far as I know. Just observing that the socket.io error we see there is not an indicator of whether that funky error pops.

[JohnMcLear]

Fixed socketIO initial upgrade error on beta.etherpad.org - updated nginx rewrite docs. Very easy to do

[OpenMatt]

• @jdotpz : I assume closing this one is a blocker for https://github.com/MozillaFoundation/mofo-lightsaber/issues/20 , right? Until this is closed, users may lose data?

[jdotpz]

I believe we're all set here, and we can reopen if we see the issue recur while in use.