search

jijo-paulose / node-xmpp-bosh

Automatically exported from code.google.com/p/node-xmpp-bosh
0 stars 0 forks source link

Encoding issues #3

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Hello,

When using node-xmpp-bosh (through Jappix), I get an issue I did not get from 
any other BOSH server. When the XML data contains some characters like 
é/à/è/... it is not well encoded.

e.g.: my name, which is "Valérian Saliou". Through node-xmpp-bosh, it gives 
"Valérian Saliou".

Might be some UTF-8 encode stuffs, or a bad server Content-Type header (?).

Thanks,

Valérian Saliou,
Jappix founder

Original issue reported on code.google.com by vanaryon on 13 May 2011 at 5:16

GoogleCodeExporter commented 9 years ago 
Can you try running the following code on your machine and let me know if you 
see the correct output:

ltx= require('ltx');
e = new ltx.Element('body').t("Valérian Saliou").tree();
console.log(e.toString());

Also, try cat(ing) the source file (the file with contents above) to ensure 
that your terminal can display the special characters. If it works well, then 
it's probably an issue with a missing header, etc...

Original comment by dhruvb...@gmail.com on 13 May 2011 at 7:35

GoogleCodeExporter commented 9 years ago 
Okay, I sent this to my friend, he will test it. Thanks!

Original comment by vanaryon on 13 May 2011 at 7:58

GoogleCodeExporter commented 9 years ago 
Hello Vanaryon, I've also added charset=UTF-8 to the Content-Type header 
(commit #197), so you could ask you friend to check that as well while he is at 
it. Let me know which one works (if any of the fixes do work).

Original comment by dhruvb...@gmail.com on 14 May 2011 at 7:30

GoogleCodeExporter commented 9 years ago 
It looks like something is either passing UTF-8 as Latin1, or encoding it 
unnecessarily.

Some protocol dumps could help us most. Could you show us an example snippet 
containing accents of both XMPP and BOSH sides?

Original comment by Str...@googlemail.com on 14 May 2011 at 9:32

GoogleCodeExporter commented 9 years ago 
Sure, here is the XMPP side:
----------------------------
<iq xmlns="jabber:client" from="jappix@jappix.com" to="jappix@jappix.com/Gajim" 
id="4" type="result">
  <query xmlns="jabber:iq:roster">
    <item subscription="both" name="Valérian Saliou" jid="vanaryon@jappix.com">
      <group>Développeurs</group>
    </item>
  </query>
</iq>

And the BOSH side:
------------------
<body xmlns="http://jabber.org/protocol/httpbind" 
stream="256C25A8-7D71-44D8-84C9-AD80A8163391">
  <iq from="jappix@jappix.com" to="jappix@jappix.com/Jappix (1305384294585)" id="2" type="result" xmlns="jabber:client" xmlns:stream="http://etherx.jabber.org/streams" version="1.0">
    <query xmlns="jabber:iq:roster">
      <item subscription="both" name="Valérian Saliou" jid="vanaryon@jappix.com">
        <group>Développeurs</group>
      </item>
    </query>
  </iq>
</body>

Original comment by vanaryon on 14 May 2011 at 2:48

GoogleCodeExporter commented 9 years ago 
To reply to your third comment, it still does not work with your patch. Might 
be BOSH-side issue rather than HTTP header issue.

Original comment by vanaryon on 14 May 2011 at 3:00

GoogleCodeExporter commented 9 years ago 
I tried redirecting the output of this node script to a file:

ltx = require('ltx');
s = 'Valérian Saliou';

e = new ltx.Element('body').c('name').t(s).tree();
console.log(e.toString());
console.log(s);

And this is what I got:

<body><name>Val�rian Saliou</name></body>
Val�rian Saliou

So, I don't think it's a problem with ltx, but it's something more deep rooted 
in node itself.

Will post on the node.js mailing list.

Original comment by dhruvb...@gmail.com on 14 May 2011 at 5:44

GoogleCodeExporter commented 9 years ago 
Okay, thanks a lot. Let me know when it will be updated, I am waiting for this 
patch to deploy node-xmpp-bosh in production on my service ;)

Original comment by vanaryon on 14 May 2011 at 5:49

GoogleCodeExporter commented 9 years ago 
Oops, it works now. I hadn't saved the source file as UTF-8. Stupid me!!

For this source file:

ltx = require('ltx');
s = 'Valérian Saliou';

e = new ltx.Element('body').c('name').t(s).tree();
console.log(e.toString());
console.log(s);

I get the following output:

<body><name>Valérian Saliou</name></body>
Valérian Saliou

Which is expected, so everything seems to be in order till now. It could be an 
issue with sending on the wire. Will need someone to confirm this though.

@vanaryon BOSH does heavy logging if you run it with --logging=DEBUG. Is there 
any way you could check the logs and see if the data was logged correctly when 
it was received from the XMPP server? Such lines are prefixed with XMPP_PROXY::

Original comment by dhruvb...@gmail.com on 14 May 2011 at 5:55

GoogleCodeExporter commented 9 years ago 
Can you push your patch on the SVN? I will check it ;)

Original comment by vanaryon on 14 May 2011 at 6:02

GoogleCodeExporter commented 9 years ago 
Well, I haven't changed anything (things seem to be fine - at the node.js and 
ltx layer, and these are the 2 entities that ever touch the data), so there's 
no patch. In case you are referring to the logging bit, you can do that by 
restarting the bosh-server as:

bosh-server --logging=DEBUG

Original comment by dhruvb...@gmail.com on 14 May 2011 at 6:17

GoogleCodeExporter commented 9 years ago 
The logged data also have the encoding issue on an UTF-8 terminal.
Maybe it comes from the XML to string parser, located in a handler?

Original comment by vanaryon on 14 May 2011 at 6:45

GoogleCodeExporter commented 9 years ago 
I checked again, and I see this line in my logs:
<active xmlns="http://jabber.org/protocol/chatstates"/><body>Valérian 
Saliou</body></message></body>

Which means that it is able to store-and-forward these special characters 
correctly (at least on the face of it). Maybe you could check if they are 
coming properly from the source??

Original comment by dhruvb...@gmail.com on 14 May 2011 at 7:02

GoogleCodeExporter commented 9 years ago 
Mhh, tryed to install it on my Debian 6 server.

I get this with node-expat:

npm ERR! Error: node-expat@1.3.0 install: `./install.sh`
npm ERR! `sh "-c" "./install.sh"` failed with 127
npm ERR!     at ChildProcess.<anonymous> 
(/usr/lib/nodejs/.npm/npm/0.3.4/package/lib/utils/exec.js:45:20)
npm ERR!     at ChildProcess.emit (events.js:45:17)
npm ERR!     at ChildProcess.onexit (child_process.js:171:12)
npm ERR! 
npm ERR! Failed at the node-expat@1.3.0 install script.
npm ERR! This is most likely a problem with the node-expat package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     ./install.sh
npm ERR! You can get their info via:
npm ERR!     npm owner ls node-expat
npm ERR! There is likely additional logging output above.

I installed the missing stuffs when searching on the web, but cannot get it to 
work. Have you an idea? I need to install it to check if I also get the 
encoding issue on my machine ;)

Original comment by vanaryon on 15 May 2011 at 8:28

GoogleCodeExporter commented 9 years ago 
This is *probably* because you don't have libexpat and libexpat1-dev (or 
libexpat1-devel) installed. Try installing them and re-running the 
installation. Also, try more verbose output. Earlier npm used to spit out all 
compiler errors on the screen (can't be sure that the new behaviour has changed 
though).

Also, make sure you have "sh" installed ;)

Original comment by dhruvb...@gmail.com on 15 May 2011 at 8:47

GoogleCodeExporter commented 9 years ago 
Still crashes. Ho do I enable verbose output?

Note: these two commands crash:
npm install ltx -g
npm install node-expat -g

Original comment by vanaryon on 15 May 2011 at 8:56

GoogleCodeExporter commented 9 years ago 
And something more, can I get a SVN access to node-xmpp-bosh rep? It will be 
easier when I will have some minor changes to do ;)

Original comment by vanaryon on 15 May 2011 at 9:16

GoogleCodeExporter commented 9 years ago 
You'll need to download this file: 
https://github.com/astro/node-expat/tarball/master

and run: bash install.sh

If that command succeeds, you are done!
(make sure that you set CPLUS_INCLUDE_PATH to the path that has the file 
expat.h)

Original comment by dhruvb...@gmail.com on 15 May 2011 at 9:21

GoogleCodeExporter commented 9 years ago 
What's your google account email ID?

Original comment by dhruvb...@gmail.com on 15 May 2011 at 9:24

GoogleCodeExporter commented 9 years ago 
The Google ID is vanaryon@gmail.com

I check if it works with your solution, thanks!

Original comment by vanaryon on 15 May 2011 at 9:32

GoogleCodeExporter commented 9 years ago 
Added!

Original comment by dhruvb...@gmail.com on 15 May 2011 at 9:34

GoogleCodeExporter commented 9 years ago 
Thanks!

./install.sh: 3: node-waf: not found
And when I do npm install node-waf or npm install waf, it returns me a 404 not 
found!

Original comment by vanaryon on 15 May 2011 at 9:36

GoogleCodeExporter commented 9 years ago 
node-waf is installed by default (it's not an npm module). It should be in the 
same location as the node executable. If you have installed node in a custom 
location using --prefix=/opt/blah/ then you'll need to add it to your PATH 
variable as:

export PATH=/opt/blah/bin/:$PATH

and check if 'node' and 'node-waf' are available.

This should get you away!

Original comment by dhruvb...@gmail.com on 15 May 2011 at 1:38

GoogleCodeExporter commented 9 years ago 
Okay, will check it in minutes.

By the way, I pushed some changes to the SVN about the GET handler ;)

Original comment by vanaryon on 15 May 2011 at 2:06

GoogleCodeExporter commented 9 years ago 
Can't find where node-waf is located on my setup.

I will wait a bit until I can chat with Link Mauve to ask me how to do ;)

Original comment by vanaryon on 15 May 2011 at 2:54

GoogleCodeExporter commented 9 years ago 
I tested with the patch, and there's still the same issue.

Original comment by vanaryon on 25 May 2011 at 4:29

GoogleCodeExporter commented 9 years ago 
Well, I'm not able to reproduce this issue on at least 2 machines I tested on. 
Is there any way I can get hold of the machine on which it is failing?

Original comment by dhruvb...@gmail.com on 26 May 2011 at 7:30

GoogleCodeExporter commented 9 years ago 
You mean all your é, è, ... are passing well with you?

I will ask Link Mauve if he can give you a SSH access tomorrow.

But there is something I am thinking about: Link Mauve is using a Nginx proxy 
to NXB. Can it break something?

Original comment by vanaryon on 26 May 2011 at 4:01

GoogleCodeExporter commented 9 years ago 
Cool!! (y)
Hopefully nginx won't break anything since it used quite a lot!! Either ways, 
we should eliminate that by connecting directly (but I don't think nginx is at 
fault).

Original comment by dhruvb...@gmail.com on 26 May 2011 at 6:01

GoogleCodeExporter commented 9 years ago 
Mhh, I am not sure Link Mauve got no issue by connecting it directly, on the 
5280 port (it is open for him, locally, not for me, the reason why I use his 
proxy).

Anyway, maybe the issue can be due to the locale encoding in use on his system 
(FR_fr, either ISO-XXXX or UTF-8 or UTF-16), which may affect node.js 
characters encoding?!

Because you seem not to encounter this issue on your machines, I think I will 
tell you more about this once I will have my own NXB running on the Jappix.com 
server.

Thanks!

Original comment by vanaryon on 26 May 2011 at 7:32

GoogleCodeExporter commented 9 years ago 
cool!! So, connecting to it directly works. Maybe it's something that the proxy 
is doing. I'd be interested in knowing what exactly happened.

Original comment by dhruvb...@gmail.com on 28 May 2011 at 7:09

GoogleCodeExporter commented 9 years ago 
Okay. I will ping you as soon as I will have NXB running on my server, but bot 
before some weeks, I am a bit busy and I must admit that node.js and the 
dependencies are a bit hard to install, need to study that point ;)

Original comment by vanaryon on 28 May 2011 at 7:36

GoogleCodeExporter commented 9 years ago 
Hi,

We compiled nodejs and node-xmpp-bosh. All is fine, but we cannot use 
node-xmpp-bosh (we can launch it, but when something dealing with TLS happen, 
it crashes). We get this log:

/home/vanaryon/node/bosh/src/starttls.js:25
var verifyError = pair._ssl.verifyError();
^
TypeError: Cannot call method 'verifyError' of undefined
at SecurePair.<anonymous> (/home/vanaryon/node/bosh/src/starttls.js:25:33)
at SecurePair.emit (events.js:61:17)
at SecurePair.maybeInitFinished (tls.js:598:10)
at CleartextStream._push (tls.js:269:17)
at SecurePair.cycle (tls.js:574:20)
at EncryptedStream.write (tls.js:96:13)
at Socket.ondata (stream.js:36:26)
at Socket.emit (events.js:64:17)
at Socket._onReadable (net.js:678:14)
at IOWatcher.onReadable [as callback] (net.js:177:10)

Link Mauve thinks it is because OpenSSL cannot be found, but the openssl 
package is installed, and we also tried re-compiling nodejs with openssl 
support. But it's the same.

Any idea?

Original comment by vanaryon on 2 Jun 2011 at 6:36

GoogleCodeExporter commented 9 years ago 
Is there any way I can reproduce this issue? For example, are you trying to log 
in to gmail? Also, can you observe the same issue on all machines or only a 
certain set of machines? What does uname -a print? Did you compile node.js with 
OpenSSL enabled? You need the OpenSSL headers as well as the libraries for this 
to work.

Original comment by dhruvb...@gmail.com on 2 Jun 2011 at 6:52

GoogleCodeExporter commented 9 years ago 
Well, we only have this issue on Debian. For other machines (Link Mauve's one) 
it works fine.

And yes we compiled nodejs with openssl enabled.

Original comment by vanaryon on 2 Jun 2011 at 7:12

GoogleCodeExporter commented 9 years ago 
And a gmail.com connection also makes NXB crash.

Original comment by vanaryon on 2 Jun 2011 at 7:16

GoogleCodeExporter commented 9 years ago 
It looks like an OpenSSL issue though: 
https://github.com/astro/node-xmpp/issues/21

You could you use to 0.4.7 till I figure out how to fix it.

Original comment by dhruvb...@gmail.com on 2 Jun 2011 at 7:22

GoogleCodeExporter commented 9 years ago 
Is it possible you join the chatroom xmpp:jappix@conference.codingteam.net?join 
to have a talk about all of this?

Thanks :)

Original comment by vanaryon on 2 Jun 2011 at 7:33

GoogleCodeExporter commented 9 years ago 
I'm in ;)

Original comment by dhruvb...@gmail.com on 2 Jun 2011 at 7:35

GoogleCodeExporter commented 9 years ago 
The fix is on line 206 of xmpp-proxy.js. Just change
toString('binary') to toString() and it should be fine.
I've tested on your server but it's not checked in. Please could you check it 
in since I won't be able to commit anything for a few days.

Please reopen if you see the same issue again.

Original comment by dhruvb...@gmail.com on 2 Jun 2011 at 10:04

GoogleCodeExporter commented 9 years ago 
Thanks a lot!

It fixed the encoding issue, so that I commited it :)

Original comment by vanaryon on 3 Jun 2011 at 8:22

GoogleCodeExporter commented 9 years ago 
Great! Thank you :-)

Original comment by dhruvb...@gmail.com on 3 Jun 2011 at 8:43