bing translator and non-well-formed html

[GoogleCodeExporter]

Comment by mightysam, Nov 3:

We have been using the this jquery translation library with Google on our site 
for close to 2 years and it's working great. Recently we tried to switch to 
Bing and I can see the site start to fall part after switching between a few 
languages.

After some investigation, I believe the problem lies on Bing's requirement on 
sending well formatted HTML tags. 
http://msdn.microsoft.com/en-us/library/ff512406.aspx which may not be 
happening because jquery is breaking up the HTML.

For example, the jquery translation library may break up the site's HTML into 
the following before sending to Bing: </div><div>Text 1</div><div>Text 2</div>

Bing will return the following after translation, stripping out incomplete 
tags: <div>Text 1</div><div>Text 2</div>

When the library tries to put the results back to the original position, the 
website will be messed up with missing and misplaced tags. Any suggestion on 
how this can be fixed?

Original issue reported on code.google.com by balazs.endresz on 6 Nov 2011 at 4:02

[GoogleCodeExporter]

Looks like it works if the text is translated as "text/plain": 
http://code.google.com/p/jquery-translate/source/diff?spec=svn242&old=241&r=242&
format=unidiff&path=%2Ftrunk%2Fsrc%2Fjquery.translate-core.js

Original comment by balazs.endresz on 6 Nov 2011 at 4:06

• Changed state: Fixed

[GoogleCodeExporter]

I have tried to switch to "text/plain" last week but it wasn't work that well. 
My page was broken down into 5 calls to the translation server, and the 2nd and 
3rd calls return empty result most of the time, thus messing up the page 
structure.

I will try to use your new file to see if that makes any difference.

Original comment by mighty...@gmail.com on 7 Nov 2011 at 8:51

[GoogleCodeExporter]

Please let me know how it worked for you if you tried it. I don't have any time 
at all now, but will try to have a look at it before the weekend.

Original comment by balazs.endresz on 8 Nov 2011 at 6:42

• Changed state: Started

[GoogleCodeExporter]

I have tried the new library and I still have the same problem.
To investigate, I compare the calls that get sent to Google vs Microsoft and 
found that that the jquery-translate library is breaking up the page and calls 
differently.

For example:
Google: <div> </div>
Microsft: <div> </div>

Google: <div>Mar</div>
Microsoft: (1st call) <div>Ma   (2nd call) r</div>...

Is it expected that calls will be split up differently when using different 
service?

Original comment by mighty...@gmail.com on 9 Nov 2011 at 10:06

[GoogleCodeExporter]

I don't really see how that could happen. Would you please send a test page by 
email, or somehow? I could help much more easily then.

Original comment by balazs.endresz on 10 Nov 2011 at 7:56

[GoogleCodeExporter]

The Microsoft API seems to place HTML tags in the wrong place:

<div>Got a Q for 311? Now U Can Text It!</div>

is translated as:

Haben <div>eine Dame für 311? Jetzt u kann Text es!<\/div>

In some cases the exact position of tags in a sentence is not easily decidable 
but this looks like something that could be fixed easily by Microsoft.

In my experience the "text/plain" encoding works fine otherwise.

Original comment by balazs.endresz on 20 Nov 2011 at 7:11

[GoogleCodeExporter]

But the tags are in the right place if I translate this with "text/html" but 
that can't handle non-well-formed HTML. So, you can choose which error you 
prefer :)

Original comment by balazs.endresz on 20 Nov 2011 at 7:23

[GoogleCodeExporter]

If you use "text/html" with the MS API then apparently you can translate the 
whole page in a single request:
$("body").translate("en", "de", {limit:Infinity});

Google has a 2000 limit because of this: http://support.microsoft.com/kb/208427

Original comment by balazs.endresz on 20 Nov 2011 at 7:36

[GoogleCodeExporter]

Yea, the tag will messed up if I use text/html.
But the problem I am facing is when using "text/plain", the library does not 
breakup the page correctly. Have you experienced that with the page I sent you?

I have tried to add the "limit:Infinity" but the page doesn't translate at all. 
To make sure it's not a format issue, I tried "limit:2000" and the page 
translates, although some parts are still messed up.

Original comment by mighty...@gmail.com on 2 Dec 2011 at 8:48

[GoogleCodeExporter]

To sum this up, the problem is the following:

"text/plain": tags are misplaced

"text/html": HTML has to be well formed, additional tags are inserted

"limit:Infinity": it worked for me on the site you sent me but that isn't going 
to work on IE, so that's not good anyway, though I'll have another look at it 
when I'll have internet at home

Original comment by balazs.endresz on 3 Dec 2011 at 1:56

[GoogleCodeExporter]

http://social.msdn.microsoft.com/Forums/en-US/microsofttranslator/thread/775fd59
2-9d40-4ca5-85a4-32ac39967c98

Original comment by balazs.endresz on 3 Dec 2011 at 2:02

[GoogleCodeExporter]

Hello all, is jquery-translate latest version from trunk working with Microsoft 
Translator? There is any working example to use it? where do i need to put my 
appid and key?

thanks in advance, regards

Original comment by fid...@gmail.com on 6 Dec 2011 at 7:27

[GoogleCodeExporter]

Hello developers, 

First of all thank you for best Project and excellent code.

With Bing I have much errors so I added some modification to code, now all look 
like all work.

developer may check self file with help Diff, and may be some thing write more 
best. Sorry not all place in code was clear.

1. In bing after translate I have white spaces between </div> and next <div> 
For solution I added to _process: function()
below row:
   this.rawTranslation = this.rawTranslation.replace(/<\/div>\W<div>/gim,'</div><div>') ;

2. some text was bad formated so I replace in function "_translate" line:
data: {appId: key, from: _this.from, to: _this.to, contentType: "text/plain", 
text: src },
to 
data: {appId: key, from: _this.from, to: _this.to, contentType: "text/html", 
text: src },

3. because all continue not work 
I return old version code in function "_process" I replace so line (may be I 
not right)
if(j++ < n)

replace to
  if(j < n)

4. All at top give some result but all continue not work.
so I go to function "_init"

And replace below code:
            substr = this.truncate( this.rawSource.substr(lastpos), o.limit);
             if(!substr) break;
             this.rawSources.push(substr);
             lastpos += substr.length;

to code:

            size_for_translat = 0 ;
            substr = "";
            for(var kk = 0; kk < this.source.length; kk++){
              text_for_translate = '<div>'+this.source[kk]+'</div>';
              len = text_for_translate.length ; 
              if( (size_for_translat + len ) < o.limit )  {
                size_for_translat = size_for_translat + len;
                substr += text_for_translate ; 
              } else {

                this.rawSources.push(substr);
                size_for_translat = len;
                substr = text_for_translate ; 
              }
            }

            if(substr) this.rawSources.push(substr);
            break;

after that all start work good. So may be 1. 2. 3. points may return to 
original state.

In attachment my file result:
Every developer may check self file with help Diff, and may be some thing write 
more best. 

At end small promotion: at ordasoft.com - you may download work version for 
site translate build on joomla CMS and yours code. Project sef Translate.

Thank you very much all !
Andrew

Original comment by akb...@gmail.com on 6 Dec 2011 at 8:08

Attachments:

• jquery.translate.js

[GoogleCodeExporter]

@akbett: Thank you for the diff, I'll merge it on Saturday at last, I'm not 
going to have internet at home until next week.

Original comment by balazs.endresz on 7 Dec 2011 at 7:34

[GoogleCodeExporter]

@akbett: 

Thanks Andrew!  I've seen much improvement with your fixes.  Thanks to everyone 
who's working on this lifesaver!

Original comment by elo...@gmail.com on 13 Dec 2011 at 4:44

[GoogleCodeExporter]

I must say that Andrew has saved my life!  His changes made my totally screwed 
up looking Bing translated pages look normal again!

Just so people know the process, I grabbed the latest Source version from the 
Download Builder (Trunk). I was using 1.3.9 previously.

I'm also now loading jQuery 1.7.1 vs 1.3.2 previously.

I had to change the translate.ui syntax to be 1.4.x compliant.

I added our Bing AppID before running the jQuery.translate function and the 
translation worked for my app. However, as others have stated, the html was all 
messed-up.

I went through Andrew's changes and made them on my local version of the plugin 
source, ran it and it's all fixed!

Thank you Andrew!!

Original comment by rlindab...@gmail.com on 13 Dec 2011 at 9:31

[GoogleCodeExporter]

Hi for this works ok , how i must put the code :

<script type="text/javascript" 
src="http://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.min.js"></script>
<script src="jquery.translate.js"></script>

And into the body tag -----

<script>
$("body").translate("en", "es", {limit:Infinity});
</script>

But no works and by other side where i must put the API -ID for use bing 

Can you put simple code for translate body of website ? , thank´s regards 

Original comment by sexcamtf...@gmail.com on 22 Dec 2011 at 5:03

[GoogleCodeExporter]

Everything just stopped working a while ago, any news?

Original comment by eatf...@gmail.com on 19 Jan 2012 at 9:58

[GoogleCodeExporter]

Hello there, at this point I must say: Thanks Andrew for Your effort and fixes 
in source code!

Do You have any problems with Internet Explorer 7 and this translate plugin 
(Bing version)? In wireshark I see that IE7 sends two request (last one to 
"V2/Ajax.svc/GetLanguageNames"), get HTTP 200OK and that's it - nothing else 
happens...

Do Andrew, or anyone, knows where I should search for problem in source code?

Original comment by krypton...@gmail.com on 23 Jan 2012 at 11:01

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Ok, I solved problem with IE7. I saw that ajax request, which handles main 
translating work (in Bing option it is that with URL 
http://api.microsofttranslator.com/V2/Ajax.svc/Translate), is not send because 
of... too long GET request (and I'm translating a lot of text).

Solution: simply add limit to Your translate options, for example "limit: 1000" 
(default if 1750).

Original comment by krypton...@gmail.com on 26 Jan 2012 at 10:55

[GoogleCodeExporter]

Hi,
I use jquery translete only for plain text and i run in this example:
$.translate( 'Šiaudų granulės ', { 
          from:'lt', 
          to:'en', 
          complete: function(translation){
                console.log(translation);
            }
    });

It seems that bing for this text in example (i didn't found more examples) 
wrapped i <div></div> returns "</div> <div> Straw pellets" which freezes 
browser. I use this plugin only for translating plain text so i just changed 
"text/plain" to "text/html" and i works fine now. 
For anybody who has similar problem (his browser freezees or crushes) and 
changing to "text/html" doesn't work for him i think that the problem lies in 
_process: function but i did't look closer to this.  

Original comment by lsobo...@gmail.com on 2 Mar 2012 at 10:21

[GoogleCodeExporter]

I looked closer to this and i think that there is infinite loop for string 
"</div> <div> Straw pellets" in this fragment of _process function:

while( (lastpos = this.rawTranslation.lastIndexOf("</div>", i)) > -1){
            i = lastpos - 1; // i==-1
            subst = this.rawTranslation.substr(0, i + 1); // subst == ""

            divst = subst.match(/<div[> ]/gi);  // divst == null
            divcl = subst.match(/<\/div>/gi); // divcl == null

            divst = divst ? divst.length : 0; // divst == 0
            divcl = divcl ? divcl.length : 0; // divcl == 0

            if(divst !== divcl + 1) continue; // 0 !== 1 ; infinite loop starts
...

Original comment by lsobo...@gmail.com on 2 Mar 2012 at 10:55

[GoogleCodeExporter]

Hi to all,

is there a guide what i must change to work the translation? 

I am using it to this website http://www.psarema-skafos.gr 

My code was

function translateTo( destLang ){
  $('body').translate('el',destLang, {
      not:          '.jq-translate-ui',
      start:        function(){$('#translate_loading').show();},
      complete:     function(){$('#translate_loading').hide();},
          fromOriginal:true
      }
  );
  $('title').translate(destLang)
}

$(document).ready(function(){
        $.translate(function(){
        $('#translate_loading').hide();
        $.translate.ui({
        tags: ['select', 'option'],
        filter: ["el","en", "de", "fr","it","tr"],
                label: $.translate.toNativeLanguage ||
         function(langCode, lang){
           return $.translate.capitalize(lang);
         },
                fromOriginal:true
     })
     .change(function(){
                var lang = $(this).val();
                translateTo(lang);
        $.cookie('destLang', lang, {path:'/'});
        })
        .val('el')
        .appendTo('#translate'); //insert the dropdown to the page
    $.translate.getBranding().appendTo($('#translate'));
        var destLang = $.cookie('destLang');
        if (destLang && destLang != 'el') {
                //alert(destLang);
                $('.jq-translate-ui').val(destLang);
                translateTo( destLang );
     }
    });

});

So i must include the modified version of the plugin for the Bing API. 

And what else; 

Thanks, Michael

Original comment by mbas...@gmail.com on 4 Apr 2012 at 11:01