Encoding issue in files

[CyrilleB79]

@michaelDCurran please see issue description here: https://groups.io/g/nvda-translations/topic/encoding_issue_with_new/104286915?p=,,,100,0,0,0::recentpostdate/sticky,,,100,2,0,104286915,previd%3D1707607888611448718,nextid%3D1683926426443401204&previd=1707607888611448718&nextid=1683926426443401204

Thanks

[michaelDCurran]

Unfortunately I cannot so far find the cause of the issue.

All I know is that the "updated PO files" commits are made by ikiwiki, specifically its compiled post-commit hook. I'm not quite sure where to find the source code for this.

Reading the ikiwiki.setup config file, it looks like we are specifying utf8 correctly.

Also, all this config and code is identical to what was on the old server. Including the same version of ikiwiki.

Do you know how far the problem extends? Have you found any other strings that clearly have a wrong encoding?

 Looking at the original .ndwn file (mp3Directcut.mdwn), I don't see that encoding problem. So it seems to be happening as ikiwiki is generating / merging the po file. And other po files for mp3DirectCut also have the issue.

I'm not quite sure what to do next...

Note that we do plan to replace the add-on website with a light-weight web interface auto-generated from the add-on store. But we probably can't get to this until 2024.1 is pretty much complete.

[CyrilleB79]

I have not checked this, but I fear that any documentation/website file containing non-ASCII character and generated since the translation server was restarted have the issue.

For a simple new (just created file), see customNotifications.fr.po.

The consequence is that the impacted string is not translated on the website despite being translated on SVN. See the Custom Notifications French page or the mp3DirectCut French page for example.

Is the ikiwiki version on the new server the same as on the old one?

Maybe the impact is not so big, given the non-ASCII characters in the English source documentation are probably not common. In addition to authors/contributors names using non-ASCII characters, there may be non-ASCII characters in the rest of the documentation, e.g. braille characters in Access8Math documentation.

So, it would be nice to have this fixed before NVDA 2024.1 so that compatible add-ons contain a fully translated documentation.

[CyrilleB79]

Cc @BlindHelp who has initially reported the issue. Thanks!

[michaelDCurran]

I think I have tracked it down to a bug in p4a 0.58+: https://ikiwiki.info/bugs/po:_buggy_UTF-8_support_with_po4a_0.58+/ Our main server has po4a 0.5.7, but it seems the l10n server is po4a 0.6.6. I'll look into trying to get this downgraded back to 0.5.7.

[michaelDCurran]

I think this has been fixed in svn revision 77881. @CyrilleB79 Could you please confirm?

[CyrilleB79]

I have had a quick look at French documentation for mp3DirectCut, Access8Math and CustomNotification. The issue seems fixed.

@BlindHelp can you confirm that the issue is fixed?

[seanbudd]

Closing as fixed given no further feedback