XML character issues in file handling

[GoogleCodeExporter]

From Jeff (Sutekh on VEKN.Presence.org):

I have encountered a bug, and I have tried to do as much troubleshooting as
possible to save you time.  It involves the deck building utility.  It
appears that certain cards with non-English characters like Día de los
Muertos, Rötshreck, Mokolé Blood and Auto-da-Fé cause the decks to become
inaccessible after they are saved as XML files and then closed.  All decks
without special characters work just fine until such a card is added.

If I open an existing deck with a card that has a non-English character, I
get an error that says "Couldn't find card [Card Name]" with the
non-English characters in the card name replaced with garbage characters. 
I have attached a JPEG with the error that I got opening an existing deck
that had a Grooming the Protégé in it.  I am able to bypass the error,
however, the card in question is removed from the deck.  If I re-add the
card, I can manipulate and save the deck as often as I want, but as soon as
I close it, it will never open again.  If I create a deck that has one of
those problem cards, the effect is pretty much the same.  I can manipulate
and save the deck as much as I want, but once it is closed, it does not
re-open.  Either way, I get the error "An error occurred while opening
[Path to Deck]\[Deck Name]."  I have attached an example JPEG of that, as
well.  I have noticed that IE will no longer open the XML files, either,
due to the invalid characters.  The previous version of ARDB won't open the
"corrupted" decks, either.

Oddly enough, looking at the source code of a deck with a card that has
non-English characters made by the previous version of ARDB reveals that
ARDB saved the XML with the special character.  I would have thought XML
was choking on the character and, thusly, ARDB was somehow doing a
translation to avoid that, but that is not the case. 

Original issue reported on code.google.com by graham.r...@gmail.com on 6 Nov 2007 at 7:19

[GoogleCodeExporter]

This is a problem in libxml parser.c under Windows.  Could someone with a Linux 
build
check what happens when they try open the attached file?

Original comment by graham.r...@gmail.com on 6 Nov 2007 at 10:47

Attachments:

• test.xml

[GoogleCodeExporter]

on my linux box (Debian Etch):
{{{
Erreur lors du chargement de la feuille de style : 
(null)file:///tmp/deck2html_eldb.xsl
}}}

If i remove the XSTL stylesheet tag the XML is displayed as UTF-8 but Zöe is 
not
displayed correctly
 :(

Original comment by meshee.k...@gmail.com on 6 Nov 2007 at 11:04

• Changed state: Started

Attachments:

• xml-issue.jpg

[GoogleCodeExporter]

oops i open it with firefox... tomorrow i will compile ardb on ubuntu and 
hopefully
i'll ba able to do your test

Original comment by meshee.k...@gmail.com on 6 Nov 2007 at 11:18

[GoogleCodeExporter]

I opened test.xml with a Suse version of ardb and got the following error 
message:

test.xml:12: parser error : Input is not proper UTF-8, indicate encoding !
Bytes: 0xF6 0x65 0x3C 0x2F
        <name>Z�e</name>
               ^

Original comment by sven.hel...@gmail.com on 7 Nov 2007 at 10:58

[GoogleCodeExporter]

If encoding is changed to <?xml version="1.0" encoding="iso-8859-1"?> XML tools 
can
open the file and extract the XML data.  Looking through the deck model code it 
seems
Ardb sets the encoding to utf-8 or nothing.  I've debugged through wxwindows on
Windows and it seems I need to build it with unicode support.  If I rebuild it 
and
then check all the encodings in the deck model class it may all work!  But who 
knows.

Original comment by graham.r...@gmail.com on 7 Nov 2007 at 11:04

[GoogleCodeExporter]

from wxARDN/INSTALL:

How to Install
=================

This program requires
libxml, libxslt, libsqlite, wxwidgets WITH UNICODE SUPPORT
(for linux, wxGTK2 is recommended)

Hope it helps

Original comment by meshee.k...@gmail.com on 7 Nov 2007 at 11:34

[GoogleCodeExporter]

I tried to build unicode versions of everything of last night but after 4 hours 
I
could still get the libs built correctly to build ardb!  I'm going to retry this
weekend but if I have no success after that should we change the file format 
that we
save decks in?  Make it a simple text format rather then one that relies on an 
XML
processor and unicode?

Original comment by graham.r...@gmail.com on 9 Nov 2007 at 10:34

[GoogleCodeExporter]

The root cause of the problem is to do with how wxWindows is translating between
different character sets.  I'm not an export on this and after spending 12 
hours on
the problem I've fixed it by removing accented characters and replacing them 
with
non-accented equivalents.  This of course breaks old decks but means new decks 
can be
saved and opened.  I will add a new issue for developing a util for converting 
across
old decks.

Original comment by graham.r...@gmail.com on 10 Nov 2007 at 1:37

• Changed state: Fixed

[GoogleCodeExporter]

Original comment by meshee.k...@gmail.com on 14 Nov 2007 at 8:22

• Added labels: Milestone-Ardb2.0