[Feature] Unicode characters in text file [sf#3]

[teambob]

Reported by andrewpunch on 2004-06-09 03:25 UTC Characters which are non-ASCII are thrown away when writing to a text file.

There is no way around this while we write to an ASCII file.

There are some other options for file formats:

• mapping to a code page (e.g. ANSI)
• quoted printable (same as email)
• UTF8
• Straight Unicode

From a design perspective this could be achieved by creating maps from a single unicode character to one or more bytes.

There could be a map for:

• ASCII
• ANSI
• iso8859 standards
• UTF8
• straight unicode

The map need not be static. It may be dynamic. For example the ASCII map may allow through all character codes with a unicode value less than 0x0080.

There must be a process for when a unicode character is not mappable using the current map.

Created on behalf of David at Nutmeg.

[teambob]

Commented by andrewpunch on 2004-06-10 10:50 UTC Logged In: YES user_id=928005

Another quick possibility is to use UTF-32 (little endian) encoding. This allows access to all the characters in a document without loss of information.

The technical specification is here: http://www.unicode.org/faq/specifications-jda.html

[teambob]

Commented by andrewpunch on 2004-10-11 06:43 UTC Logged In: YES user_id=928005

Detirmination: Text writer will output UTF8 by default in next version. This will be compatible with ASCII for english characters, but will keep other characters.

ASCII, UTF16/32 and other mappings will be available as options in later versions.

[teambob]

Updated by andrewpunch on 2004-10-11 06:43 UTC

• summary: Non-ASCII characters in text file --> Unicode characters in text file

[teambob]

Updated by andrewpunch on 2004-10-11 06:45 UTC

• priority: 5 --> 8

[teambob]

Commented by andrewpunch on 2005-04-11 12:41 UTC Logged In: YES user_id=928005

This is scheduled for inclusion in 3.2.0 as UTF8 output for "text" files.