Cernelius
commented
3 years ago The problem persists after I change all " to ".
Closed tripleabuilderbot closed 3 years ago
Cernelius
commented
3 years ago The problem persists after I change all " to ".
Cernelius
commented
3 years ago As far as myself goes, I got around this issue by first downloading a map and then swapping the map's game file with the one I want, before the program would create the notes file from the game file, so to have the program creating the notes file out of that.
So I'm fine, even though I still cannot understand what was wrong (when I just manually created the notes file myself instead of letting the program do it).
By the way, I had to add back the line breaks (talking about the useless line breaks in the file, which are ignored by the program upon visualizing the notes, not any code creating line breaks) because the program didn't port them to the newly created notes file.
DanVanAtta
commented
3 years ago @Cernelius could you attach the file that was generating this problem?
Cernelius
commented
3 years ago @Cernelius could you attach the file that was generating this problem?
I don't have it any longer, but this problem is the same if I try manually to create the notes file starting from the 270BC Wars game currently in the repository.
(It's a "txt" file because the correct extension could not be uploaded herein, so you need to change it to the correct format.)
DanVanAtta
commented
3 years ago I was able to repro. It looks like the file encoding is ISO-8859. How exactly did you create that game notes file @Cernelius, which application did you use? Did you specify an encoding when saving it?
Cernelius
commented
3 years ago As I said:
their code is pasted inside a blank "270BC_Wars.notes.html"
Meaning that I simply take the "270BC_Wars.notes.html" generated by the program, delete everything which is in it and paste what I want (which doesn't work).
DanVanAtta
commented
3 years ago Which application are you using to open the file and to paste new contents? Which application are you doing the 'copy' from?
Cernelius
commented
3 years ago I think I used http://www.jedit.org/, but I may have used Windows Notepad instead.
Cernelius
commented
3 years ago My jEdit is at version 5.5. I see the latest stable version of it is apparently 5.6.0 instead.
DanVanAtta
commented
3 years ago There are ISO-8859-1 characters in that file, when rendered in Java Swing it looks like the characters will simply not be there. If converting to UTF-8, the characters will potentially be converted to unknown character symbols. Ideally files are saved in UTF-8, it should be an option in the text editor how the file is saved.
With that said, we can add fallback encoding options to read the additional encoding types.
Example, the NBSP below:
Cernelius
commented
3 years ago There are ISO-8859-1 characters in that file, when rendered in Java Swing it looks like the characters will simply not be there. If converting to UTF-8, the characters will potentially be converted to unknown character symbols. Ideally files are saved in UTF-8, it should be an option in the text editor how the file is saved.
With that said, we can add fallback encoding options to read the additional encoding types.
Example, the NBSP below:
I don't understand what those characters are and how you can find that "NBSP" in there. As I said, I just took the game notes of the "270BC Wars" game currently in the repository and copy-pasted them, adding nothing else. As you can see in this screenshot, there is nothing but a space where you are pointing at:
https://raw.githubusercontent.com/triplea-maps/270bc_wars/master/map/games/270BC_Wars.xml
And all works fine for the "xml" file. Only the "html" files has such problems. So it cannot be because of characters created by the program which java swing cannot see, because I create "xml" files with the same editor to be read by the program, unless the editor creates them for "html" files but not for "xml" files.
Cernelius
commented
3 years ago And, as I said
I'm fine, even though I still cannot understand what was wrong
because, so far at least, if I just let the program create the "html" files, and then I edit them pasting nothing from other files, I'm not having any problems. The problems are only if I copy-paste from the "xml" file to the "html" file (which is something I don't need to do after the game notes I want are already in the "html" file).
So, unless I'm missing something, nothing of this should be a problem any longer once past this phase of moving the game notes out of the "xml".
Are there any plans massively to remove all game notes from the "xml" after releasing 2.6 or will useless and increasingly outdated copies of the game notes now in the "html" files be kept within the "xml" game files too?
DanVanAtta
commented
3 years ago The copy/paste into the file, or when you saved it, or a combination of the two, converted the files encoding from ASCII text to ISO-8859. Look for an encoding option when saving the file if there is one.
Notice the below shows the encoding:
dan@dan-xps:~/triplea/downloadedMaps/270bc_wars/map/games$ file *
270BC_Wars.notes.html: HTML document, ISO-8859 text, with very long lines, with CRLF line terminators
270BC_Wars.xml: XML 1.0 document, ASCII text, with very long linesFor reference and comparison:
dan@dan-xps:~/triplea/downloadedMaps/world_at_war/map/games$ file *
WAW40.notes.html: HTML document, ASCII text, with very long lines
WAW40.xml: XML 1.0 document, ASCII text, with very long lines
World_At_War.notes.html: HTML document, ASCII text, with very long lines
World_At_War.xml: XML 1.0 document, ASCII text, with very long linesAre there any plans massively to remove all game notes from the "xml" after releasing 2.6 or will useless and increasingly outdated copies of the game notes now in the "html" files be kept within the "xml" game files too?
No plans to do a bulk removal, but once 2.6 is launched the expectation is for those XML elements to no longer be maintained and perhaps eventually pruned.
Cernelius
commented
3 years ago You know that pruning those has no hard compatibility issues right? The "notes" property is optional, so removing it just leaves a game with no notes.
I'm not a fan of this thing of possibly huge game notes remaining indefinitely in two places (Obviously, I've already removed them from whithin the "xml" file for my `270_bc_wars" map I intend to update on the first of April.). Moreover, it can be confusing with users in 2.5 and lower looking at one game notes and users in 2.6 and higher looking at an other ones (Rather better the ones in 2.5 and lower see no notes at all.). I'm under the impression many users stay without updating TripleA for over a year if not forced by compatibility breakages or major problems.
By the way, the problem I've pointed out of all line breaks being eliminated is not much of a problem for me (I've recreated the line breaks rapidly), but I can see it being generally a substantial issue in the moment you open the notes file to read it and everything is on a single line (Say, for example, you need to fix something in the notes of Civil War and all the notes are a single line.).
Cernelius
commented
3 years ago Previously you said "UTF-8" and now you say "ASCII". Are they the same thing?
Anyways, I've no idea on how to get those information or how to configure jEdit to save in "ASCII".
Cernelius
commented
3 years ago This is what jEdit has to say:
Character Encodings
A character encoding is a mapping from a set of characters to their on-disk representation. jEdit can use any encoding supported by the Java platform.
Buffers in memory are always stored in UTF-16 encoding, which means each character is mapped to an integer between 0 and 65535. UTF-16 is the native encoding supported by Java, and has a large enough range of characters to support most modern languages.
When a buffer is loaded, it is converted from its on-disk representation to UTF-16 using a specified encoding.
The default encoding, used to load files for which no other encoding is specified, can be set in the Encodings pane of the Utilities> Options dialog box; see the section called “The Encodings Pane”. Unless you change this setting, it will be your operating system's native encoding, for example MacRoman on the MacOS, windows-1252 on Windows, and ISO-8859-1 on Unix.
An encoding can be explicitly set when opening a file in the file system browser's Commands>Encoding menu.
Note that there is no general way to auto-detect the encoding used by a file, however jEdit supports "encoding detectors", of which there are some provided in the core, and others may be provided by plugins through the services api. From the encodings option pane the section called “The Encodings Pane”, you can customize which ones are used, and the order they are tried. Here are some of the encoding detectors recognized by jEdit:
BOM: UTF-16 and UTF-8Y files are auto-detected, because they begin with a certain fixed character sequence. Note that plain UTF-8 does not mandate a specific header, and thus cannot be auto-detected, unless the file in question is an XML file.
XML-PI: Encodings used in XML files with an XML PI like the following are auto-detected:
<?xml version="1.0" encoding="UTF-8">
html: Encodings specified in HTML files with a content= attribute in a meta element may be auto-detected:
<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8">
python: Python has its own way of specifying encoding at the top of a file.
# -*- coding: utf-8 -*-
buffer-local-property: Enable buffer-local properties' syntax (see the section called “Buffer-Local Properties”) at the top of the file to specify encoding.
# :encoding=ISO-8859-1:
The encoding that will be used to save the current buffer is shown in the status bar, and can be changed in the Utilities>Buffer Options dialog box. Note that changing this setting has no effect on the buffer's contents; if you opened a file with the wrong encoding and got garbage, you will need to reload it. File>Reload with Encoding is an easy way.
If a file is opened without an explicit encoding specified and it appears in the recent file list, jEdit will use the encoding last used when working with that file; otherwise the default encoding will be used.
Commonly Used Encodings
While the world is slowly converging on UTF-8 and UTF-16 encodings for storing text, a wide range of older encodings are still in widespread use and Java supports most of them.
The simplest character encoding still in use is ASCII, or “American Standard Code for Information Interchange”. ASCII encodes Latin letters used in English, in addition to numbers and a range of punctuation characters. Each ASCII character consists of 7 bits, there is a limit of 128 distinct characters, which makes it unsuitable for anything other than English text. jEdit will load and save files as ASCII if the US-ASCII encoding is used.
Because ASCII is unsuitable for international use, most operating systems use an 8-bit extension of ASCII, with the first 128 values mapped to the ASCII characters, and the rest used to encode accents, umlauts, and various more esoteric used typographical marks. The three major operating systems all extend ASCII in a different way. Files written by Macintosh programs can be read using the MacRoman encoding; Windows text files are usually stored as windows-1252. In the Unix world, the 8859_1 character encoding has found widespread usage.
On Windows, various other encodings, referred to as code pages and identified by number, are used to store non-English text. The corresponding Java encoding name is windows- followed by the code page number, for example windows-850.
Many common cross-platform international character sets are also supported; KOI8_R for Russian text, Big5 and GBK for Chinese, and SJIS for Japanese.Notice that it says
The default encoding, used to load files for which no other encoding is specified, can be set in the Encodings pane of the Utilities> Options dialog box; see the section called “The Encodings Pane”. Unless you change this setting, it will be your operating system's native encoding, for example MacRoman on the MacOS, windows-1252 on Windows, and ISO-8859-1 on Unix.
I'm on Windows and you say that the file I saved has been turned into ISO-8859, which is what should happen in Unix not in Windows. My guess is that the editor sees there are Unix characters, so it goes for ISO-8859 even though I'm on Windows.
Definitely nothing problematic is happening by just opening the file with jEdit and saving it since I did this many times. This problem does not happen if I let the "html" file be created, and then I open and modify it with jEdit by writing myself. It happens only if I let the "html" file be created and copy something from the "xml" file and then open the "html" file and paste into it.
Cernelius
commented
3 years ago Maybe it would be better not using any fallback and, instead, forcing all TripleA files to have the same encoding, documenting it for map-makers and having an error giving good information on how to make the file work?
For example, what should be better set in this for working with TripleA files?
Any suggestions?
Cernelius
commented
3 years ago By the way, I think a gaming program should not go ahead like this creating any map files, thus substantially changing the maps. Anything like this belongs to map-updating utilities for map-makers, not for any users.
As far as the program goes, I believe it should either simply display no game notes for old maps or (if backward support is preferred) keep supporting the old game notes within the "xml" file as a fall back if no notes file is found.
Moreover, I wonder if such a thing (not displaying the game notes anymore for old maps unless updated) should be considered compatibility breaking and the current TripleA should rather be versioned 3.0.
User Description
The notes for this game works fine in 2.5 yet cannot display if their code is pasted inside a blank "270BC_Wars.notes.html"
Here are some lines of the start and end of them:
...
Log Message
Error reading file: C:\Users\001\triplea\downloadedMaps\270bc_wars\map\games\270BC_Wars.notes.html, Input length = 1
TripleA Version
2.6.89
Java Version
11.0.9.1
Operating System
Windows 10
Stack Trace