Fix accented character display issues on different computers

jackreid commented 4 years ago

Based on both past experiences with Shea running the code and with his more recent screenshots, there are some issues with displaying certain accent characters across different computers. Maybe this is an operating system issue? There must be someway to have accented characters in translations.csv that render properly.

@seamuslo , if you have more insight into why this occurs, let me know. Otherwise I will investigate myself when I get a chance. Obviously testing any fix will require you to run it.

seamuslo commented 4 years ago

@jackreid can you remind me where in the code or the data spreadsheets the labels for the graph dropdowns are pulled from? Doing ctrl+F in SD_UI, SD_lib, the temporal spreadsheet, and the shapefields spreadsheet couldn't find "..infectada medida"

jackreid commented 4 years ago

In the code itself is all English. The graph dropdowns are based on the longform names of each SD Object generated by SD_Map in SDlib. So if you look in translations.csv for a given Spanish or Portuguese phrase, that will show you what the English equivalent is. In this case, infectada medida is Measured Infected Population, I believe.

seamuslo commented 3 years ago

So looking at the translations.csv file, this is what I see: Would you be able to: 1) send me your translations.csv file via email (in case somehow its journey through github is what's messing it up) 2) send me a screenshot of your translations.csv so if I have to manually correct the accented characters I don't have to rely on google translate?

seamuslo commented 3 years ago

To follow up on your email, trying to swap in the translations.csv file you emailed me does not fix the issue.

To answer your question, this issue does show up in the Vida UI, but not in a text editor like notepad

jackreid commented 3 years ago

Well that's weird. I have been unable to find anything online that points to the issue. Could you try replacing a few of the problematic entries in translations.csv so that they appear properly for you and then pushing? I can try it and see if it works properly Windows->Linux even if it doesn't work in the current Linux->Windows configuration.

jackreid commented 3 years ago

@seamuslo I have a couple of of follow-up questions to try and resolve this:

Could you open up translations.csv in Notepad, use Save As, and select a different encoding (specifically ANSI, but if it doesn't work, try the others)? Also tell me what it defaults to. See if this fixes the issue in either excel or tkinter.
In Python, could you run the following code and send me the results?

from tkinter import font, Tk
root = Tk()
print(font.families())

seamuslo commented 3 years ago

To answer your questions:

when opening in notepad, it defaults to UTF-8. When using "save as" to encode it as ANSI, that clears up the issue with the weird characters when opened in Excel. However, when I try to run SD_UI with the newly encoded translations file, I'll get the following error:

Traceback (most recent call last): File "c:/Users/slomb/OneDrive/Desktop/SD_UI/SD_UI_v1_5.py", line 1198, in UI = SD_UI(tuning = 0, File "c:/Users/slomb/OneDrive/Desktop/SD_UI/SD_UI_v1_5.py", line 116, in init self.title(self.translate('System Dynamics Visualization'))
File "c:/Users/slomb/OneDrive/Desktop/SD_UI/SD_UI_v1_5.py", line 256, in translate if row[in_lang] == phrase: KeyError: 'english'

As for running the code you asked, here is the (quite long) output: (.venv) PS C:\Users\slomb\OneDrive\Desktop\SD_UI> & c:/Users/slomb/OneDrive/Desktop/SD_UI/.venv/Scripts/python.exe c:/Users/slomb/OneDrive/Desktop/SD_UI/tkintertest.py ('System', 'Terminal', 'Fixedsys', 'Modern', 'Roman', 'Script', 'Courier', 'MS Serif', 'MS Sans Serif', 'Small Fonts', 'Adobe Heiti Std R', '@Adobe Heiti Std R', 'Adobe Ming Std L', '@Adobe Ming Std L', 'Adobe Myungjo Std M', '@Adobe Myungjo Std M', 'Adobe Pi Std', 'Adobe Song Std L', '@Adobe Song Std L', 'Courier Std', 'Kozuka Gothic Pr6N M', '@Kozuka Gothic Pr6N M', 'Kozuka Mincho Pr6N R', '@Kozuka Mincho Pr6N R', 'Myriad CAD', 'DS ISO 1', 'TeamViewer15', 'Apex New Medium', 'Jaapokki', 'Jaapokki enchance', 'Jaapokki subtract', 'Marlett', 'Arial', 'Arabic Transparent', 'Arial Baltic', 'Arial CE', 'Arial CYR', 'Arial Greek', 'Arial TUR', 'Arial Black', 'Bahnschrift Light', 'Bahnschrift SemiLight', 'Bahnschrift', 'Bahnschrift SemiBold', 'Bahnschrift Light SemiCondensed', 'Bahnschrift SemiLight SemiConde', 'Bahnschrift SemiCondensed', 'Bahnschrift SemiBold SemiConden', 'Bahnschrift Light Condensed', 'Bahnschrift SemiLight Condensed', 'Bahnschrift Condensed', 'Bahnschrift SemiBold Condensed', 'Calibri', 'Calibri Light', 'Cambria', 'Cambria Math', 'Candara', 'Candara Light', 'Comic Sans MS', 'Consolas', 'Constantia', 'Corbel', 'Corbel Light', 'Courier New', 'Courier New Baltic', 'Courier New CE', 'Courier New CYR', 'Courier New Greek', 'Courier New TUR', 'Ebrima', 'Franklin Gothic Medium', 'Gabriola', 'Gadugi', 'Georgia', 'Impact', 'Ink Free', 'Javanese Text', 'Leelawadee UI', 'Leelawadee UI Semilight', 'Lucida Console', 'Lucida Sans Unicode', 'Malgun Gothic', '@Malgun Gothic', 'Malgun Gothic Semilight', '@Malgun Gothic Semilight', 'Microsoft Himalaya', 'Microsoft JhengHei', '@Microsoft JhengHei', 'Microsoft JhengHei UI', '@Microsoft JhengHei UI', 'Microsoft JhengHei Light', '@Microsoft JhengHei Light', 'Microsoft JhengHei UI Light', '@Microsoft JhengHei UI Light', 'Microsoft New Tai Lue', 'Microsoft PhagsPa', 'Microsoft Sans Serif', 'Microsoft Tai Le', 'Microsoft YaHei', '@Microsoft YaHei', 'Microsoft YaHei UI', '@Microsoft YaHei UI', 'Microsoft YaHei Light', '@Microsoft YaHei Light', 'Microsoft YaHei UI Light', '@Microsoft YaHei UI Light', 'Microsoft Yi Baiti', 'MingLiU-ExtB', '@MingLiU-ExtB', 'PMingLiU-ExtB', '@PMingLiU-ExtB', 'MingLiU_HKSCS-ExtB', '@MingLiU_HKSCS-ExtB', 'Mongolian Baiti', 'MS Gothic', '@MS Gothic', 'MS UI Gothic', '@MS UI Gothic', 'MS PGothic', '@MS PGothic', 'MV Boli', 'Myanmar Text', 'Nirmala UI', 'Nirmala UI Semilight', 'Palatino Linotype', 'Segoe MDL2 Assets', 'Segoe Print', 'Segoe Script', 'Segoe UI', 'Segoe UI Black', 'Segoe UI Emoji', 'Segoe UI Historic', 'Segoe UI Light', 'Segoe UI Semibold', 'Segoe UI Semilight', 'Segoe UI Symbol', 'SimSun', '@SimSun', 'NSimSun', '@NSimSun', 'SimSun-ExtB', '@SimSun-ExtB', 'Sitka Small', 'Sitka Text', 'Sitka Subheading', 'Sitka Heading', 'Sitka Display', 'Sitka Banner', 'Sylfaen', 'Symbol', 'Tahoma', 'Times New Roman', 'Times New Roman Baltic', 'Times New Roman CE', 'Times New Roman CYR', 'Times New Roman Greek', 'Times New Roman TUR', 'Trebuchet MS', 'Verdana', 'Webdings', 'Wingdings', 'Yu Gothic', '@Yu Gothic', 'Yu Gothic UI', '@Yu Gothic UI', 'Yu Gothic UI Semibold', '@Yu Gothic UI Semibold', 'Yu Gothic Light', '@Yu Gothic Light', 'Yu Gothic UI Light', '@Yu Gothic UI Light', 'Yu Gothic Medium', '@Yu Gothic Medium', 'Yu Gothic UI Semilight', '@Yu Gothic UI Semilight', 'HoloLens MDL2 Assets', 'Agency FB', 'Algerian', 'Book Antiqua', 'Arial Narrow', 'Arial Rounded MT Bold', 'Baskerville Old Face', 'Bauhaus 93', 'Bell MT', 'Bernard MT Condensed', 'Bodoni MT', 'Bodoni MT Black', 'Bodoni MT Condensed', 'Bodoni MT Poster Compressed', 'Bookman Old Style', 'Bradley Hand ITC', 'Britannic Bold', 'Berlin Sans FB', 'Berlin Sans FB Demi', 'Broadway', 'Brush Script MT', 'Bookshelf Symbol 7', 'Californian FB', 'Calisto MT', 'Castellar', 'Century Schoolbook', 'Centaur', 'Century', 'Chiller', 'Colonna MT', 'Cooper Black', 'Copperplate Gothic Bold', 'Copperplate Gothic Light', 'Curlz MT', 'Dubai', 'Dubai Light', 'Dubai Medium', 'Elephant', 'Engravers MT', 'Eras Bold ITC', 'Eras Demi ITC', 'Eras Light ITC', 'Eras Medium ITC', 'Felix Titling', 'Forte', 'Franklin Gothic Book', 'Franklin Gothic Demi', 'Franklin Gothic Demi Cond', 'Franklin Gothic Heavy', 'Franklin Gothic Medium Cond', 'Freestyle Script', 'French Script MT', 'Footlight MT Light', 'Garamond', 'Gigi', 'Gill Sans MT', 'Gill Sans MT Condensed', 'Gill Sans Ultra Bold Condensed', 'Gill Sans Ultra Bold', 'Gloucester MT Extra Condensed', 'Gill Sans MT Ext Condensed Bold', 'Century Gothic', 'Goudy Old Style', 'Goudy Stout', 'Harlow Solid Italic', 'Harrington', 'Haettenschweiler', 'High Tower Text', 'Imprint MT Shadow', 'Informal Roman', 'Blackadder ITC', 'Edwardian Script ITC', 'Kristen ITC', 'Jokerman', 'Juice ITC', 'Kunstler Script', 'Wide Latin', 'Lucida Bright', 'Lucida Calligraphy', 'Lucida Fax', 'Lucida Handwriting', 'Lucida Sans', 'Lucida Sans Typewriter', 'Magneto', 'Maiandra GD', 'Matura MT Script Capitals', 'Mistral', 'Modern No. 20', 'Monotype Corsiva', 'MT Extra', 'Niagara Engraved', 'Niagara Solid', 'OCR A Extended', 'Old English Text MT', 'Onyx', 'MS Outlook', 'Palace Script MT', 'Papyrus', 'Parchment', 'Perpetua', 'Perpetua Titling MT', 'Playbill', 'Poor Richard', 'Pristina', 'Rage Italic', 'Ravie', 'MS Reference Sans Serif', 'MS Reference Specialty', 'Rockwell Condensed', 'Rockwell', 'Rockwell Extra Bold', 'Script MT Bold', 'Showcard Gothic', 'Snap ITC', 'Stencil', 'Tw Cen MT', 'Tw Cen MT Condensed', 'Tw Cen MT Condensed Extra Bold', 'Tempus Sans ITC', 'Viner Hand ITC', 'Vivaldi', 'Vladimir Script', 'Wingdings 2', 'Wingdings 3', 'HYSWLongFangSong', '@HYSWLongFangSong', 'SWAstro', 'OLF SimpleSansOC', 'SWComp', 'SWGothe', 'SWGothg', 'SWGothi', 'SWGrekc', 'SWGreks', 'SWIsop1', 'SWIsop2', 'SWIsop3', 'SWIsot1', 'SWIsot2', 'SWIsot3', 'SWItal', 'SWItalc', 'SWItalt', 'SWMap', 'SWMath', 'SWMeteo', 'SWMono', 'SWMusic', 'SWRomnc', 'SWRomnd', 'SWRomns', 'SWRomnt', 'SWScrpc', 'SWScrps', 'SWSimp', 'SWTxt', 'SWGDT', 'SWLink', '3ds', '3DS Fonticon', 'Courier10 BT', 'Dutch801 Rm BT', 'UniversalMath1 BT', 'Monospac821 BT', 'Swis721 BdOul BT', 'Swis721 BT', 'Swis721 LtCn BT', 'CATIA Symbols', 'SymbolMono BT', 'SymbolProp BT', 'ESRI SDS 2.00 1', 'ESRI SDS 1.95 2', 'ESRI SDS 1.95 1', 'ESRI NIMA DNC LN', 'ESRI Ordnance Survey', 'ESRI NIMA DNC PT', 'ESRI Climate & Precipitation', 'ESRI ERS Operations S1', 'ESRI Cartography', 'ESRI SDS 2.00 2', 'ESRI NIMA VMAP1&2 PT', 'ESRI Arrowhead', 'ESRI Surveyor', 'ESRI NIMA VMAP1&2 LN', 'ESRI AMFM Electric', 'ESRI AMFM Gas', 'ESRI AMFM Sewer', 'ESRI AMFM Water', 'ESRI Hydrants', 'ESRI NIMA City Graphic PT', 'ESRI Telecom', 'ESRI Pipeline US 1', 'ESRI NIMA City Graphic LN', 'ESRI MilMod 02', 'ESRI Environmental & Icons', 'ESRI MilRed 01', 'ESRI MilSym 01', 'ESRI IGL Font16', 'ESRI MilSym 02', 'ESRI IGL Font20', 'ESRI IGL Font21', 'ESRI IGL Font22', 'ESRI IGL Font23', 'ESRI IGL Font24', 'ESRI IGL Font25', 'ESRI MilSym 03', 'ESRI MilSym 04', 'ESRI MilSym 05', 'ESRI MilMod 01', 'ESRI Public1', 'ESRI Geometric Symbols', 'ESRI ArcPad', 'ESRI Default Marker', 'ESRI Transportation & Civic', 'ESRI Caves 1', 'ESRI Caves 2', 'ESRI Caves 3', 'ESRI Dimensioning', 'ESRI Oil, Gas, & Water', 'ESRI North', 'ESRI Geology USGS 95-525', 'ESRI Elements', 'ESRI Commodities', 'ESRI Shields', 'ESRI US MUTCD 1', 'ESRI US MUTCD 2', 'ESRI US MUTCD 3', 'ESRI Weather', 'ESRI Geology AGSO 1', 'ESRI US Forestry 1', 'ESRI US Forestry 2', 'ESRI Geology', 'ESRI Conservation', 'ESRI Crime Analysis', 'ESRI Enviro Hazard Incident', 'ESRI Enviro Hazard Analysis', 'ESRI Enviro Hazard Sites', 'ESRI Hazardous Materials', 'ESRI Fire Incident NFPA', 'ESRI Business', 'ESRI ERS Infrastructures S1', 'ESRI Mil2525C Modifiers', 'ESRI Meteorological 01', 'DIN', 'Apex New', 'Venus Rising', 'Strasua', 'Gunship', 'CMU Typewriter Text Variable Wi')

jackreid commented 3 years ago

Okay, well that is progress. Could you try using the ANSI-encoded csv and trying something additional. Just above the line that causes the error, there is a line (244 in my version at the moment) that says with open(self.translations) as csv_file:

Could you replace it with with open(self.translations, encoding='ANSI') as csv_file:

If that doesn't work, also try replacing 'ANSI' with 'ISO-8859-1', 'ISO-8859-1', 'mbcs', and 'Windows-1250-1258'

seamuslo commented 3 years ago

I did the replacement of the line that you mentioned, and got the same error for 'ANSI','ISO-8859-1', 'ISO-8859-1', 'mbcs', : File "c:/Users/slomb/OneDrive/Desktop/SD_UI/SD_UI_v1_5.py", line 1191, in UI = SD_UI(tuning = 0, File "c:/Users/slomb/OneDrive/Desktop/SD_UI/SD_UI_v1_5.py", line 111, in init self.title(self.translate('System Dynamics Visualization'))
File "c:/Users/slomb/OneDrive/Desktop/SD_UI/SD_UI_v1_5.py", line 249, in translate if row[in_lang] == phrase: KeyError: 'english'

For 'Windows-1250-1258', I got a different error: File "c:/Users/slomb/OneDrive/Desktop/SD_UI/SD_UI_v1_5.py", line 1191, in UI = SD_UI(tuning = 0, File "c:/Users/slomb/OneDrive/Desktop/SD_UI/SD_UI_v1_5.py", line 111, in init self.title(self.translate('System Dynamics Visualization')) File "c:/Users/slomb/OneDrive/Desktop/SD_UI/SD_UI_v1_5.py", line 245, in translate with open(self.translations, encoding='Windows-1250-1258') as csv_file: LookupError: unknown encoding: Windows-1250-1258

jackreid commented 3 years ago

Bleh. Could you try replacing some subset of the accented characters in the original translations.csv such that it properly renders on yours (you could do all of them if you really wanted) and then push that? It can be on some other branch if you would like. If I compare what works on yours with how mine are usually written, maybe I can figure out a way to just write the working ones on my computer.

seamuslo commented 3 years ago

Okay, I dug a little deeper and I got it to work, but it required a few extra steps. To elaborate:

I noticed that when I used Notepad to save with the ANSI encoding, it got rid of the erroneous characters, but it did not separate the sheet into the 3 column format that the original CSV. old translations with the original encoding: new ANSI version:

If instead of opening the newly encoding ANSI file with the "open with>excel" functionality, I first open excel, then open the ANSI encoded text file, use the commas to format it into 3 different columns, it instead looks like this:

When I use these newly formatted CSV, it works within the Vida prototype and the accent issue seems to be resolved:

I can go through this procedure from now on every time the translations document is updated if we think this is the best solution.

I accidentally pushed it to Master (my bad), but if it causes issues on your end, feel free to revert back to the last commit. This change is also now implemented on my Shealombardo branch.

jackreid commented 3 years ago

This all makes sense. The one issue is that the 'ANSI' encoding apparently doesn't exist on my computer. An equivalent one, 'ISO-8859-15' does, however. Could you check if you have that one on yours? If so, we can just use 'ISO-8859-15'. If not, we can just have a conditional that checks for whichever one a computer has and uses that one.

I will make sure to keep saving the csv to 'ISO-8859-15' encoding whenever I update it and hopefully we won't have this problem anymore.

seamuslo commented 3 years ago

I used that new encoding and it works for me! I pushed it to my branch. I'm making some other Indonesia updates today, when I finish those, I'll merge back into Master.

jackreid commented 3 years ago

Excellent it seems like this is all resolved, so I will close it.

mitmedialab / Vida_Modeling

Fix accented character display issues on different computers #21