ObsidianX / 3dstools

Tools for extracting and packing resources found in 3DS games
51 stars 15 forks source link

msbt.py Using a MSBT containing the Nintendo private-used 3DS Home Menu symbol leads to UnicodeEncodeError: 'ascii' codec can't encode character u'\ue073' #34

Open schrmh opened 1 year ago

schrmh commented 1 year ago

When runniny msbt.py with python2, I get the following error on a EU_English menu_msbt.lz (which is effectively a MsgStdBn/MSBT file) which is part of the european 3DS Home Menu:

Traceback (most recent call last):
  File "msbt.py", line 624, in <module>
    msbt.to_json(args.json)
  File "msbt.py", line 189, in to_json
    json.dump(output, open(filename, 'w'), indent=2, sort_keys=True, ensure_ascii=False)
  File "/usr/lib/python2.7/json/__init__.py", line 190, in dump
    fp.write(chunk)
UnicodeEncodeError: 'ascii' codec can't encode character u'\ue073' in position 9: ordinal not in range(128)

This character is within the Unicode Private Use Area and shows the 3DS home menu icon (e.g. displayed on the physical home menu button of each 3DS) when displayed on a 3DS.

After that error, the output .json looks like this:

{
  "strings": {
    "base_2b_cancel": [
      "Back"
    ], 
    "base_2b_newmake": [
      "Create"
    ], 
    "create_bu": [
      "Save Data Backup"
    ], 
    "dat_bu_notitle": [
      "Could not display"
    ], 
    "dat_bu_sub0": [
      "Select a save location."
    ], 
    "dat_dlg_bu_brkn": [
      "The backup data management\ninformation is corrupted.\nWould you like to create the\ninformation again?\n\u000e\u0001", 
      "\u0002UBackup data titles may not be\ndisplayed properly afterwards.\u000f\u0001", 
      ""
    ], 
    "dat_dlg_bu_cr0": [
      "The saved backup data\nwill be overwritten.\n\nIs this OK?"
    ], 
    "dat_dlg_bu_cr1": [
      "Saving data...\n\nDo not remove the SD Card\nor turn off the power."
    ], 
    "dat_dlg_bu_cr1_jan": [
      "Saving data...\n\nDo not remove the microSD\ncard or turn off the power."
    ], 
    "dat_dlg_bu_cr1_sac": [
      "Saving data...\n\nDo not remove the microSD\ncard or turn off the power."
    ], 
    "dat_dlg_bu_cr2": [
      "Saved.\n\nSaved backup data can be\nrestored or deleted via\nData Management in the\nSystem Settings."
    ], 
    "dat_dlg_bu_crerr": [
      "Data could not be saved."
    ], 
    "dat_dlg_bu_crng": [
      "Failed to create backup data\nmanagement information."
    ], 
    "dat_dlg_bu_ining": [
      ""
    ], 
    "dat_dlg_bu_iniok": [
      ""
    ], 
    "dat_dlg_bu_init": [
      ""
    ], 
    "dat_dlg_bu_mkext": [
      "Creating save data backup\nmanagement information...\nDo not remove the SD Card\nor turn off the power."
    ], 
    "dat_dlg_bu_mkext_jan": [
      "Creating save data backup\nmanagement information...\n\u000e\u0001", 
      "\u0002ZDo not remove the microSD card\nor turn off the power.\u000f\u0001", 
      ""
    ], 
    "dat_dlg_bu_mkext_sac": [
      "Creating save data backup\nmanagement information...\n\u000e\u0001", 
      "\u0002ZDo not remove the microSD card\nor turn off the power.\u000f\u0001", 
      ""
    ], 
    "dat_dlg_bu_nosav": [
      "There is no save data that can\nbe backed up."
    ], 
    "dat_dlg_bu_re1": [
      "Restoring backup data...\nDo not remove the SD Card\nor turn off the power."
    ], 
    "dat_dlg_bu_re1_jan": [
      "Restoring backup data...\nDo not remove the microSD\ncard or turn off the power."
    ], 
    "dat_dlg_bu_re1_sac": [
      "Restoring backup data...\nDo not remove the microSD\ncard or turn off the power."
    ], 
    "dat_dlg_bu_re2": [
      "The save data has been\nrestored from the backup."
    ], 
    "dat_dlg_bu_re3": [
      "The save data has been\nrestored from the backup.\n\n\u000e\u0001", 
      "\u0002UThe restored backup data\nhas been deleted.\u000f\u0001", 
      ""
    ], 
    "dat_dlg_bu_reerr": [
      "The save data could not be\nrestored from the backup."
    ], 
    "dat_dlg_bu_unsup": [
      ""
    ], 
    "dat_dlg_sd_full": [
      "There is not enough free\nspace on this SD Card."
    ], 
    "dat_dlg_sd_full_jan": [
      "There is not enough free\nspace on this microSD card."
    ], 
    "dat_dlg_sd_full_sac": [
      "There is not enough free\nspace on this microSD card."
    ], 
    "dat_no_backup": [
      "There is no backup data."
    ], 
    "lau_1b_badge_plt": [
      "Place Badges"
    ], 
    "lau_1b_icon_big": [
      "Enlarge Icons"
    ], 
    "lau_1b_icon_small": [
      "Shrink Icons"
    ], 
    "lau_1b_make_folder": [
      "Create Folder"
    ], 
    "lau_1b_ok": [
      "OK"
    ], 
    "lau_1b_petit_open": [
      "HOME Menu Settings"
    ], 
    "lau_1b_start": [
      "Open"
    ], 
    "lau_1b_unwrap": [
      "Unwrap"
    ], 
    "lau_2b_close": [
      "Close"
    ], 
    "lau_2b_delete": [
      "Stop"
    ], 
    "lau_2b_folder_name": [
      "%d (New Folder)"
    ], 
    "lau_2b_folder_noname": [
      "(No name)"
    ], 
    "lau_2b_folder_open": [
      "Open"
    ], 
    "lau_2b_folder_setting": [
      "Settings"
    ], 
    "lau_2b_manual": [
      "Manual"
    ], 
    "lau_2b_property": [
      "View Details"
    ], 
    "lau_2b_restart": 
schrmh commented 1 year ago

I looked at the file using "MSBT Editor Reloaded" and at lau_2b_restart the Hex is 73 E0 20 00 52 00 65 00 73 00 75 00 6D 00 65 00 00 00 which results in the text  Resume So yeah 73 E0 20 00 is this private unicode symbol followed by a space which needs to be implemented somehow. Shouldn't be too hard but I don't really use python that much.

If it is done once for one symbol I can likely adjust it without much effort for all the other Unicode PUA symbols that Nintendo uses for the 3DS Home Menu.