Importing records containing latin-1 characters fails

[ctjacobs]

PyQSO fails to import ADIF records containing latin-1 characters (e.g. ä). The addition of new records may fail as well.

The patch below causes PyQSO to try and replace the latin-1 characters with UTF-8 ones. The ADIF records will then be added instead of being skipped. However, I think some characters may be lost that way. Supporting all the different character encoding schemes seems rather difficult - Anyone have any experience with this sort of thing?

Thanks to Timo OH7JHA for reporting this problem.

diff --git a/pyqso/adif.py b/pyqso/adif.py
index 96da451..d0a459d 100644
--- a/pyqso/adif.py
+++ b/pyqso/adif.py
@@ -251,7 +251,7 @@ class ADIF:
                   continue
                else:
                   if( (r[field_name] != "NULL") and (r[field_name] != "") ):
-                     f.write("<%s:%d>%s\n" % (field_name.lower(), len(r[field_name]), r[field_name]))
+                     f.write("<%s:%d>%s\n" % (field_name.lower(), len(r[field_name]), r[field_name].encode("utf-8")))
             f.write("<eor>\n")

          logging.debug("Finished writing records to the ADIF file.")
@@ -259,8 +259,9 @@ class ADIF:

       except IOError as e:
          logging.error("I/O error %d: %s" % (e.errno, e.strerror))
-      except:
-         logging.error("Unknown error occurred when writing the ADIF file.")
+      except Exception as e: # All other exceptions.
+         logging.error("An exception occurred when writing the ADIF file.")
+         logging.error(e)

       return

diff --git a/pyqso/logbook.py b/pyqso/logbook.py
index 4f265ab..9d478ed 100644
--- a/pyqso/logbook.py
+++ b/pyqso/logbook.py
@@ -181,6 +181,7 @@ class Logbook(Gtk.Notebook):
          self.db_disconnect() # Destroy any existing connections first.
          self.connection = sqlite.connect(path)
          self.connection.row_factory = sqlite.Row
+         self.connection.text_factory = lambda x: unicode(x, "utf-8", errors="replace")
       except sqlite.Error as e:
          # PyQSO can't connect to the database.
          logging.exception(e)

[ctjacobs]

Worked-around by 416497774711ea132965301d659ad16f482782f7. Any characters that can't be decoded when writing/reading from an ADIF file are replaced by a replacement marker. Not a great solution but I didn't really want to go down the "try to guess the character encoding from all the possible options" route as this may cause problems later on at the database level.