ASCII-8BIT to UTF-8 Encoding::UndefinedConversionError

tedsluis commented 4 years ago

I run drivesync in a official ruby:2.6.6 docker container, see dockerfile

During downloading files from google drive to my local host, drivesync sometimes fail while processing a file with a special character, see two examples below:

file name: Bokningsbekräftelse/Kvitto.pdf special character: ä UTF-8: \xc3\xa4

Updating file Aimer39V1.0.0.pdf ...
Uploading file Bokningsbekräftelse/Kvitto.pdf ...
Traceback (most recent call last):
        21: from drivesync.rb:64:in `<main>'
        20: from drivesync.rb:23:in `sync'
        19: from /drivesync/src/synchronizer.rb:71:in `run'
        18: from /drivesync/src/synchronizer.rb:279:in `sync'
        17: from /drivesync/src/synchronizer.rb:279:in `each'
        16: from /drivesync/src/synchronizer.rb:282:in `block in sync'
        15: from /drivesync/src/synchronizer.rb:166:in `upload_file'
        14: from /drivesync/src/drive_manager.rb:108:in `upload'
        13: from /drivesync/src/drive_manager.rb:178:in `traverse_and_create'
        12: from /drivesync/src/drive_manager.rb:178:in `each'
        11: from /drivesync/src/drive_manager.rb:181:in `block in traverse_and_create'
        10: from /drivesync/src/drive_manager.rb:167:in `create_folder'
         9: from /usr/local/bundle/gems/google-api-client-0.23.4/generated/google/apis/drive_v3/service.rb:601:in `create_file'
         8: from /usr/local/bundle/gems/google-api-client-0.23.4/lib/google/apis/core/base_service.rb:360:in `execute_or_queue_command'
         7: from /usr/local/bundle/gems/google-api-client-0.23.4/lib/google/apis/core/http_command.rb:91:in `execute'
         6: from /usr/local/bundle/gems/google-api-client-0.23.4/lib/google/apis/core/api_command.rb:64:in `prepare!'
         5: from /usr/local/bundle/gems/representable-3.0.4/lib/representable/json.rb:44:in `to_json'
         4: from /usr/local/bundle/gems/multi_json-1.13.1/lib/multi_json.rb:139:in `dump'
         3: from /usr/local/bundle/gems/multi_json-1.13.1/lib/multi_json/adapter.rb:25:in `dump'
         2: from /usr/local/bundle/gems/multi_json-1.13.1/lib/multi_json/adapters/json_common.rb:19:in `dump'
         1: from /usr/local/bundle/gems/multi_json-1.13.1/lib/multi_json/adapters/json_common.rb:19:in `to_json'
/usr/local/bundle/gems/multi_json-1.13.1/lib/multi_json/adapters/json_common.rb:19:in `encode': "\xC3" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)

file name: 1980 12 Sacré Coeur 5.jpg special character: é UTF-8: \xc3\xa8

Updating file Google Foto's/2008/01/1977 7 12half  jaar.jpg ...
Downloading file Google Foto's/2008/01/1977 8 12half jaar.jpg ...
Uploading file Google Foto's/2008/01/1980 12 Sacré Coeur 5.jpg ...
Traceback (most recent call last):
        19: from drivesync.rb:64:in `<main>'
        18: from drivesync.rb:23:in `sync'
        17: from /drivesync/src/synchronizer.rb:71:in `run'
        16: from /drivesync/src/synchronizer.rb:279:in `sync'
        15: from /drivesync/src/synchronizer.rb:279:in `each'
        14: from /drivesync/src/synchronizer.rb:282:in `block in sync'
        13: from /drivesync/src/synchronizer.rb:166:in `upload_file'
        12: from /drivesync/src/drive_manager.rb:116:in `upload'
        11: from /usr/local/bundle/gems/google-api-client-0.23.4/generated/google/apis/drive_v3/service.rb:601:in `create_file'
        10: from /usr/local/bundle/gems/google-api-client-0.23.4/lib/google/apis/core/base_service.rb:360:in `execute_or_queue_command'
         9: from /usr/local/bundle/gems/google-api-client-0.23.4/lib/google/apis/core/http_command.rb:91:in `execute'
         8: from /usr/local/bundle/gems/google-api-client-0.23.4/lib/google/apis/core/upload.rb:141:in `prepare!'
         7: from /usr/local/bundle/gems/google-api-client-0.23.4/lib/google/apis/core/upload.rb:51:in `prepare!'
         6: from /usr/local/bundle/gems/google-api-client-0.23.4/lib/google/apis/core/api_command.rb:64:in `prepare!'
         5: from /usr/local/bundle/gems/representable-3.0.4/lib/representable/json.rb:44:in `to_json'
         4: from /usr/local/bundle/gems/multi_json-1.13.1/lib/multi_json.rb:139:in `dump'
         3: from /usr/local/bundle/gems/multi_json-1.13.1/lib/multi_json/adapter.rb:25:in `dump'
         2: from /usr/local/bundle/gems/multi_json-1.13.1/lib/multi_json/adapters/json_common.rb:19:in `dump'
         1: from /usr/local/bundle/gems/multi_json-1.13.1/lib/multi_json/adapters/json_common.rb:19:in `to_json'
/usr/local/bundle/gems/multi_json-1.13.1/lib/multi_json/adapters/json_common.rb:19:in `encode': "\xC3" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)

To resolve this (each time a file name contains a special character) I need to:

rename the file name in my google drive (to some name without a special character)
rename the file name in the manifest file.
rename the file name on my local machine.
restart drivesync

I think my docker container supports UTF-8, because I am able to rename the files from inside the container:

[root@i7nuc Docker-Linux-sync-Google-Drive]# podman run -it \
  -v /mnt/extdisk/googledrive:/root/Documents/drive:z localhost/google-drivesync /bin/bash 

root@12303cbbee3c:/drivesync# mv "/root/Documents/drive/Google Foto's/2008/01/1980 12 Sacré Coeur 5.jpg"  "/root/Documents/drive/Google Foto's/2008/01/1980 12 Sacre Coeur 5.jpg" 

root@12303cbbee3c:/drivesync# ls -l "/root/Documents/drive/Google Foto's/2008/01/1980 12 Sacre Coeur 5.jpg"
-rw-r--r--. 1 root root 801428 May 27 12:24 "/root/Documents/drive/Google Foto's/2008/01/1980 12 Sacre Coeur 5.jpg"

My locale setting inside the container:

root@12303cbbee3c:/drivesync# locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

What can I do to resolve this? (I have no experience with Ruby).

tedsluis commented 4 years ago

Fixed! I have added this environment variable in the Dockerfile:

ENV LC_ALL en_US.UTF-8

Now it works:

Local folder is 3 files behind and 0 files ahead of remote
Starting sync at 2020-05-28 07:10:29 +0200                                                                             
Downloading file Google Foto's/2008/01/1977 7 12½  jaar.jpg ...
Downloading file Aimer39平台镜像文件生成工具使用说明_V1.0.0.pdf  ...
Downloading file Google Foto's/2008/01/1980 12 Sacré Coeur 5.jpg ...
Sync complete.

Katlemm commented 3 years ago

Do you have an idea how to fix this issue, if you not used the dockerfile? I have a normal installation and get the same issue. #50

MStadlmeier / drivesync

ASCII-8BIT to UTF-8 Encoding::UndefinedConversionError #38