search

jmathai / elodie

An EXIF-based photo assistant, organizer and workflow automation tool.
https://bit.ly/introducing-elodie
Apache License 2.0
1.27k stars 139 forks source link

Elodie crashes when non-ASCII character in filename. #379

Open filipfigzalski opened 4 years ago

filipfigzalski commented 4 years ago

I have folders with polish characters in name. When Elodie runs into them, I get this error:

Traceback (most recent call last):
  File "elodie.py", line 378, in <module>
    main()
  File "P:\Python\Python37-32\lib\site-packages\click\core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "P:\Python\Python37-32\lib\site-packages\click\core.py", line 696, in main
    rv = self.invoke(ctx)
  File "P:\Python\Python37-32\lib\site-packages\click\core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "P:\Python\Python37-32\lib\site-packages\click\core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "P:\Python\Python37-32\lib\site-packages\click\core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "elodie.py", line 141, in _import
    trash, allow_duplicates)
  File "elodie.py", line 68, in import_file
    media, allowDuplicate=allow_duplicates, move=False)
  File "c:\Users\Filip\Desktop\elodie-master\elodie\filesystem.py", line 525, in process_file
    metadata = media.get_metadata()
  File "c:\Users\Filip\Desktop\elodie-master\elodie\media\base.py", line 93, in get_metadata
    'date_taken': self.get_date_taken(),
  File "c:\Users\Filip\Desktop\elodie-master\elodie\media\photo.py", line 61, in get_date_taken
    exif = self.get_exiftool_attributes()
  File "c:\Users\Filip\Desktop\elodie-master\elodie\media\media.py", line 128, in get_exiftool_attributes
    self.exif_metadata = ExifTool().get_metadata(source)
  File "c:\Users\Filip\Desktop\elodie-master\elodie\external\pyexiftool.py", line 340, in get_metadata
    return self.execute_json(filename)[0]
  File "c:\Users\Filip\Desktop\elodie-master\elodie\external\pyexiftool.py", line 322, in execute_json
    return json.loads(self.execute(b"-j", *params).decode("utf-8"))
  File "P:\Python\Python37-32\lib\json\__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "P:\Python\Python37-32\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "P:\Python\Python37-32\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
crishnakh commented 4 years ago

I'm having this issue too.

xagrr commented 3 years ago

I'm not sure if I'm having quite the same issue but heres what it returns when I try to import from a "Masters" folder from an old iPhoto/Apple Photos library

/source/Masters/2014/09/12/20140912-103840/IMG_9457.JPG -> /destination/20140910/Apple-iPhone 5s/20140910-125554-unknown location-img_9457.jpg
Traceback (most recent call last):
  File "./elodie.py", line 378, in <module>
    main()
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "./elodie.py", line 140, in _import
    dest_path = import_file(current_file, destination, album_from_folder,
  File "./elodie.py", line 67, in import_file
    dest_path = FILESYSTEM.process_file(_file, destination,
  File "/elodie/elodie/filesystem.py", line 543, in process_file
    directory_name = self.get_folder_path(metadata)
  File "/elodie/elodie/filesystem.py", line 364, in get_folder_path
    this_path = self.get_dynamic_path(part, mask, metadata)
  File "/elodie/elodie/filesystem.py", line 387, in get_dynamic_path
    folder = folder.replace(
TypeError: replace() argument 2 must be str, not int
jmathai commented 3 years ago

May be related to #291.

A couple of requests.

  1. Do you know exactly which image it fails to import?
  2. Can you paste the relevant output by also passing in the --debug flag?
  3. What does your config.ini look like (appears you are customizing the file name)?

TypeError: replace() argument 2 must be str, not int

It might be helpful to see if casting the return of get_dynamic_path() on line 389.

str(self.get_dynamic_path(i[1:], i, metadata))
jmathai commented 3 years ago

@filipfigzalski and @crishnakh if you're still having this problem then please share more details as well.

xagrr commented 3 years ago

I have about 7000 files (not sure how many of them are pictures/movies) left in that directory and it processes 50-100 at a time before crashing like this. I'd left in the last file it tried to pull above. I've left a bit more info here with --debug on:

/source/Masters/2013/09/17/20130917-161815/._IMG_7222.JPG is not a valid media file. Skipping...
/source/.Trash-0/files/IMG_9231.JPG already at /destination/20140908/Apple-iPhone 5s/20140908-004349-unknown location-img_9231.jpg.
Original checksum returned None for /source/.Trash-0/files/IMG_9231.JPG. Skipping...
/source/.Trash-0/files/IMG_0644.JPG already at /destination/20130225/Apple-iPhone 3GS/20130225-112415-goa-img_0644.jpg.
Original checksum returned None for /source/.Trash-0/files/IMG_0644.JPG. Skipping...
/source/Masters/2014/06/21/20140621-125726/._IMG_8716.JPG is not a valid media file. Skipping...
/source/Masters/2013/06/07/20130607-231453/._IMG_6633.JPG is not a valid media file. Skipping...
/source/Masters/2013/03/05/20130305-221512/IMG_3238.JPG -> /destination/20130301/Apple-iPhone 4S/20130301-220501-gujarat-img_3238.jpg
/source/Masters/2013/11/27/20131127-193044/IMG_7434.JPG -> /destination/20131117/Apple-iPhone 4S/20131117-111232-philadelphia-pennsylvania-img_7434.jpg
Traceback (most recent call last):
  File "./elodie.py", line 378, in <module>
    main()
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "./elodie.py", line 140, in _import
    dest_path = import_file(current_file, destination, album_from_folder,
  File "./elodie.py", line 67, in import_file
    dest_path = FILESYSTEM.process_file(_file, destination,
  File "/elodie/elodie/filesystem.py", line 543, in process_file
    directory_name = self.get_folder_path(metadata)
  File "/elodie/elodie/filesystem.py", line 364, in get_folder_path
    this_path = self.get_dynamic_path(part, mask, metadata)
  File "/elodie/elodie/filesystem.py", line 387, in get_dynamic_path
    folder = folder.replace(
TypeError: replace() argument 2 must be str, not int

config.ini

[MapQuest]
key=[_redacted_]
prefer_english_names=True

[Directory]
date=%Y%m%d
custom=%camera_make-%camera_model
full_path=%date/%custom

[File]
date=%Y%m%d-%H%M%S
location=%city-%state
name=%date-%location-%original_name-%album.%extension

edit: copied in the correct log edit2: worth mentioning that I'm using this docker container and running this on an Unraid server. I couldn't get the Trash folder to map properly so it makes a hidden file inside my input folder.

jmathai commented 3 years ago

Thanks. A couple more things to help narrow down the file would be helpful.

Could you add this line here to log the file before it hits the error.

    for current_file in files:
        # *********** Add this line below ***********
        log.all('Preparing to process {}'.format(current_file))
        dest_path = import_file(current_file, destination, album_from_folder,
                    trash, allow_duplicates)
jmathai commented 3 years ago

I've not been able to replicate this issue. I can do some debugging if one of you can send me a zip file including the file which causes the error.

396 seems to have had the problem be in the name of a folder - so a zip file which extracts the folder would be helpful there. // @angrycuban13

jmathai commented 3 years ago

Probably a duplicate of #251.

reykjavi commented 3 years ago

Hi,

I stumbled upon the same issue. I let myself to insert another debugging string to elodie code (that's all I can with Python :)) to identify a file and a path that brings trouble. Here's what I've got:

{"--- The current file is: ":"c:\photolib\photoarc\Фото из сада\Флешка 1\ФОТО\Средняя 2014-2015 гг\ПРАЗДНИК ОСЕНИ\DSCN0346.JPG"}
Traceback (most recent call last):
  File "./elodie.py", line 379, in <module>
    main()
  File "C:\Python\Python38\lib\site-packages\click\core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "C:\Python\Python38\lib\site-packages\click\core.py", line 696, in main
    rv = self.invoke(ctx)
  File "C:\Python\Python38\lib\site-packages\click\core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Python\Python38\lib\site-packages\click\core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Python\Python38\lib\site-packages\click\core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "./elodie.py", line 141, in _import
    dest_path = import_file(current_file, destination, album_from_folder,
  File "./elodie.py", line 66, in import_file
    dest_path = FILESYSTEM.process_file(_file, destination,
  File "C:\repo\elodie\elodie\filesystem.py", line 524, in process_file
    metadata = media.get_metadata()
  File "C:\repo\elodie\elodie\media\base.py", line 93, in get_metadata
    'date_taken': self.get_date_taken(),
  File "C:\repo\elodie\elodie\media\photo.py", line 61, in get_date_taken
    exif = self.get_exiftool_attributes()
  File "C:\repo\elodie\elodie\media\media.py", line 128, in get_exiftool_attributes
    self.exif_metadata = ExifTool().get_metadata(source)
  File "C:\repo\elodie\elodie\external\pyexiftool.py", line 340, in get_metadata
    return self.execute_json(filename)[0]
  File "C:\repo\elodie\elodie\external\pyexiftool.py", line 322, in execute_json
    return json.loads(self.execute(b"-j", *params).decode("utf-8"))
  File "C:\Python\Python38\lib\json\__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "C:\Python\Python38\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python\Python38\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Also there is output of exiftool executable of that file:

ExifTool Version Number         : 12.16
File Name                       : DSCN0346.JPG
Directory                       : c:/photolib/photoarc/Фото из сада/Флешка 1/ФОТО/Средняя 2014-2015 гг/ПРАЗДНИК ОСЕНИ
Warning                         : FileName encoding not specified
File Size                       : 2.3 MiB
File Modification Date/Time     : 2014:10:29 11:26:06+03:00
File Access Date/Time           : 2021:01:24 00:54:23+03:00
File Creation Date/Time         : 2021:01:16 18:39:07+03:00
File Permissions                : rw-rw-rw-
File Type                       : JPEG
File Type Extension             : jpg
MIME Type                       : image/jpeg
Exif Byte Order                 : Little-endian (Intel, II)
Image Description               :
Make                            : NIKON
Camera Model Name               : COOLPIX L23
Orientation                     : Horizontal (normal)
X Resolution                    : 300
Y Resolution                    : 300
Resolution Unit                 : inches
Software                        : COOLPIX L23    V1.1
Modify Date                     : 2014:10:29 11:26:05
Y Cb Cr Positioning             : Co-sited
Exposure Time                   : 1/30
F Number                        : 3.3
Exposure Program                : Program AE
ISO                             : 110
Sensitivity Type                : Standard Output Sensitivity
Exif Version                    : 0230
Date/Time Original              : 2014:10:29 11:26:05
Create Date                     : 2014:10:29 11:26:05
Components Configuration        : Y, Cb, Cr, -
Compressed Bits Per Pixel       : 2
Exposure Compensation           : 0
Max Aperture Value              : 2.7
Metering Mode                   : Multi-segment
Light Source                    : Unknown
Flash                           : Auto, Did not fire
Focal Length                    : 6.1 mm
Maker Note Version              : 2.00
Color Mode                      : Color
Quality                         : Normal
White Balance                   : Auto
Focus Mode                      : AF-S
Flash Setting                   : Normal
White Balance Fine Tune         : 0
ISO Selection                   : Auto
Compression                     : JPEG (old-style)
Preview Image Start             : 1834
Preview Image Length            : 36996
Image Processing                :
Face Detect Frame Size          : 320 240
Faces Detected                  : 0
Image Adjustment                : Normal
Manual Focus Distance           : 0.500011444
Digital Zoom                    : 1
AF Area Mode                    : Single Area
AF Point                        : Center
AF Points In Focus              : (none)
Scene Mode                      :
Saturation Adj                  : 0
Noise Reduction                 : Off
Scene Assist                    :
Date Stamp Mode                 : Off
Retouch History                 : None
Image Stabilization             : VR-Off
User Comment                    :
Flashpix Version                : 0100
Color Space                     : sRGB
Exif Image Width                : 3648
Exif Image Height               : 2736
Interoperability Index          : R98 - DCF basic file (sRGB)
Interoperability Version        : 0100
File Source                     : Digital Camera
Scene Type                      : Directly photographed
Custom Rendered                 : Normal
Exposure Mode                   : Auto
Digital Zoom Ratio              : 0
Focal Length In 35mm Format     : 43 mm
Scene Capture Type              : Standard
Gain Control                    : Low gain up
Contrast                        : Normal
Saturation                      : Normal
Sharpness                       : Normal
Subject Distance Range          : Unknown
Thumbnail Offset                : 60432
Thumbnail Length                : 2581
Image Width                     : 3648
Image Height                    : 2736
Encoding Process                : Baseline DCT, Huffman coding
Bits Per Sample                 : 8
Color Components                : 3
Y Cb Cr Sub Sampling            : YCbCr4:2:2 (2 1)
Aperture                        : 3.3
Image Size                      : 3648x2736
Megapixels                      : 10.0
Preview Image                   : (Binary data 36996 bytes, use -b option to extract)
Scale Factor To 35 mm Equivalent: 7.0
Shutter Speed                   : 1/30
Thumbnail Image                 : (Binary data 2581 bytes, use -b option to extract)
Circle Of Confusion             : 0.004 mm
Field Of View                   : 45.4 deg
Focal Length                    : 6.1 mm (35 mm equivalent: 43.0 mm)
Hyperfocal Distance             : 2.65 m
Light Value                     : 8.2

Let me know if I can help you with any other details.

Thanks!

jmathai commented 3 years ago

Could you share the file with me via URL or email? jaisen (at) github-username (dot) com.

reykjavi commented 3 years ago

Could you share the file with me via URL or email? jaisen (at) github-username (dot) com.

Sent.

xagrr commented 3 years ago

I'm sorry I can't be responsive about this right now - I'm currently moving from NJ to Texas.

On Sun, Jan 24, 2021 at 4:09 AM reykjavi notifications@github.com wrote:

Could you share the file with me via URL or email? jaisen (at) github-username (dot) com.

Sent.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jmathai/elodie/issues/379#issuecomment-766315349, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADCEWURRYW5LYBYDWRLMVETS3PPV5ANCNFSM4OS7WAIQ .

reykjavi commented 3 years ago

Hey Jaisen,

I'm getting back to the issue as I'd like to help you with this and help other users as well. Actually, the problem is about non-ascii characters as said in this issue. It can be a filename or a path that consists of such characters. For me it's the path name. We had a debug session with you earlier, but we believed it's something about a file itself :) I just checked my assumption by taking the erroneous file and copying it under C:\temp. Then elodie processed it just fine. The original path, however, has some cyrillic chars in it -- "c:\photolib\photoarc\Фото из сада\Флешка 1\ФОТО\Средняя 2014-2015 гг\ПРАЗДНИК ОСЕНИ\DSCN0346.JPG"

Huh!! May we hope you'll deal with this bug once? :)

jmathai commented 3 years ago

This bug indicates file name - but I think you're saying it's the path?

I'm curious if there's an underlying reason why non-ascii errors present themselves. We fixed a version of this in #167 andd a few other places.

xagrr commented 3 years ago

I’ll have to immerse myself back into this - most likely this summer. Sorry I haven’t been more of a help so far.

On Apr 9, 2021 at 6:45 PM, <Jaisen Mathai @.***)> wrote:

This bug indicates file name - but I think you're saying it's the path?

I'm curious if there's an underlying reason why non-ascii errors present themselves. We fixed a version of this in #167 (https://github.com/jmathai/elodie/issues/167) andd a few other places.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub (https://github.com/jmathai/elodie/issues/379#issuecomment-817011449), or unsubscribe (https://github.com/notifications/unsubscribe-auth/ADCEWUU5T2J6PPMV5LZS2MLTH57O7ANCNFSM4OS7WAIQ).

reykjavi commented 3 years ago

@jmathai I believe it can be both, path or file (to OS it looks identical). I pulled the recent version of elodie, but the issue is still there. You can reproduce it by creating the path I provided above.

xagrr commented 3 years ago

Did you still need help resolving this?

On 4/11/2021 12:26 PM, reykjavi wrote:

@jmathai https://github.com/jmathai I believe it can be both, path or file (to OS it looks identical). I pulled the recent version of elodie, but the issue is still there. You can reproduce it by creating the path I provided above.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jmathai/elodie/issues/379#issuecomment-817333923, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADCEWURNXGILEMITFPA2KULTIHEUHANCNFSM4OS7WAIQ.

reykjavi commented 2 years ago

Hey,

I have moved to Mac recently and checked elodie with my photo archive again. Just to make you aware — there is no issue described in this topic on Mac. Therefore I can state it's Windows-specific issue.

crishnakh commented 2 years ago

Hi @reykjavi I have it on mac... so it's not only windows-specific issue.