Import Fails: json.decoder.JSONDecodeError

[fairlygood]

Hi,

Recieving the following message when trying to import on Windows 10:

Traceback (most recent call last): File "C:\Users\Jamie Adams\bin\elodie\elodie.py", line 345, in <module> main() File "C:\Users\Jamie Adams\AppData\Local\Programs\Python\Python36\lib\site-packages\click\core.py", line 716, in __call__ return self.main(*args, **kwargs) File "C:\Users\Jamie Adams\AppData\Local\Programs\Python\Python36\lib\site-packages\click\core.py", line 696, in main rv = self.invoke(ctx) File "C:\Users\Jamie Adams\AppData\Local\Programs\Python\Python36\lib\site-packages\click\core.py", line 1060, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\Users\Jamie Adams\AppData\Local\Programs\Python\Python36\lib\site-packages\click\core.py", line 889, in invoke return ctx.invoke(self.callback, **ctx.params) File "C:\Users\Jamie Adams\AppData\Local\Programs\Python\Python36\lib\site-packages\click\core.py", line 534, in invoke return callback(*args, **kwargs) File "C:\Users\Jamie Adams\bin\elodie\elodie.py", line 115, in _import trash, allow_duplicates) File "C:\Users\Jamie Adams\bin\elodie\elodie.py", line 64, in import_file media, allowDuplicate=allow_duplicates, move=False) File "C:\Users\Jamie Adams\bin\elodie\elodie\filesystem.py", line 336, in process_file media.set_original_name() File "C:\Users\Jamie Adams\bin\elodie\elodie\media\media.py", line 246, in set_original_name if self.get_original_name() is not None: File "C:\Users\Jamie Adams\bin\elodie\elodie\media\media.py", line 143, in get_original_name exiftool_attributes = self.get_exiftool_attributes() File "C:\Users\Jamie Adams\bin\elodie\elodie\media\media.py", line 129, in get_exiftool_attributes metadata = et.get_metadata(source) File "C:\Users\Jamie Adams\bin\elodie\elodie\external\pyexiftool.py", line 330, in get_metadata return self.execute_json(filename)[0] File "C:\Users\Jamie Adams\bin\elodie\elodie\external\pyexiftool.py", line 314, in execute_json return json.loads(self.execute(b"-j", *params).decode("latin-1")) File "C:\Users\Jamie Adams\AppData\Local\Programs\Python\Python36\lib\json\__init__.py", line 354, in loads return _default_decoder.decode(s) File "C:\Users\Jamie Adams\AppData\Local\Programs\Python\Python36\lib\json\decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Users\Jamie Adams\AppData\Local\Programs\Python\Python36\lib\json\decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Anything I can provide to help debug? I'm not using any special configuration in config.ini.

[dixonge]

I just got the same error, after successfully processing almost 2GB of images. Just died

Traceback (most recent call last):
  File "C:\Users\dixon\elodie\elodie.py", line 345, in <module>
    main()
  File "C:\Users\dixon\AppData\Local\Programs\Python\Python36\lib\site-packages\click\core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\dixon\AppData\Local\Programs\Python\Python36\lib\site-packages\click\core.py", line 696, in main
    rv = self.invoke(ctx)
  File "C:\Users\dixon\AppData\Local\Programs\Python\Python36\lib\site-packages\click\core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\dixon\AppData\Local\Programs\Python\Python36\lib\site-packages\click\core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\dixon\AppData\Local\Programs\Python\Python36\lib\site-packages\click\core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\dixon\elodie\elodie.py", line 115, in _import
    trash, allow_duplicates)
  File "C:\Users\dixon\elodie\elodie.py", line 64, in import_file
    media, allowDuplicate=allow_duplicates, move=False)
  File "C:\Users\dixon\elodie\elodie\filesystem.py", line 336, in process_file
    media.set_original_name()
  File "C:\Users\dixon\elodie\elodie\media\media.py", line 246, in set_original_name
    if self.get_original_name() is not None:
  File "C:\Users\dixon\elodie\elodie\media\media.py", line 143, in get_original_name
    exiftool_attributes = self.get_exiftool_attributes()
  File "C:\Users\dixon\elodie\elodie\media\media.py", line 129, in get_exiftool_attributes
    metadata = et.get_metadata(source)
  File "C:\Users\dixon\elodie\elodie\external\pyexiftool.py", line 330, in get_metadata
    return self.execute_json(filename)[0]
  File "C:\Users\dixon\elodie\elodie\external\pyexiftool.py", line 312, in execute_json
    return json.loads(self.execute(b"-j", *params).decode("utf-8"))
  File "C:\Users\dixon\AppData\Local\Programs\Python\Python36\lib\json\__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "C:\Users\dixon\AppData\Local\Programs\Python\Python36\lib\json\decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\dixon\AppData\Local\Programs\Python\Python36\lib\json\decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[jmathai]

Do either of you know which file this error occurred? If you could send me the file that may help me reproduce the bug.

[dixonge]

This is the last thing in the buffer before the error (file attached)

F:\Pictures\ONLINE\Social Media\IMG_20170204_093148736_kindlephoto-647753632.jpg -> F:\Pictures\Google Photos Staging\2017-02-Feb\Unknown Location\2017-02-04_15-31-48-img_20170204_093148736_kindlephoto-647753632.jpg

[jmathai]

@dixonge looks like that's the last photo which succeeded. It would be helpful to know which one caused an error. Could you add print(_file) to this line and run the import command again?

[dixonge]

It will probably be later tonight or tomorrow.

Speaking of files, what determines the file order? It seems to be rather random and across folders...

[jmathai]

@dixonge sort order is based on the default of os.walk which is not guaranteed to be of any order. However, there's a pull request to sort them alphabetically which has not been merged yet.

[dixonge]

To the ‘move = False’ line? I’m not sure where to put it. Could you show me in context?

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: Jaisen Mathaimailto:notifications@github.com Sent: Tuesday, October 10, 2017 1:33 AM To: jmathai/elodiemailto:elodie@noreply.github.com Cc: Glenn Dixonmailto:dixonge@posteo.net; Mentionmailto:mention@noreply.github.com Subject: Re: [jmathai/elodie] Import Fails: json.decoder.JSONDecodeError (#251)

@dixongehttps://github.com/dixonge looks like that's the last photo which succeeded. It would be helpful to know which one caused an error. Could you add print(_file) to this linehttps://github.com/jmathai/elodie/blob/master/elodie/filesystem.py#L324 and run the import command again?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/jmathai/elodie/issues/251#issuecomment-335374631, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABgWCgD7EqKva3REj6GpdRvKfOjTMzSaks5sqw-7gaJpZM4Phx7w.

[jmathai]

Sure, here you go.

    def process_file(self, _file, destination, media, **kwargs):
        print(_file)
        move = False
        if('move' in kwargs):
            move = kwargs['move']

[dixonge]

I am rerunning the import now, but it looks like elodie is remembering the previous import, because my external drive now has a different drive letter, and I am getting notifications like "matched checksum but file not found at F:..." So far all of my imports have been tests using a small group of folders. Once I get all of my EXIF and file information/naming decided, will I need to reset elodie before starting my final imports?

[jmathai]

There's a local database at ~/.elodie/hash.json. You can delete that file whenever you want to reimport photos. There's also a --allow-duplicates flag you can pass in where it skips the check altogether.

[dixonge]

and, of course, it imported 83 files from 6 folders into the target folder inside 8 year-based folders with no errors....

[jmathai]

Is it the same corpus of photos that caused an error before and with the same config file?

[dixonge]

Same files, but config file has been changed – mainly just dropped the location subfolders and used year-only for dated folders. Still working out the exact folder structure I want.

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: Jaisen Mathaimailto:notifications@github.com Sent: Friday, October 13, 2017 3:34 PM To: jmathai/elodiemailto:elodie@noreply.github.com Cc: Glenn Dixonmailto:dixonge@posteo.net; Mentionmailto:mention@noreply.github.com Subject: Re: [jmathai/elodie] Import Fails: json.decoder.JSONDecodeError (#251)

Is it the same corpus of photos that caused an error before and with the same config file?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/jmathai/elodie/issues/251#issuecomment-336560439, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABgWCoa-Fd5dGn9I4A8CuZLgUoBiBdb3ks5sr8lRgaJpZM4Phx7w.

[jmathai]

@dixonge for the sake of this bug do you mind rerunning the import with the original config file which caused the error and letting me know what file causes it? That will help me reproduce.

[dixonge]

Running now, but if it stops like it did last time I don't think I will have any better idea which file was the one being processed. It is currently going through the six folders randomly. Would --debug help?

[jmathai]

Yes, I would include the --debug. Also, the print(_file) should output the file that is being imported before failing.

[dixonge]

83 complete, 0 errors

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: Jaisen Mathaimailto:notifications@github.com Sent: Friday, October 13, 2017 6:06 PM To: jmathai/elodiemailto:elodie@noreply.github.com Cc: Glenn Dixonmailto:dixonge@posteo.net; Mentionmailto:mention@noreply.github.com Subject: Re: [jmathai/elodie] Import Fails: json.decoder.JSONDecodeError (#251)

Yes, I would include the --debug. Also, the print(_file) should output the file that is being imported before failing.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/jmathai/elodie/issues/251#issuecomment-336587339, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABgWCnOokpA1VdQOqOzkeaigQeK9WHIPks5sr-zcgaJpZM4Phx7w.

[evanjt]

I get this error too. It happens on different files (which I can tell after adding the print(_file) line in. I've looked at those files and I don't see anything irregular about them. I'm trying to run this on a fairly large directory (24GB).

This is my output, from running this command: "./elodie.py import --destination="~/2013-elodie" ~/2013 --debug":

/home/evan/2013/combined/combined/06_22_img_6301.jpg /home/evan/2013/combined/combined/06_22_img_6301.jpg -> /home/evan/2013-elodie/2013-06-Jun/Unknown Location/2013-06-22_02-46-35-06_22_img_6301.jpg /home/evan/2013/combined/combined/09_24_15689814391.jpg Traceback (most recent call last): File "./elodie.py", line 345, in main() File "/home/evan/anaconda3/lib/python3.6/site-packages/click/core.py", line 716, in call return self.main(args, kwargs) File "/home/evan/anaconda3/lib/python3.6/site-packages/click/core.py", line 696, in main rv = self.invoke(ctx) File "/home/evan/anaconda3/lib/python3.6/site-packages/click/core.py", line 1060, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/evan/anaconda3/lib/python3.6/site-packages/click/core.py", line 889, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/evan/anaconda3/lib/python3.6/site-packages/click/core.py", line 534, in invoke return callback(args, *kwargs) File "./elodie.py", line 115, in _import trash, allow_duplicates) File "./elodie.py", line 64, in import_file media, allowDuplicate=allow_duplicates, move=False) File "/home/evan/elodie/elodie/filesystem.py", line 337, in process_file media.set_original_name() File "/home/evan/elodie/elodie/media/media.py", line 286, in set_original_name if self.get_original_name() is not None: File "/home/evan/elodie/elodie/media/media.py", line 183, in get_original_name exiftool_attributes = self.get_exiftool_attributes() File "/home/evan/elodie/elodie/media/media.py", line 131, in get_exiftool_attributes metadata = et.get_metadata(source) File "/home/evan/elodie/elodie/external/pyexiftool.py", line 330, in get_metadata return self.execute_json(filename)[0] File "/home/evan/elodie/elodie/external/pyexiftool.py", line 314, in execute_json return json.loads(self.execute(b"-j", params).decode("latin-1")) File "/home/evan/anaconda3/lib/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/home/evan/anaconda3/lib/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/home/evan/anaconda3/lib/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[WilfriedBl]

I just found this one year old thread. Has it been solved by now? I ran into the same error. Not by using elodie but in my own program which also uses pyexiftool. Scanning many images in varies folders, I noticed the error "Expecting value: line 1 column 1 (char 0)" occurs always, if the path name contains accented characters, such us ä, ö, ü or é.

[jmathai]

@WilfriedBl there have been various unicode errors but if you share your photo I can see if it fails.

[WilfriedBl]

Thank you for your quick response! My question is not related to elodie, but only to pyexiftool (which I believe you use too ..?) and I am curious if you experienced (and hopefully solved) a similar problem. It does not appear only for a single photo, but for all images residing in folders with path name containing accented characters. Your response reminded me to do a more isolated test. So I copied one of my pictures to a different folder with the name 'testä'. Running my program which uses pyexiftool, indeed brings the error message mentioned above - renaming the folder to "test" and running it again, works fine.

[jmathai]

We made this change to pyexiftool.py which may help.

[WilfriedBl]

Thank you! I was hoping for something like that, but unfortunately the change didn't solve the problem. In fact it seems the json,loads in line 312 never raises an exception and I get: Traceback (most recent call last): File "D:/PyCharmProjects/Daminion/DAMinspect.py", line 623, in inspect_each_item() File "D:/PyCharmProjects/Daminion/DAMinspect.py", line 333, in inspect_each_item exif_been_read = et.get_tags(exiftool_tags_to_read, str(path_and_file)) File "D:\PyCharmProjects\Daminion\exiftool.py", line 364, in get_tags return self.get_tags_batch(tags, [filename])[0] File "D:\PyCharmProjects\Daminion\exiftool.py", line 356, in get_tags_batch return self.execute_json(params) File "D:\PyCharmProjects\Daminion\exiftool.py", line 315, in execute_json return json.loads(self.execute(b"-j", params).decode("latin-1")) File "C:\Program Files\Python36\lib\json__init__.py", line 354, in loads return _default_decoder.decode(s) File "C:\Program Files\Python36\lib\json\decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Program Files\Python36\lib\json\decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[jmathai]

Seems like the json.loads line does throw the exception. Unsure if it's the same as what the commit I pointed to fixes though.

File "D:\PyCharmProjects\Daminion\exiftool.py", line 315, in execute_json
return json.loads(self.execute(b"-j", *params).decode("latin-1"))

[WilfriedBl]

Thanks a lot so far. At least, I have a starting point for further investigation now or simply try to exclude path names with accented characters. Have a nice weekend.

[WilfriedBl]

Hello Jaisen, just to let you know the final solution for my problem: I needed to add .encode('latin-1') to the file name in my code where I call the ExifTool. This removed the exception, if there is any of the special characters éèêáàâÉÈÊÁÀÂäöüßÄÖÜ in the path name. However, pyexiftool still throws an exception if the path contains 'æ', but since I found only a single path with it, renaming the folder was the easier solution. I wish you a happy new year!

[vkhobor]

Hi, I have encountered this issue with elodie. JSONDecodeError is thrown. This fix (https://github.com/jmathai/elodie/issues/251#issuecomment-450622451) indeed fixed this particular Error for me, but on other pictures it fails, when the filename has 'ű'.

 def get_metadata(self, filename):
        """Return meta-data for a single file.

        The returned dictionary has the format described in the
        documentation of :py:meth:`execute_json()`.
        """
        return self.execute_json(filename.encode('latin-1'))[0] 
// changed from: return self.execute_json(filename)[0] 

in here: https://github.com/jmathai/elodie/blob/4186d4309a7b1c2b9ddca1f27275ed204d14110b/elodie/external/pyexiftool.py#L334-L341

The problem was for me when special chars are in filename is that exiftool did not read in any JSON. I modified code to print out the parameters passed to exiftool and the returned JSON in the code. Without the fix these are the outputs:

Filename: á.jpg

python3 ./elodie.py import --destination="A:\elodie-test" "A:\arch\archive-kep2\asztal!\test" --debug --album-from-folder Parameters given to exiftool: (b'-j', b'A:\arch\archive-kep2\asztal!\test\\xc3\xa1.jpg') Returned JSON: b''

Filename: a.jpg

python3 ./elodie.py import --destination="A:\elodie-test" "A:\arch\archive-kep2\asztal!\test" --debug --album-from-folder Parameters given to exiftool: (b'-j', b'A:\arch\archive-kep2\asztal!\test\a.jpg') Returned JSON: b'[{\r\n "SourceFile": "A:/arch/archive-kep2/asztal!/test/a.jpg",\r\n "ExifTool:ExifToolVersion": 12.10,\r\n "File:FileName": "a.jpg",\r\n "File:Directory": "A:/arch/archive-kep2/asztal!/test",\r\n "File:FileSize": 210976,\r\n "File:FileModifyDate": "2020:11:20 06:31:09+01:00",\r\n "File:FileAccessDate": "2020:11:20 06:31:09+01:00",\r\n "File:FileCreateDate": "2020:11:20 04:58:54+01:00",\r\n "File:FilePermissions": 666,\r\n "File:FileType": "JPEG",\r\n "File:FileTypeExtension": "JPG",\r\n "File:MIMEType": "image/jpeg",\r\n "File:ImageWidth": 1137,\r\n "File:ImageHeight": 674,\r\n "File:EncodingProcess": 0,\r\n "File:BitsPerSample": 8,\r\n "File:ColorComponents": 3,\r\n "File:YCbCrSubSampling": "2 2",\r\n "JFIF:JFIFVersion": "1 1",\r\n "JFIF:ResolutionUnit": 1,\r\n "JFIF:XResolution": 96,\r\n "JFIF:YResolution": 96,\r\n "XMP:XMPToolkit": "Image::ExifTool 12.10",\r\n "XMP:Album": "test",\r\n "Composite:ImageSize": "1137 674",\r\n "Composite:Megapixels": 0.766338\r\n}]\r\n' A:\arch\archive-kep2\asztal!\test\a.jpg already at A:\elodie-test\2020\test\2020-11-20_03-58-54-nev.jpg.

I hope this maybe helps.

[jmathai]

@vkhobor were you able to make a code that was more generic which worked for you?

[vkhobor]

I couldn't sorry.