search

zimmerst / phoshare

Automatically exported from code.google.com/p/phoshare
Other
0 stars 0 forks source link

Ascii error with Slovak codepage #8

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. export library which have national characters in Event name

What version of the product are you using? On what operating system?
Phoshare 1.1
 MAc OS X 10.6.4

Log from terminal:

Reading iPhoto database from /Users/MiRo/Pictures/iPhoto Library...
Scanning iPhoto data for photos to export...
Scanning existing files in export folder...
Exporting photos from iPhoto to export folder...
New file: /Users/MiRo/Dropbox/Photos/1_apríl na Javorine/PICT0264.jpg (convert)
Updating IPTC for /Users/MiRo/Dropbox/Photos/1_apríl na Javorine/PICT0264.jpg 
because of date (None instead of 2006-04-01 10:47:13)
Traceback (most recent call last):
  File "//Applications/Phoshare.app/Contents/Resources/Phoshare.py", line 927, in <module>
    main()
  File "//Applications/Phoshare.app/Contents/Resources/Phoshare.py", line 923, in main
    exclude_folders, options)
  File "//Applications/Phoshare.app/Contents/Resources/Phoshare.py", line 787, in export_iphoto
    library.generate_files(options)
  File "//Applications/Phoshare.app/Contents/Resources/Phoshare.py", line 757, in generate_files
    self.named_folders[ndir].generate_files(options)
  File "//Applications/Phoshare.app/Contents/Resources/Phoshare.py", line 594, in generate_files
    self.files[f].generate(options)
  File "//Applications/Phoshare.app/Contents/Resources/Phoshare.py", line 277, in generate
    self.check_iptc_data(self.export_file, options)
  File "//Applications/Phoshare.app/Contents/Resources/Phoshare.py", line 455, in check_iptc_data
    new_rectangles, new_persons)
  File "/Applications/Phoshare.app/Contents/Resources/tilutil/exiftool.py", line 217, in update_iptcdata
    filepath, result)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 56: 
ordinal not in range(128)
logout

Original issue reported on code.google.com by miroslav...@gmail.com on 24 Sep 2010 at 9:16

GoogleCodeExporter commented 9 years ago 
An unfortunate drawback of using Python for this tool is that it's hard to make 
all parts of it work reliably with non-ASCII characters because Python doesn't 
use Unicode natively.

Do you feel comfortable editing the Python code of Phoshare? The change below 
should fix this error. In line 217 of file 
/Applications/Phoshare.app/Contents/Resources/tilutil/exiftool.py, change
     filepath, result)
to
     su.fsenc(filepath), result)
See also below. I'll post a patch soon, but it would be great if you could try 
this, and report if that fixes all your problems. It was actually trying to 
print an error about a problem updating the metadata of an image. That will 
still happen, but at least the export should continue with this change.

Tilman

$ diff -c src/tilutil/exiftool.py 
dist-1.1/Phoshare.app/Contents/Resources/tilutil/exiftool.py
*** src/tilutil/exiftool.py 2010-09-24 14:40:10.000000000 -0700
--- dist-1.1/Phoshare.app/Contents/Resources/tilutil/exiftool.py    2010-09-23 
10:06:54.000000000 -0700
***************
*** 214,219 ****
          return True
      else:
          print >> sys.stderr, "Failed to update IPTC data in image %s: %s" % (
!             su.fsenc(filepath), result)
          return False

--- 214,219 ----
          return True
      else:
          print >> sys.stderr, "Failed to update IPTC data in image %s: %s" % (
!             filepath, result)
          return False

Original comment by tsporkert on 24 Sep 2010 at 10:02

GoogleCodeExporter commented 9 years ago 
Thanks,
after change of code export works.

The names of created directories from events are not with proper national 
characters.

Original comment by miroslav...@gmail.com on 26 Sep 2010 at 7:13

GoogleCodeExporter commented 9 years ago 
Directories names are OK, that was not problem with Phoshare.

Original comment by miroslav...@gmail.com on 26 Sep 2010 at 7:15

GoogleCodeExporter commented 9 years ago 
Thanks for verifying the fix. I'll roll it into the next update.

Original comment by tilma...@google.com on 26 Sep 2010 at 3:46

GoogleCodeExporter commented 9 years ago

Original comment by tsporkert on 14 Oct 2010 at 5:02

GoogleCodeExporter commented 9 years ago 
Hi,
Still got the same Error with Version 1.3 and OSX 10.6.4
I'm using german characters.

Tried to fix it with the workaround above, but couldn't find the file 
exiftool.py.

Any suggestion? Many thanks.

Original comment by cstu...@hotmail.com on 26 Oct 2010 at 7:15

GoogleCodeExporter commented 9 years ago 
That workaround is no longer necessary, because I fixed this a long time ago in 
Phoshare. I'm surprised you are still seeing this problem - I actually made a 
test case with special characters in the event name, photo title, keywords, 
etc. I'll have another look.

Original comment by tilma...@google.com on 26 Oct 2010 at 7:23

GoogleCodeExporter commented 9 years ago 
Hi,
I think I encountered the same error as is stated in comment 6.
In my iPhoto database it's caused by german umlauts in some face names, which 
generates an error message like this:
Exporting photos from iPhoto to export folder...
value 0 0.487500 0.481667 0.005833
Error: 'ascii' codec can't encode character u'\xe4' in position 123: ordinal 
not in range(128)

I propose the following fix for this:

diff Phoshare.py.old Phoshare.py
454c454
<                         region_names[p], ','.join(str(c) for c in
---
>                         region_names[p].encode("utf-8"), ','.join(str(c) for 
c in

Could somebody please confirm that this is sufficient and the iptc data is 
written correctly into the file?

Original comment by sebastia...@gmail.com on 9 Nov 2010 at 9:48

GoogleCodeExporter commented 9 years ago 
Sebastian, thanks for the suggestion. A more appropriate fix is
454c454
<                         region_names[p], ','.join(str(c) for c in
---
>                         su.fsenc(region_names[p]), ','.join(str(c) for c in
This does not affect IPTC data, but just the console output of what Phoshare is 
doing.

I do have German umlauts in some of my faces, and have not observed this error. 
But I'll incorporate the fix into the next update.

You can test the IPTC data with Picasa (http://picasa.google.com). It will read 
the face rectangles when it loads the image.

Original comment by tsporkert on 10 Nov 2010 at 7:40

GoogleCodeExporter commented 9 years ago 
I couldn't completely follow this fix.

Do I need to change in the file "phoshare.py" the string
"region_names[p], ','.join(str(c) for c in"
to 
"su.fsenc(region_names[p]), ','.join(str(c) for c in"?

This didn't work for me. Am I wrong?
Thanks

Original comment by cstu...@hotmail.com on 10 Nov 2010 at 7:24

GoogleCodeExporter commented 9 years ago 
This did not work for you in what way? Phoshare didn't run at all anymore, or 
you are still getting the same error as before?

Original comment by tsporkert on 10 Nov 2010 at 7:38

GoogleCodeExporter commented 9 years ago 
got still the same error:
Error: 'ascii' codec can't decode byte 0xcc in position 59: ordinal not in 
range(128)

Original comment by cstu...@hotmail.com on 10 Nov 2010 at 7:49

GoogleCodeExporter commented 9 years ago 
Thank you. For me the problem is solved changing line 454 of Phoshare v1.3 like 
you suggested in Comment 9. IPTC data is written properly to the file (I 
checked it via exiftool) so the error was just raised by trying to display some 
umlaut in the progress text box of the Phoshare window.

@cstu...@hotmail.com: Could you please post the one or two lines of output 
above the line "Error: 'ascii'..."? That's how I figured out where to look for 
the error I encountered. Maybe these lines give some hint what data Phoshare is 
trying to update/write.

Original comment by sebastia...@gmail.com on 10 Nov 2010 at 9:59

GoogleCodeExporter commented 9 years ago 
Sebastian,

Here are some more lines:

Creating folder /Volumes/Fotos/Link-to-Fotos/06.11.2010/Originals
New file: /Volumes/Fotos/Link-to-Fotos/06.11.2010/Originals/IMGP5939.jpg (link)
New file: /Volumes/Fotos/Link-to-Fotos/06.11.2010/IMGP5942.jpg (link)
Error: 'ascii' codec can't decode byte 0xcc in position 59: ordinal not in 
range(128)

Original comment by cstu...@hotmail.com on 11 Nov 2010 at 2:43

GoogleCodeExporter commented 9 years ago 
I do get the same problem when using Phoshare 1.3

It gives a message like this when it is copying the photo with the title 
"07_1547_DSCF1452 - Många ödlor"

Then it gives you this message:
New file: /Volumes/Files/Backup/BackupiPhotoLibrary-01/Madeira med 
föräldrarna/07 1547 DSCF1452 - Ma nga o dlor.jpg (copy)
Error: 'ascii' codec can't decode byte 0xcc in position 93: ordinal not in 
range(128)

And I was checking line 454 in 
/Applications/Phoshare.app/Contents/Resources/Phoshare.py

But did not find the same code as you had, instead I got

454     def make_album_basename(self, orig_basename, index, name_template):
455         """creates unique file name."""
456         album_basename = None

but on line 401 you have 

400                         su.fsenc(export_file),
401                         region_names[p],
402                         ','.join(str(c) for c in region_rectangles[p]),
403                         ','.join(str(c) for c in photo_rectangles[p]))
404                     persons_diff = True

That I changed into

401                         su.fsenc(region_names[p]),
402                         ','.join(str(c) for c in region_rectangles[p]),
403                         ','.join(str(c) for c in photo_rectangles[p]))
404                     persons_diff = True

But now still the same problem.

When I run Phoshare a second time for this event it passes for this photo but 
get stuck on next one.

Updating IPTC for /Volumes/Files/Backup/BackupiPhotoLibrary-01/Madeira med 
föräldrarna/07 1547 DSCF1452 - Ma nga o dlor.jpg because of GPS (0.000000, 
0.000000) vs (32.744999, -16.994900)
New file: /Volumes/Files/Backup/BackupiPhotoLibrary-01/Madeira med 
föräldrarna/08 1400 DSCF1587 - Brant  smal va g mot bergen.jpg (copy)
Error: 'ascii' codec can't decode byte 0xcc in position 105: ordinal not in 
range(128)

Suggestions?

Original comment by daniel.e...@gmail.com on 11 Nov 2010 at 5:44

GoogleCodeExporter commented 9 years ago 
Sorry for all the problems, and thanks for all the help trying to narrow this 
down.

There are actually two problems here:
1. the iPhoto library, IPTC metadata, and your desktop use different character 
encodings, and somewhere 
   in Phoshare a string is not properly re-encoded during processing. That triggers the error that you
   are seeing. In this particular case, it's probably trying to send an UTF-8 encoded character to a place that
   expects ASCII only.
2. Phoshare doesn't print a stack dump when that happens, so we don't know 
where in Phoshare this is
   happening.
Without the stack dump (2), we can only guess where (1) is happening. I have a 
test setup full of special characters, but cannot reproduce this error. I'll 
see what I can do about (2).

Original comment by tsporkert on 11 Nov 2010 at 6:13

GoogleCodeExporter commented 9 years ago 
Could you all please try Phoshare 1.3.1? I fixed one or two cases that could 
give an encoding error. But most importantly, Phoshare should now always print 
a stack dump if such an error occurs, which will point straight to the place in 
the code where the error occurs.

Thanks

Original comment by tsporkert on 12 Nov 2010 at 7:59

GoogleCodeExporter commented 9 years ago 
Thanks for the support!
Here's the new error message:

New file: /Volumes/Fotos/Link-to-Fotos/06.11.2010/IMGP5939.jpg (link)
Creating folder /Volumes/Fotos/Link-to-Fotos/06.11.2010/Originals
New file: /Volumes/Fotos/Link-to-Fotos/06.11.2010/Originals/IMGP5939.jpg (link)
New file: /Volumes/Fotos/Link-to-Fotos/06.11.2010/IMGP5942.jpg (link)
Error: 'ascii' codec can't decode byte 0xcc in position 59: ordinal not in 
range(128)

Traceback (most recent call last):
  File "phoshare_ui.pyc", line 731, in export_thread
  File "/Users/Hugo/Phoshare-2.app/Contents/Resources/Phoshare.py", line 751, in export_iphoto
  File "/Users/Hugo/Phoshare-2.app/Contents/Resources/Phoshare.py", line 721, in generate_files
  File "/Users/Hugo/Phoshare-2.app/Contents/Resources/Phoshare.py", line 552, in generate_files
  File "/Users/Hugo/Phoshare-2.app/Contents/Resources/Phoshare.py", line 218, in generate
  File "/Users/Hugo/Phoshare-2.app/Contents/Resources/Phoshare.py", line 317, in check_iptc_data
  File "tilutil/exiftool.pyc", line 76, in get_iptc_data
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 59: 
ordinal not in range(128)

Hope this helps.

Original comment by cstu...@hotmail.com on 12 Nov 2010 at 5:09

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
I was able to reproduce this error by importing an image with a non-ascii 
character into an iPhoto library. Could you give Phoshare-1.3.2 a try - that 
should fix this problem. It's available from the Downloads page, or here:

http://code.google.com/p/phoshare/downloads/detail?name=Phoshare-1.3.2.zip

Original comment by tsporkert on 13 Nov 2010 at 8:14

GoogleCodeExporter commented 9 years ago 
with 1.3.2 it run almost through.

New error message:

*************************
Updating IPTC for /Volumes/Fotos/iPhoto 
Library/Originals/2007/05.12.2007/Kabel.JPG because of date (None instead of 
2006-02-26 13:34:26)
New file: /Volumes/Fotos/Link-to-Fotos/2006-01/Sanierung NUe_55.jpg (link)
Error: 'ascii' codec can't decode byte 0xcc in position 65: ordinal not in 
range(128)

Traceback (most recent call last):
  File "phoshare_ui.pyc", line 731, in export_thread
  File "/Users/hugo/Downloads/Phoshare-3.app/Contents/Resources/Phoshare.py", line 751, in export_iphoto
  File "/Users/hugo/Downloads/Phoshare-3.app/Contents/Resources/Phoshare.py", line 721, in generate_files
  File "/Users/hugo/Downloads/Phoshare-3.app/Contents/Resources/Phoshare.py", line 552, in generate_files
  File "/Users/hugo/Downloads/Phoshare-3.app/Contents/Resources/Phoshare.py", line 218, in generate
  File "/Users/hugo/Downloads/Phoshare-3.app/Contents/Resources/Phoshare.py", line 317, in check_iptc_data
  File "tilutil/exiftool.pyc", line 118, in get_iptc_data
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 65: 
ordinal not in range(128)

Original comment by cstu...@hotmail.com on 13 Nov 2010 at 12:30

GoogleCodeExporter commented 9 years ago 
I think we're getting closer. This one was an obscure problem with bad date 
information in an image file. Besides hopefully fixing this problem, I've also 
wrapped all the places where errors like this can happen with an error handler 
so that the export will just continue

http://code.google.com/p/phoshare/downloads/detail?name=Phoshare-1.3.3.zip

Original comment by tsporkert on 14 Nov 2010 at 7:13

GoogleCodeExporter commented 9 years ago 
First of all, many thanks for your effort! I really appreciate that.
I'm sorry, this time i got an error in the very beginning:

*******************
Reading iPhoto database from /Volumes/Fotos/iPhoto Library...
Version 8.1.2 library with 17369 images
Phoshare.py --export "/Volumes/Fotos/Link-to-Fotos" --iphoto 
"/Volumes/Fotos/iPhoto Library" --events "." --originals --link --faces --iptc 
--gps
Scanning iPhoto data for photos to export...
Scanning existing files in export folder...
Exporting photos from iPhoto to export folder...
Error: a float is required

Traceback (most recent call last):
  File "phoshare_ui.pyc", line 737, in export_thread
  File "phoshare_main.pyc", line 746, in export_iphoto
  File "phoshare_main.pyc", line 716, in generate_files
  File "phoshare_main.pyc", line 547, in generate_files
  File "phoshare_main.pyc", line 261, in generate
  File "phoshare_main.pyc", line 208, in _check_need_to_export
TypeError: a float is required

Original comment by cstu...@hotmail.com on 14 Nov 2010 at 6:23

GoogleCodeExporter commented 9 years ago 
http://code.google.com/p/phoshare/downloads/detail?name=Phoshare-1.3.4.zip

will fix that.

Tilman

P.S. I'm beginning to regret my choice of Python as the programming language 
for Phoshare. I picked it because it allows users to easily customize Phoshare. 
You can just edit the source code (which is included in the package), restart 
the app, and it will use your changes. No need to install development tools or 
rebuild the application. But it also makes it a lot more fragile.  With a 
compiled language like Java, almost all of the issues that have surfaced in 
this thread would either not exist in the first place (Java uses Unicode 
strings internally and properly decodes/encodes  throughout), and the Java 
compiler would have caught the rest (like the date object mixup in 
Phoshare-1.3.3).

Original comment by tsporkert on 14 Nov 2010 at 7:39

GoogleCodeExporter commented 9 years ago 
Also like to thank you for all your effort Tilman. Very appreciated.
It is sad that Python make it that fragile, but also like your idea to use 
language that others could customize.

I also got this error with Phoshare 1.3.4

Updating IPTC for /Volumes/Files/Backup/BackupiPhotoLibrary-01/Madeira med 
föräldrarna/07 1433 PICT0006 - Daniel pa  flygplatsen.jpg because of persons 
( instead of Daniel Engvall)
Error: 'str' object is not callable

Traceback (most recent call last):
  File "phoshare_ui.pyc", line 737, in export_thread
  File "phoshare_main.pyc", line 746, in export_iphoto
  File "phoshare_main.pyc", line 716, in generate_files
  File "phoshare_main.pyc", line 547, in generate_files
  File "phoshare_main.pyc", line 276, in generate
  File "phoshare_main.pyc", line 409, in check_iptc_data
  File "tilutil/exiftool.pyc", line 206, in update_iptcdata
  File "tilutil/systemutils.pyc", line 110, in pout
TypeError: 'str' object is not callable

I am quite sure I miss something here ;) but should not "phoshare_main" be 
included in the package (neither in source). And when trying to run the 
Phoshare.py from the downloaded source with Python 2.7 it gives me error 
message when importing the phoshare_ui, picassaweb.

But if there is anything I could do to support you such as attaching example 
files that fails just let me know.

Thanks,
/Daniel

Original comment by daniel.e...@gmail.com on 14 Nov 2010 at 10:56

GoogleCodeExporter commented 9 years ago 
Daniel, that stack trace was enough information to locate another encoding 
problem, this time with the output of "exiftool". Apparently, something went 
wrong with a metadata update, and the error message contained some non-ascii 
characters. I think I've got that covered now, plus the "str" error. It will 
still have a problem with the metadata on this particular image, but Phoshare 
should now continue with the other images.

http://code.google.com/p/phoshare/downloads/detail?name=Phoshare-1.3.5.zip

Regarding the source files... Starting with Phoshare 1.2, I've been 
distributing Phoshare as a real MacOS app, to eliminate the dependency on 
Python on your system. The tool (py2app) that makes the package pre-compiles 
all sources files, and bundles them all into a binary .zip file. I've just 
dialed that back a bit, so that the Phoshare code itself remain included in 
source form. You can run it with your own Python interpreter after setting 
PYTHONPATH:
   Phoshare.app/Contents/Resources
   export PYTHONPATH=lib/python2.7:$PYTHONPATH
   ./Phoshare.py

And you can always download the sources from this site. I've just synchronized 
the SVN repository to version 1.3.5.

Original comment by tsporkert on 15 Nov 2010 at 1:00

GoogleCodeExporter commented 9 years ago 
Hi Tilman

Great work! Now it has run through without error.
Two last questions i got for the understanding:
- Phoshare doesn't fix the problems with the ascii codes, it just skips the 
fotos and leave the metadata unchanged in this cases -> is that correct?
- If used with the "hardlink" option, where does it store the Metadata. In the 
original file within the library?

Many thanks again.
Chris

Original comment by cstu...@hotmail.com on 16 Nov 2010 at 7:32

GoogleCodeExporter commented 9 years ago 
Chris,
1. No, Phoshare should do the right thing for all photos, including cases where 
the file names or metadata include non-ascii characters. However, should it 
encounter a situation where it comes across an unsolved encoding problem, it 
should now continue with the next image instead of aborting the entire export. 
If that happens, I'd like to know about it, so I can work on fixing that.
2. Yes, if you use the link option, the metadata of the image file stored 
inside the iPhoto library gets updated. Please make sure you have a working 
backup solution in place, just in case something goes wrong during the update, 
and the image file becomes unreadable. Phoshare uses the excellent exiftool 
program (http://www.sno.phy.queensu.ca/~phil/exiftool/) for all metadata 
operations, so it's highly unlikely that corruption occurs, but you never know.

Thanks for helping debug the encoding problems.

Tilman

Tilman

Original comment by tilma...@google.com on 16 Nov 2010 at 7:50

GoogleCodeExporter commented 9 years ago 
Also like to thank you for a great job Tilman..

I also ran Phoshare 1.3.5 on my library and went through and I checked some 
random pictures and all of them had the proper GPS, and keywords within :-D

Minor problem is that some of the swedish/unicode letters from the event are 
missing in the folders, but that I could live with. Don't fully see the pattern 
most of the event/folder names are ok while some are not using the same swedish 
letters.

Examples:

"Andreas Högvakt" = OK
"Besöker Claessons i Lysekil" = OK
"Cykloperna 50 år på Åland" = OK
"Ute med Eriks båt" = OK

The letters ö, Å, å for the events below became different
Event "Bilder från Mallorca" became folder "Bilder fra_n Mallorca"
Event "Jurmo Ålands skärgård" became folder "Jurmo A_lands ska_rga_rd"
Event "Mingel på Göteborg" became folder "Mingel pa_ Go_teborg"

But like I said, minor problem that would be nice if it could be anything done 
in the future.

Cheers,
/Daniel

Original comment by daniel.e...@gmail.com on 16 Nov 2010 at 10:08

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
That is currently done on purpose, to make the names compatible with a wide 
range of file systems. For example, if you are exporting onto a FAT formatted 
USB stick or a drive shared from a Windows computer, only ASCII characters, 
numbers, space, a few other characters are allowed. And another user has even 
asked for stripping spaces from the names (see 
http://code.google.com/p/phoshare/issues/detail?id=5) - which is almost the 
opposite of your request.

I already have a few requests for naming customization, like including the date 
in the folder names. I'll probably add a naming template option to Phoshare, to 
allow for some customization of the folder names.

Original comment by tilma...@google.com on 16 Nov 2010 at 10:25