TypeError: not all arguments converted during string formatting

[koellcode]

createAlbum <type 'exceptions.Exception'> Traceback (most recent call last): File "/Users/thomas/lycheesync/lycheedao.py", line 178, in createAlbum cur.execute(query, (album['name'])) File "/Library/Python/2.7/site-packages/MySQLdb/cursors.py", line 187, in execute query = query % tuple([db.literal(item) for item in args]) TypeError: not all arguments converted during string formatting addFileToAlbum <type 'exceptions.Exception'> Traceback (most recent call last): File "/Users/thomas/lycheesync/lycheedao.py", line 273, in addFileToAlbum res = cur.execute(query, data) File "/Library/Python/2.7/site-packages/MySQLdb/cursors.py", line 205, in execute self.errorhandler(self, exc, value) File "/Library/Python/2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue OperationalError: (1048, "Column 'album' cannot be null")

[koellcode]

This exception occurs immediately when starting sync. In lychee not one Album was imported. I just tried a small folder with 5 Albums.

[koellcode]

I've started it with the -d option first (Lychee DB was empty). Now without -d it seems to work ... Some of the albums had German Umlauts. Maybe this could be the reason too (now renamed it)

[koellcode]

no, after the second album the same exception occurs (no umlauts) :(

On each retry. The whole first allbum works fine, if the second one starts to be processed, it fails with the exception above.

The albums appears in Lychee, but with no images (except the first one)

I've checked out: 641fea4b43dd34d6ee24a42d76c33cc6f08124ae

this one works..

[GustavePate]

Hi Thomas,

I think it's a corner case of the #23 bug. I recently told the database to handle "string protection".

I tried to add to my test case an album containg photos, all with german characters üÄ and ß. No error occurs (but the encoding in Lychee is not correct, I will fix this).

So, I'm a little stuck here. can't reproduce your use case.

To ease the problem reproduction, I will need a sample of one album name containg photos that trigger the problem. I will add them to my test cases. Please also tell me which command line do you fire to launch lycheesync (I will nedd the switch you use).

Thanks !

[GustavePate]

I just commit something that won't do any harm and that, perhaps, fixes your problem. The mysql driver wasn't told to be utf8 compliant, I think it will help even if it doesn't solve your problem. German characters are displayed perfectly in Lychee after a fresh sync.

Let me know if it solves your problem.

[koellcode]

Thank you I try it out in the evening. But after all, The umlaut should not be part of the Problems, cause I've tried the whole workflow again after renaming some of my albums, and the problem was there again (-d). But like I said, 641fea4 works totally for me. In some hours I can provide more Informations (exact workflow, software versions, cmd line calls..)

[GustavePate]

Thanks cause I'm stuck without additional info (especially a few images / directory name that cause the problem) I'm pretty sure it has something to do with character encoding, cause it's the only thing related to your problem which has change between 641fea4 and the present state.

[koellcode]

Ok I'm on a5fd86778544ca26c0345f760605f10dc04630a3 now. Same behavior. Output with command:

media-server% sudo python main.py /usr/local/var/www/LightroomGate/ /usr/local/var/www/lychee conf.json -d Password: createAlbum <type 'exceptions.Exception'> Traceback (most recent call last): File "/Users/thomas/lycheesync/lycheedao.py", line 181, in createAlbum cur.execute(query, (album['name'])) File "/Library/Python/2.7/site-packages/MySQLdb/cursors.py", line 187, in execute query = query % tuple([db.literal(item) for item in args]) TypeError: not all arguments converted during string formatting addFileToAlbum <type 'exceptions.Exception'> Traceback (most recent call last): File "/Users/thomas/lycheesync/lycheedao.py", line 276, in addFileToAlbum res = cur.execute(query, data) File "/Library/Python/2.7/site-packages/MySQLdb/cursors.py", line 205, in execute self.errorhandler(self, exc, value) File "/Library/Python/2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue OperationalError: (1048, "Column 'album' cannot be null")

Same as before... My Folder structure is: . .. .sync Aegypten Andechs Bad-Kohlgrub Bodensee-Bad-Schachen Chaeserugg Italien-Pompeij Italien-Rom Lanzarote

maybe this helps

[GustavePate]

Thanks.

The .sync is dodgy. could you add the -v flag to your command line, there will be more contextual information.

Thanks !

[GustavePate]

It's not the .sync folder, neither the - :( Really strange... with the -v flag we will see what folder causes the probleme

[koellcode]

media-server% sudo python main.py -d -v /usr/local/var/www/LightroomGate /usr/local/var/www/lychee conf.json

Program Launched with args:
* dropDB:True
* replace:False
* verbose:True
* srcdir:/usr/local/var/www/LightroomGate
* lycheepath:/usr/local/var/www/lychee
* conf:conf.json
* sort_by_name:True
* link:False
Program Launched with conf:
* dbHost:localhost
* db:lychee
* dbUser:lychee
* dbPassword:...
* thumbQuality:80
* publicAlbum:0
Other conf elements:
* user:_www
* group:_www
* uid:70
* gid:70

INFO album list in db: {} createAlbum <type 'exceptions.Exception'> Traceback (most recent call last): File "/Users/thomas/lycheesync/lycheedao.py", line 181, in createAlbum cur.execute(query, (album['name'])) File "/Library/Python/2.7/site-packages/MySQLdb/cursors.py", line 187, in execute query = query % tuple([db.literal(item) for item in args]) TypeError: not all arguments converted during string formatting INFO: adding to lychee /usr/local/var/www/LightroomGate/Aegypten/DSC01007.jpg addFileToAlbum <type 'exceptions.Exception'> Traceback (most recent call last): File "/Users/thomas/lycheesync/lycheedao.py", line 276, in addFileToAlbum res = cur.execute(query, data) File "/Library/Python/2.7/site-packages/MySQLdb/cursors.py", line 205, in execute self.errorhandler(self, exc, value) File "/Library/Python/2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue OperationalError: (1048, "Column 'album' cannot be null") ERROR: while adding to lychee /usr/local/var/www/LightroomGate/Aegypten/DSC01007.jpg INFO: adding to lychee /usr/local/var/www/LightroomGate/Aegypten/DSC01011.jpg addFileToAlbum <type 'exceptions.Exception'> Traceback (most recent call last): File "/Users/thomas/lycheesync/lycheedao.py", line 276, in addFileToAlbum res = cur.execute(query, data) File "/Library/Python/2.7/site-packages/MySQLdb/cursors.py", line 205, in execute self.errorhandler(self, exc, value)

[koellcode]

and now the second try without the -d flag (works with the first album, until the second and then BOOM):

media-server% sudo python main.py -v /usr/local/var/www/LightroomGate /usr/local/var/www/lychee conf.json

Program Launched with args:
* dropDB:False
* replace:False
* verbose:True
* srcdir:/usr/local/var/www/LightroomGate
* lycheepath:/usr/local/var/www/lychee
* conf:conf.json
* sort_by_name:False
* link:False
Program Launched with conf:
* dbHost:localhost
* db:lychee
* dbUser:lychee
* dbPassword: xxx
* thumbQuality:80
* publicAlbum:0
Other conf elements:
* user:_www
* group:_www
* uid:70
* gid:70

INFO album list in db: {'aegypten': 15L} INFO: adding to lychee /usr/local/var/www/LightroomGate/Aegypten/DSC01007.jpg INFO: adding to lychee /usr/local/var/www/LightroomGate/Aegypten/DSC01011.jpg INFO: adding to lychee /usr/local/var/www/LightroomGate/Aegypten/DSC01012.jpg .... .... INFO: adding to lychee /usr/local/var/www/LightroomGate/Aegypten/DSC01357.jpg INFO: adding to lychee /usr/local/var/www/LightroomGate/Aegypten/DSC01365.jpg createAlbum <type 'exceptions.Exception'> Traceback (most recent call last): File "/Users/thomas/lycheesync/lycheedao.py", line 181, in createAlbum cur.execute(query, (album['name'])) File "/Library/Python/2.7/site-packages/MySQLdb/cursors.py", line 187, in execute query = query % tuple([db.literal(item) for item in args]) TypeError: not all arguments converted during string formatting INFO: adding to lychee /usr/local/var/www/LightroomGate/Andechs/IMGP3197.jpg addFileToAlbum <type 'exceptions.Exception'> Traceback (most recent call last): File "/Users/thomas/lycheesync/lycheedao.py", line 276, in addFileToAlbum res = cur.execute(query, data) File "/Library/Python/2.7/site-packages/MySQLdb/cursors.py", line 205, in execute self.errorhandler(self, exc, value) File "/Library/Python/2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue OperationalError: (1048, "Column 'album' cannot be null")

[GustavePate]

Thank you for your help.

So OK, bad logging strategy, I log the album being inserted after the insert, if it's ok. So I have no more clue. In my last commit I display more contextual info if an exception occurs. Could you git pull and replay the -d -v command line. It should give us the album that fails and with luck the reason why.

Thanks in advance.

[koellcode]

media-server% sudo python main.py -v -d /usr/local/var/www/LightroomGate /usr/local/var/www/lychee conf.json Password:

Program Launched with args:
* dropDB:True
* replace:False
* verbose:True
* srcdir:/usr/local/var/www/LightroomGate
* lycheepath:/usr/local/var/www/lychee
* conf:conf.json
* sort_by_name:True
* link:False
Program Launched with conf:
* dbHost:localhost
* db:lychee
* dbUser:lychee
* dbPassword:xxx
* thumbQuality:80
* publicAlbum:0
Other conf elements:
* user:_www
* group:_www
* uid:70
* gid:70

INFO album list in db: {} INFO try to createAlbum:('aegypten', '1424469087', '0') ERROR createAlbum:aegypten -> {'path': '/usr/local/var/www/LightroomGate/Aegypten', 'relpath': 'Aegypten', 'id': None, 'photos': [], 'name': 'aegypten'} Traceback (most recent call last): File "/Users/thomas/lycheesync/lycheedao.py", line 182, in createAlbum cur.execute(query, (album['name'])) File "/Library/Python/2.7/site-packages/MySQLdb/cursors.py", line 187, in execute query = query % tuple([db.literal(item) for item in args]) TypeError: not all arguments converted during string formatting INFO: adding to lychee /usr/local/var/www/LightroomGate/Aegypten/DSC01007.jpg ERROR addFileToAlbum originalname:DSC01007.jpg originalpath:/usr/local/var/www/LightroomGate/Aegypten id:14244690873700 albumname:aegypten albumid:None thumbnailfullpath:/usr/local/var/www/lychee/uploads/thumb/5fd9e9ac846fe6d5dc4e94421ab68efe.jpg thumbnailx2fullpath:/usr/local/var/www/lychee/uploads/thumb/5fd9e9ac846fe6d5dc4e94421ab68efe@2x.jpg title: description:2015-02-20 22:51:27 url:5fd9e9ac846fe6d5dc4e94421ab68efe.jpg public:0 type:image/jpeg width:4249.0 height:2823.0 size:4618 KB star:0 thumbUrl:5fd9e9ac846fe6d5dc4e94421ab68efe.jpg srcfullpath:/usr/local/var/www/LightroomGate/Aegypten/DSC01007.jpg destfullpath:/usr/local/var/www/lychee/uploads/big/5fd9e9ac846fe6d5dc4e94421ab68efe.jpg sysdate:2015-02-20 systime:22:51:27 checksum:8b4f065c6d8dd686aae8da3dbf11fe03c9be1d4b Exif: iso: aperture: make: model: shutter: focal: _takedate: takedate: taketime: orientation: 0

ERROR addFileToAlbum ('14244690873700', '5fd9e9ac846fe6d5dc4e94421ab68efe.jpg', 0, 'image/jpeg', 4249.0, 2823.0, '4618 KB', 0, '5fd9e9ac846fe6d5dc4e94421ab68efe.jpg', None, '', '', '', '', '', '', '1424386800', '2015-02-20 22:51:27', 'DSC01007.jpg', '8b4f065c6d8dd686aae8da3dbf11fe03c9be1d4b') ERROR addFileToAlbum <type 'exceptions.Exception'> Traceback (most recent call last): File "/Users/thomas/lycheesync/lycheedao.py", line 277, in addFileToAlbum res = cur.execute(query, data) File "/Library/Python/2.7/site-packages/MySQLdb/cursors.py", line 205, in execute self.errorhandler(self, exc, value) File "/Library/Python/2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue OperationalError: (1048, "Column 'album' cannot be null")

[koellcode]

I wonder why the album names are all lowercase. Btw, the dates from the images are the lycheesync import dates, not the exif dates (thats what i've expected)

[GustavePate]

Lycheesync lower cases albums name and the exif date are taken into account if they exists (and if they are parseable by the library) but it's another subject, we can keep it for later ;)

For now, your lycheesync doesn't manage to execute:

     select id from lychee_albums where title='aegypten'

which is sad enough for now ;)

Are you sure that the db user you configured lycheesync with has the right to select things ? Obviously, he has the right to insert things, because the album line is created before this select.

After a failed -d run (so the first album is created but empty), If you log into mysql with the user/password from conf:

             mysql -u lychee -p
             use lychee;
             select id from lychee_albums where title='aegypten';

does it select you the line ?

If yes, I commit something to be closer to mysqldb documentation but I don't believe it will change anything, could you try it ?

Thanks in advance.

[koellcode]

Here my output:

MariaDB [lychee]> select id from lychee_albums where title='aegypten'; +----+ | id | +----+ | 45 | +----+ 1 row in set (0.00 sec)

MariaDB [lychee]>

The album id is correct i've crosschecked it. Like i've said before, the older commit works completely. The lychee frontend shows me all successful imported albums too. Its definitely not a DB rights problem :(

[GustavePate]

"the older commit works completely" => not really, the older commit was a complete mess with folders containing apostrophes / quotes ;)

"Its definitely not a DB rights problem" I think its one indeed ;)

"MariaDB" I never tested lycheesync with MariaDB, I checked, the database driver I use seems to be compatible but string protection is somewhat a corner case and maybe buried in the longer than my arm "incompatibilities between mysql and mariadb' listing.

However, we are not sure, I just commit something that will display you the query as it's send to the database. The line with the faulty sql command will begin with:

 ERROR while executing: 

[koellcode]

ERROR while executing: insert into lychee_albums (title, sysstamp, public, password) values ('andechs', '1424474586', '0', NULL)

Looks good for me, and executes fine in mysql client .... he tries this at any album.

okay the albums are now in lychee, but without images :D

[GustavePate]

wrong for me:

Your exception throws on:

  cur.execute(query, (album['name']))

The associated query is of the form

  select id from lychee_albums where title='ok'

The query I log is the last that make it to the db engine, even if it's a completly wrong query. For one reason, this query is never send to your db engine, the driver implodes before that.

I know that the driver I use is deeply linked to mysql runtime (in your case mariadb...) In my refactor wish list I wanted to switch to another driver (pure python implementation). Either that, or I find another way to protect strings, independently from the db engine.

I just commited a last something, I don't think it will make a difference but if you can try, who knows ?

If it does nothing, just fallback to 641fea4 (or install mysql, which is not a good solution at term), I will notify you here when I will have implemented one of the two solution mentioned above (another driver / custom string protection).

see you,

And many thanks for your help on this particular difficult bug !

[GustavePate]

I switch for a quick fix, a more standard way of escaping strings based on the actual driver. It should be compatible with mariadb.

Tell me if it's better: 23881bcae99574592884c478234c1f539e4979ad

[koellcode]

Looks good. I've reimported all my albums with successfully :)

[GustavePate]

ok, just saw you comment on my yesterday "last try" commit => lol. I'm glad this last solution works.

Thanks for your involvement in squashing this bug.