Existing folders not used

[i814u2]

I'm experiencing an issue when resuming a set upload (added more photos to a folder), or when the total items are greater than 21.

The existing folder is not used in almost all cases. I thought for sure it worked in one test, but that may have just been less than 21 photos/movies in the upload.

Example: I just uploaded 324 photos, 1 movie. All from one folder. The movie and 20 photos went into a set. The other 303 photos were tagged properly but not placed into any set.

The folder/set name is "2014-01 Costa Rica".

I know that the one that re-used an existing set did not have a dash in it, but was something like "200809 description". However, I've had a similarly named folder fail to use the existing set.

The debug log shows that tags2set.py finds the proper set ID, and then it just lists all 325 items as it adds each. Yet the actual set only contains the first 21 items.

Let me know if I can help at all.

I'm running this on an Ubuntu 13.04 server. I have the tags2set.py from commit b7e486d65d.

[richq]

Hmm, I wonder if dashes and spaces together go wrong? Seems far fetched, but checking my sets I have some with spaces, some with dashes, but not both together... If you find out more let me know, I'll keep an eye out for mess ups - I thought I'd fixed this problem with that version :(

[i814u2]

I just tested it alongside my exif issue(#16), and it didn't reuse a set which it just created. The name contains no spaces: iphonemix2test

I'll re-post the info about the process, for anyone who only lands here: So I have a new test set with 21 photos, but between the two runs, I uploaded 26 photos.

Quick recap of the upload counts: Attempt 1 21 uploaded set created, 21 added 3 uploaded (not in a set) 1 failed Attempt 2 1 uploaded (not in a set) 1 failed (same file as 1st attempt) 1 uploaded (not in a set)

Is it failing simply because 1 file fails? I would expect that it would place all completed uploads into a set, then report on failures.

[richq]

I don't think so, but maybe. I've had photos fail too and I don't think they have messed up the sets. Can you upload that failed photo via the web to flickr? I've used exiftran in the past to fix exif data that flickr just fails to parse, even though the JPEG itself is fine.

[i814u2]

Yes, manually uploading this file works. The main issue here is that the existing set is not used.

I have files with issues across a little more than 650 sets (can't remember the exact number right now). So even if I fix the files (between exif-reading hacks and exiftool fixes), I need the existing set to get used so I don't need to manually add various items to hundreds of sets.

Also, even if the files upload fine, it seems to refuse to use existing sets for me (past 21 items).

[richq]

OK, thanks for the info.

On 02/23/2014 09:19 PM, i814u2 wrote:

Yes, manually uploading this file works. The main issue here is that the existing set is not used.

— Reply to this email directly or view it on GitHub https://github.com/richq/folders2flickr/issues/15#issuecomment-35841993.

[i814u2]

I just moved my history file to an archive folder, made sure only one directory was in the monitored directory, updated the tags2set.py, and re-ran the upload.

It uploaded 21 photos, created the set, added the files (21), continued to upload the last 5 photos but never added them to the set. Directory name is still "iphonemix2test".

So it still is not adding the files to the existing set, at least not when the set was created during the same run. I didn't test having a set, then running upload to see if they all go (testing that now).

[i814u2]

I reset the test again (removed set/photos from flickr, removed history file, etc). I put 4 photos into a directory and uploaded. I verified that flickr had a new set with 4 photos. Added 22 photos to the directory and uploaded. None of the 22 photos were added to the existing set.

[richq]

I appreciate the thorough testing, but are you really sure that you're running the updated code? I ask because I've been unable to reproduce the problem :-/

I've ran the 4 + 22 test and it worked as expected. 4 in a set, then 26 in the same set. The log says first run "tags2set: create set iphonemix2test with photo <Flickr Photo 123456789>" then on the next run "tags2set: Found existing set iphonemix2test".

Then with 26 photos in the directory, I removed the set + photos from the web, and reran the test. That also worked. After 21 it created the set, then kept going and added the remaining ones to that set. In the log it says "created new set", then "added 21 photos".

[i814u2]

Great question. I'm newer to git (previously experience limited to cloning projects), so I may have messed up with my tinkering. I actually wasn't sure at one point myself. I removed the file, did a pull and the file existed with the new line at the end, so I figured I was good.

I just backed up my directory, cloned f2f again, copied over the .flickrToken and uploader.ini files and ran another test. For this test I took a folder of images not from a Canon camera (to avoid the EXIF issue I was having). That allowed me to run only the code as you have it in the latest commit (I verified I have the latest commit with git log. If that's not the best practice, please correct me). It uploaded 21 images, created a set, added them, then uploaded the next 22 files. It did not add those files to the set. The directory/set name is "RANDOM". My python version is 2.7.4 Is there anything else I can test, or anything I've missed?

[richq]

That all sounds correct - I mean your process does, that it fails is a disaster!

Normally when the script finishes it adds the "remaining" files to sets as needed. It should create or "edit" (i.e. add photos to) a set every 21 files so as not to leave you with a load of loose photos if something goes wrong. Previously it ran for hours, adding photos, and only added sets at the very end. If there was a failure of the network in the middle, you got no sets at all. But if the script runs and finishes normally, all files should be added to their corresponding sets.

Were there any errors in those 22 files? I guess not if they uploaded OK.

I'll give that test a go too later though it might be the weekend now before I get chance to check this again.

[i814u2]

All of the files worked fine. 43 existed in Flickr when I was done. 21 were in the set "RANDOM" I had no issues mentioned for the last 22.

On Feb 27, 2014, at 11:43, Richard Quirk notifications@github.com wrote:

That all sounds correct - I mean your process does, that it fails is a disaster!

Normally when the script finishes it adds the "remaining" files to sets as needed. It should create or "edit" (i.e. add photos to) a set every 21 files so as not to leave you with a load of loose photos if something goes wrong. Previously it ran for hours, adding photos, and only added sets at the very end. If there was a failure of the network in the middle, you got no sets at all. But if the script runs and finishes normally, all files should be added to their corresponding sets.

Were there any errors in those 22 files? I guess not if they uploaded OK.

I'll give that test a go too later though it might be the weekend now before I get chance to check this again.

— Reply to this email directly or view it on GitHub.

[richq]

Turns out that I was using an ancient version that worked for this case! How embarrassing. To fix the issue I'm going to add unit tests, so it is less annoying to check this stuff. Thank you for your patience and thorough testing of this problem, if only all bug reports were this complete :)

[richq]

this should finally be fixed with commit 478a38f26e60971fedcbaa49dd512c2a2cbb1cd5

[i814u2]

It does work now. 43 photos added to one set. The only odd part (not worth re-opening this issue) is that it always says "Generating set" instead of "Adding to". Also, it takes quite a while (after that message is displayed) for it to keep uploading. Just a note to keep in mind for future updates.

Thanks for all the prompt attention. Once I wrap my head around the code more, I'll try to help improve where I can.

[richq]

Yeah, I've fixed that "generating" thing, it did look a bit shocking. The actual updating of the set seems to take surprisingly long on the flickr side, so not much that I can do there. Perhaps I'll tweak the "every X" setting so it can upload more photos before trying a set (or if it sees that it's at 90%, don't create a set that will be updated really soon) but that's for another day...