search

pkolarov / folders2flickr

Automatically exported from code.google.com/p/folders2flickr
7 stars 2 forks source link

Very slow on large collections #8

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 

Suggestion: Use sqlite for storing history. Lookup of existing entries would be 
miles faster giving good performance.

Original issue reported on code.google.com by andrey....@gmail.com on 30 May 2011 at 12:07

GoogleCodeExporter commented 9 years ago 
yes, thats a possible upgrade, however it will be a bit more complex and 
"heavy", plus I have not noticed any real delay looking up entries...

Original comment by pkola...@gmail.com on 13 Jun 2011 at 8:00

GoogleCodeExporter commented 9 years ago

Original comment by pkola...@gmail.com on 13 Jun 2011 at 8:01

GoogleCodeExporter commented 9 years ago 
It has taken more than 24hr for 2GB for me ... don't know why ...

Original comment by carlzhu...@gmail.com on 23 May 2013 at 8:28

GoogleCodeExporter commented 9 years ago 
I have a roughly 250 GiB set of photos (perhaps 10,000 photos, my last 10 years 
plus a lot of older photos I've scanned) I'm trying to upload to Flickr, to 
take advantage of their 'free 1TiB'.  I ran this script on my Linux server, 
where I store these.  The script stopped doing I/Os or using CPU time after 
about 12 hours.  I let it run about another 24, but can't tell what it is 
doing.  There were no updates to the files during that time, it seems to be 
hung.  Is it just incompatible with such a large set?  Anyway, it looks like 
this would be a very useful script for people like myself who want to store 
relatively large image collections, but it doesn't appear to scale to Terabyte 
sized collections.

The sqlite suggestion might help address scaling issues.

Original comment by JoePowe...@gmail.com on 25 May 2013 at 3:17

GoogleCodeExporter commented 9 years ago 
If you have a large set, and if it's the first upload, just comment out the 
call to flickr2history.reshelf(...) in uploadr.py.
It's hanging there forever and it's not necessary.

Original comment by oliviers...@gmail.com on 28 May 2013 at 6:02

GoogleCodeExporter commented 9 years ago 
When I upload a large folder (~10GB) with subfolders, the "sets" are NOT 
created with the upload (even after upload finishes).

When I upload a small folder (300MB) with subfolders, the "sets" are created. 

How do I fix this inconsistency? More importantly, is there a way to create the 
sets after I finish uploading the large folder, which took many hours ...

Thanks.

Original comment by carlzhu...@gmail.com on 29 May 2013 at 2:13

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
I have the same problem carlzhu has.

Is there a way to create the sets after the files have been uploaded?

Thanks in advance!

Original comment by froemel....@googlemail.com on 31 Oct 2013 at 6:35