Flickr2Piwigo sees only 190,168 photos and presents errors. It chops up long album titles.

[ElliotMcGucken]

Thank you! I have just set piwigo up on a brand new Centos server! I hope to help you debug Flickr2Piwigo. At the moment, it is transferring files, with some issues:

Here is a screenshot!

Some things I've noticed:

1. It says "Selection 190168 elements ready for impotration". I actually have over 300,000 images total! Most of those are in albums too!

Here is the site: http://66.45.225.82/pw/piwigo/

2. Flickr2Piwigo is taking long album titles and chopping them up, so I am getting multiple album titles.

For instance, it takes an album title from flickr which has 85 photos, chops it up, and creates five albums, each which has 85 photos:

http://66.45.225.82/pw/piwigo/

Albums: **- and Beautiful! Athena! [85]

• Fit [85]
• Thin [85]
• and a Gold 45 Revolver Gun! Short Shorts Denim Daisy Dukes Jeans Shorts! Tall [85]
• Cowboy Boots [85]
• Pretty Cowgirl Model Goddess with Cowboy Hat [85]**

The good news is that it is transferring photos! Thanks!

Also, a question:

Is there a way to run Flickr2Piwigo from the linux shell or command line, rather than from a browser?

I have a dedicated server where I would like to leave it running so as to transfer all my images.

A related question to my above question is that if one closes the browser running Flickr2Piwigo and then opens a new browser on a new computer and runs Flickr2Piwigo, will Flickr2Piwigo continue where it last left off when the first browser was closed? Or will Flickr2Piwigo begin all over again, and download every gallery starting from the very first gallery again?

Thanks! :)

[samwilson]

Thanks for reporting this; I'll hopefully get to examining things this weekend.

And no, if you start again it'll do it all again (let's aim to fix that though! and maybe add a cronjob possibility, for on-going syncing).

[ElliotMcGucken]

Thanks Sam!

So if I start it again, will it overwrite the images and albums, or will it add the same images again under different names and in different albums? Will it take up new disk space with the exact same images, as it downloads them again?

[ElliotMcGucken]

It seems that if it doesn't download every single image in one go, then one must delete all the photos and databases and reinstall and start over.

I guess this means that each time one needs to update it with some new flickr photos, one must delete the installation, database, and all the images downloaded so far, and reinstall and start over?

Thanks!

[samwilson]

No, if it's not working correctly now, we'll fix it so it does work correctly. There will be no need to delete the database! :)

[ElliotMcGucken]

Thanks! After downloading 8,000 photos and closing one browser, I opened a new one and logged in. Flickr2Piwigo has put me in an endless loop for the last few hours. I press "Import All my Pictures," it says "processing" for ten or fifteen minutes, and then takes me back to the beginning, and no photos are added. I have done this a couple dozen times over the last few hours, so I will probably just reinstall and start the whole thing over.

[ElliotMcGucken]

Greetings! So I deleted the old installation last night and started with a brand new one. I left the browser running overnight, and after it transferred a few thousand photos from flickr to piwigo, the browser crashed.

So now I am trying to start the transfer again, and again I am in a loop. I press "Import All my Pictures," it says "processing" for ten or fifteen minutes, and then takes me back to the beginning, and no photos are added. I can list all my albums, but there is no way to transfer photos.

It seems that if Flickr2Piwigo does not successfully transfer every photo in one single browser-based run, then there is no way to transfer the photos it failed to transfer?

Thanks!

P.S. Each time I get to the "Begin Transfer" screen, I hit the button to begin the transfer, and it takes me back to "Import all my Pictures" screen. I hit the "Import all my Pictures" and after ten minutes it takes me to the "Begin Transfer" screen. I hit the button to begin the transfer, and it takes me back to "Import all my Pictures" screen and again and again. This is the loop i have been stuck in in two different installations.

[ElliotMcGucken]

A simple, and great, solution would be a simple linux shell command to transfer all the photos for flickr.

It would likely simplify your work, and solve 90%+ of the transfer needs! :)

And if the transfer died, the script could just pick up where it last left off?

Thanks! :)

[samwilson]

Yes, I'm also thinking that a shell command would be a great addition, and perhaps will work on such a thing before getting some of these other bugs ironed out. You're right, it wouldn't be too hard.

[ElliotMcGucken]

Thanks! Yes the shell command would make Flickr2Piwigo even more popular, and after it works, it could easily become a browser button used during the setup of Piwigo too. :)

Regarding learning about object-oriented programming for php and the Flickr API, what resources/books might you recommend? Thanks! :)

[ElliotMcGucken]

Hello Sam!

Might you have any idea when a command-line script for Flickr2Piwigo would be available?

It would be great for new installations! Simple functionality would rock:

1. Import all flickr photos & data into fresh Piwigo installation.
2. Pick up where it left off, on case the import dies during import.

And too, the above functionality would also be able to update the Piwigo installation, via a regular chron job or just running the script now and then.

Any idea on how long that might take?

Thanks!

[samwilson]

The CLI tool is being tracked in #13.

As for books, I'm not really sure. Just start coding something you need! And go from there. Developing a library is a good learning experience, because you're thinking from the point of view of programmers, rather than users, and so you make nicer APIs. :-)

[samwilson]

@ElliotMcGucken I have started work on this, and would love any feedback. The PR is not yet complete, but is working for the "import all" function (it has broken some other things, that I'll fix tomorrow). If you could test it it'd be great! You just need to checkout the import-all branch. The album names should be working too.

[ElliotMcGucken]

Thanks Sam!

I installed your import-all branch in a brand new installation of Piwigo!

It now sees all my photos, but it doesn't seem to be transferring them. After about 15 minutes of "processing," it still shows this:

There is no sign of the new Piwigo database being populated, nor of the photos actually being transferred.

Should I just wait longer for the transfer to begin?

Also, if I stop it and restart it, should it still work just the same?

Thanks! Elliot

[ElliotMcGucken]

Hello Sam!

I did another Fresh Install of your import-all branch in a brand new installation of Piwigo!!

It is able to list all the albums after a couple tries/reloads and some errors.

But then, when I try to import the photos, it doesn't seem to do anything--again it just hangs on this page, saying that it is "Processing:"

Thanks!

[samwilson]

Hm, bother. :)

Yep you should be able to stop and start all you like.

Could you try opening the javascript console (pressing F12 usually does it) and reloading the page and seeing if there are any errors output there?

I'll keep working on it.

[ElliotMcGucken]

No Javascript errors seen!

[samwilson]

I think this is because of the deprecated function warning you're getting.

• What version of PHP are you using?
• What database type?
• Can you set $conf['show_php_errors'] = 0; in your local/config/config.inc.php to hide the notices, and see if anything changes.
• In the 'network' tab of the developer tools, can you click on the last request that starts with ws.php and have a look at the response that it's getting?

Thanks!

[ElliotMcGucken]

Thanks! There is no config.inc.php file in that directory:

Do I need to create config.inc.php in that directory? What all does it need?

Where might I find the "developer tools"? Are they in Piwigo?

Here is my PHP version:

Here is my MYSQL version:

Thanks!

[samwilson]

The local config stuff is explained here: http://piwigo.org/doc/doku.php?id=user_documentation:use:features:conf_locale — basically it involves creating that file if it doesn't exist and putting the following in it:

<?php

$conf['show_php_errors'] = 0;

(Or you can do the equivalent setting in your php.ini file by setting the error_reporting setting.)

The problem is that the create_function() function has been deprecated in PHP 7.2. The fix for this seems to be being tracked in https://github.com/Piwigo/Piwigo/pull/834

Oh and "Developer tools" is the name of that sidebar that appears when you hit F12. It's jolly useful, if a little confusing!

[ElliotMcGucken]

Thanks Sam!

Adding the file config.inc.php makes it work now:

<?php

$conf['show_php_errors'] = 0;

It is importing! Will let you know how it goes!

Thanks,

Elliot

[ElliotMcGucken]

Update! It froze after importing 275 photos and hung for ten minutes or so.

So I restarted it, and it looks like it is starting the import all over from the very beginning?

Is it actually re-downloading all the files again? Or is it skipping the ones it already has?

If I stop it and hit refresh again, it starts all over again.

Does this look right?

P.S. It DOES look like it is re-downloading everything again, as it is eating up new disk space.

[samwilson]

it shouldn't re-download: it checks the file name of each photo before downloading. The Flickr ones are given names like flickr-<username>-<id> and if it finds a match it'll say "Already imported: " in the little popup message. That's the idea anyway!

I've updated the patch; you should redownload it.

It will start the download from the beginning each time, note. But it'll go a lot quicker for the ones it's already got.

Also, make sure to purge your browser cache when reloading the import page (ctrl-shift-r) to make sure it's running with the latest javascript.

[samwilson]

Oh, just to clarify, if you're running this from a git checkout, you'll need to:

git fetch
git reset origin/import-all --hard

because I'm doing a single-commit patch.

[ElliotMcGucken]

Thanks Sam!

Just to be clear, just now I made a brand new, fresh installation of Piwigo and your Flickr2Piwigo import-all branch. In order to install your Flickr2Piwigo import-all branch, I just now entered:

git clone -b import-all https://github.com/mistic100/Flickr2Piwigo.git

Is that enough to give me the latest version, on a brand new installation made just now?

Or must I also enter:

git fetch git reset origin/import-all --hard

And if I need to enter these commands too, can you please provide the full text of these commands for this case, without abstraction?

Thanks!

Elliot

[ElliotMcGucken]

Greetings Sam!

The new installation seemed to be working. It seemed to pick up where it left off, but then, when I examine it, it says it is redownloading photos from the first set, even though the Processing number is far higher, as shown here:

Is it re-downloading the images from the first set still, thusly beginning all over, even though the processing number is far higher?

Also now and then an "Error 500" appears.

Thanks! Elliot

[ElliotMcGucken]

Dear Sam,

Yes it seems that when the browser is refreshed, Flickr2Piwigo is still re-downloading the photos from flickr again, and thus the albums have fare more photos than the original flickr albums. For instance, see how this album has 517 photos?

On flickr, that same gallery only has 228 photos!

So after I hit the upload, even though the Flickr2Piwigo "processing" appears to quickly skip the first 3,000 photos, it actually begins all over again, loading all the same photos into the gallery, so that now there are copies.

Note how there or two or three of everything now:

Thank you!

[ElliotMcGucken]

Dear Sam,

So when I close the Chrome Browser and start the Flickr2Piwigo process all over again, it again begins at 0--it doesn't skip any images, and begins downloading the very first, with the Processing Counter completely reset to 0:

[samwilson]

This is strange. Perhaps something is going on with the filenames, or your username (which is included in the filenames). I'm not sure how, on a page reload, it is starting from the higher number? It should start again from 0 and count up, and skip any that it's already imported (but it does have to check, because you may have uploaded more to Flickr in the meantime).

I have added some debug logging statements. You should have a daily log file in _data/logs/ that will contain some useful things. Could you try running again with the most recent code?

Thanks!

[ElliotMcGucken]

Thanks Sam! I just made a brand-new fresh install of everything.

I copied the log file of today's brand-new installation here for you to view: http://66.45.225.82/logs/ http://66.45.225.82/logs2/ (logfile for same install after a couple more browser refreshes--just a bit more log data)

Flickr2Piwigo uploaded the first album, and then just started giving me errors for about twenty minutes, even though it was still eating up disk space as if images were being transferred, even though they weren't being placed in any albums.

Even though it seemed to transfer these photos (from the second flickr album which it failed to create): http://66.45.225.82/18pw/piwigo/index.php?/recent_pics/start-330 , it failed to create the album and place them in the album, and when I click on the thumbnail, there is no photo--just a blank page: http://66.45.225.82/18pw/piwigo/picture.php?/332/recent_pics

So then I refreshed the browser, it imported one extra photo I had uploaded to flickr, and then it just started giving me errors.

So all in all, it was only able to import one album.

I just refreshed the page again. Right now, the "progress" is still stuck on 0 , and it is only giving me errors:

It says I have 368 photos, but most of them do not lead to a larger image page when clicked: http://66.45.225.82/18pw/piwigo/index.php?/recent_pics/start-330

Only 161 photos are in an album and clickable, as seen here: http://66.45.225.82/18pw/piwigo/index.php?/category/1

Hopefully the log files help you!

[samwilson]

You've still got PHP errors turned off don't you? I'm wondering if there might be some problem with PHP 7.2 that we're not seeing properly because of that. Do you have the option of using PHP 7.1 instead? And then turning PHP errors on again, and we can see what might be coming up?

It seems that it is correctly determining duplicate files and not importing them:

[2018-03-18 17:09:31]   [DEBUG] Attempting to import Flickr photo 39042363960
[2018-03-18 17:09:31]   [DEBUG] No duplicate filename found for flickr-45SURF Hero's Odyssey Mythology Landscapes & Godde-39042363960
[2018-03-18 17:09:33]   [DEBUG] Photo already imported (determined by checksum) with Piwigo ID: 209

so I'm not sure why it's ending up with duplicates (unless it's importing more than one to start with, before finding the dupe; but that's confusing).

I'm sorry for all this drama. I hope we can figure it out! Certainly, being able to turn PHP errors on again for things like http://66.45.225.82/18pw/piwigo/picture.php?/333/recent_pics will help a lot.

[ElliotMcGucken]

Thanks Sam!

I downgraded my server to php 7.1, and all the former errors associated with php 7.2 naturally disappeared!

I did a brand new fresh install of everything! http://66.45.225.82/19pw/piwigo/

This time Flickr2Piwigo downloaded the first two albums successfully, but then it got stuck on the third album, which it was unable to create. It does seem to import the pictures, but then, when one clicks on the thumbnail, it gives the php error shown below. I think there is a problem with Flickr2Piwigo handling the album and photo titles for this particular album?! For instance, see here and click on any of the images,none of which made it into an album as their flickr album was not created:

http://66.45.225.82/19pw/piwigo/index.php?/recent_pics/start-495

Clicking on a thumbnail gives an error:

Here is the logfile: http://66.45.225.82/logs19/

Also, it was unable to create the album. (This problem also seemed to happen in yesterday's installation).

Here is the album title:

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

Is there something there that can be better handled with a simple php/mysql command in your code? Fixing it for this album will likely fix it for many!

Here is the photo title of each photo in the album:

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

And here is the description:

Irish Lass St. Patty's Day! 45SURF Swimsuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, Helen, and Aphrodite! Sexy Hot dx4/dt=ic Model! Cowboy Boots, Cowboy Hat & Bikini!

Support epic fine art! 45surf ! Bitcoin: 1FMBZJeeHVMu35uegrYUfEkHfPj5pe9WNz

My Epic Book: Photographing Women Models!
geni.us/m90Ms Portrait, Swimsuit, Lingerie, Boudoir, Fine Art, & Fashion Photography Exalting the Venus Goddess Archetype: How to Shoot Epic ...

Epic! Beautiful Surf Fine Art Portrait Swimsuit Bikini Models!

Follow me friends! facebook.com/mcgucken instagram.com/elliotmcgucken facebook.com/goldennumberratio instagram.com/45surf

Epic books, prints, & more! geni.us/aEG4

Exalt your photography with Golden Ratio Compositions! geni.us/eeA1 Golden Ratio Compositions & Secret Sacred Geometry for Photography, Fine Art, & Landscape Photographers: How to Exalt Art with Leonardo da Vinci's, Michelangelo's!

Epic Landscape Photography: geni.us/TV4oEAz A Simple Guide to the Principles of Fine Art Nature Photography: Master Composition, Lenses, Camera Settings, Aperture, ISO, ... Hero's Odyssey Mythology Photography)

Epic Art & Gear for your Epic Hero's Odyssey: geni.us/9fnvAMw

Enjoy my physics dx4/dt=ic celebrating the light, time, and dimension of photography!! Light Time Dimension Theory: The Foundational Physics Unifying Einstein's Relativity and Quantum Mechanics: A Simple, Illustrated Introduction to the Physical amzn.to/2A4IMfM

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

Thanks Sam!

P.S. The main web page still shows a lot of errors, and the processing number lags behind what it has transferred. For instance, while it is stuck at "160 Processing," it is actually stuck after downloading 666 photos:

It has halted after 666 photos: http://66.45.225.82/19pw/piwigo/index.php?/recent_pics/start-660

[ElliotMcGucken]

Just a small update. After refreshing Flickr2Piwigo, I let it run for a couple more hours, and there has been no new progress at all.

It is still stuck at 666 total photos: http://66.45.225.82/19pw/piwigo/index.php?/recent_pics/start-660 and the "Processing" is stuck at 1:

I have copied the most recent log files to 19logs_2: http://66.45.225.82/logs19_2/

Here is the latest screenshot of the browser, after a reload running for two hours:

[samwilson]

Right, I think I've found and fixed a few issues. Your username has an ampersand in it, which was confusing the checking; the reporting back of checksum-matched already-imported photos was not working; and there was a chance that long album names were being truncated prior to creation, and so the creation was failing.

The other thing I'd like you to check is the 'orphan' photos that get imported. If there's no album on Flickr, then the photo will be imported as an orphan. This is fine, but I think you won't be able to view it in the front end till you've given it an album.

Could you have another crack with the latest code, and see how you go? :-) Thanks!!

[ElliotMcGucken]

Thanks Sam!

Here are the log files for today's install: http://66.45.225.82/20logs/

I created a brand new fresh install of everything today, and I'm seeing some of the same errors as yesterday (I think the errors are due to characters such as apostrophes and/or periods in the photo titles). Again, when it gets to the album:

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

It fails to create the album again. Just like yesterday, it creates the thumbnails:

http://66.45.225.82/20pw/piwigo/index.php?/recent_pics/start-480

But when one clicks on a thumbnail, one gets the same error again:

The titles of the photos are the same as the album title:

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

This character string has 251 characters in it. Might that be a problem after addslashes and all?

Are the apostrophes ' in "St. Patty's" and "St. Patrick's Day" problems perhaps? Or are the periods in "St." a problem?

I have long titles, but once your code is debugged for the long titles, it should be great!

Orphan Photo Test To test the orphan photo importation, I uploaded 274 new photos to Flickr without placing them in an album. It appears they were imported by Flickr2Piwigo as the browser reported Success! with each photo title "Nikon D810 Beach Portraits Santa Barbara! White Summer Dress! 45Epic 45SURF Swimsuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddesses! Athletic Action Portraits of Swimsuit Bikini Models! High Res Venus! Sexy Hot dx4/dt=ic !", but they weren't visible. Your code said "Success: Already imported:" when I refreshed the browser to restart the importation:

but there is still no way to view these photos in Piwigo--they show up nowhere.

ONLY photos in albums are seen anywhere:

http://66.45.225.82/20pw/piwigo/index.php?/categories

The orphan photos appear nowhere in "recent_pics" either: http://66.45.225.82/20pw/piwigo/index.php?/recent_pics

I then added the photos to a brand new album on Flickr, and I refreshed Flickr2Piwigo, but Flickr2Piwigo failed to create the new album. It still stated, "already imported" for all the photos, skipping them, but again, there is no way to view these photos at my Piwigo install.

Here are the log files for today's install: http://66.45.225.82/20logs/

Hope this helps!

Thanks, Elliot :)

[ElliotMcGucken]

Greetings Sam!

Flickr2Piwigo has been running fairly well for a few hours now! Here are the logfiles for up until this moment: http://66.45.225.82/20logs.2/

It has been skipping many albums. Here are the Piwigo albums: http://66.45.225.82/20pw/piwigo/

And here are the flickr albums: https://www.flickr.com/photos/herosjourneymythology45surf/albums

It seems that many of the albums that are skipped are albums that were created some time after the photos were uploaded to flickr? It seems to do better with photos that are uploaded into albums, and/or at the same time that the albums are created.

And too, it still gives errors:

Here are the most recent logfiles which are up to date: http://66.45.225.82/20logs.2/

I just stopped Flickr2Piwigo for the day.

We're getting close!

Thanks Sam!

Elliot :)

[samwilson]

You can see orphans at /admin.php?page=batch_manager&filter=prefilter-no_album

I've added some more logging. I'm really at a bit of a loss as to what is going wrong! :-(

It should be logging the titles of the albums it's creating, could you check to see if they're what you expect? It doesn't do anything about the creation time of the album, so that's not the problem. It only creates the albums as it encounters the photos in them, so it may not do them in any particular order.

Sorry I'm not very helpful this evening. :-) I'll keep thinking and debugging.

[ElliotMcGucken]

Thanks Sam!

So basically Flickr2Piwigo imports photos in chronological order, as opposed to albums in chronological order? That means that if I uploaded photos a year ago, but just put them in an album created today, the album won't be created by Flickr2Piwigo until the photos are encountered during the chronological import?

Were you able to fix this bug from the previous install:

Here are the log files for today's install: http://66.45.225.82/20logs/

I created a brand new fresh install of everything today, and I'm seeing some of the same errors as yesterday (I think the errors are due to characters such as apostrophes and/or periods in the photo titles). Again, when it gets to the album:

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

It fails to create the album again. Just like yesterday, it creates the thumbnails:

http://66.45.225.82/20pw/piwigo/index.php?/recent_pics/start-480

But when one clicks on a thumbnail, one gets the same error again:

The titles of the photos are the same as the album title:

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

This character string has 251 characters in it. Might that be a problem after addslashes and all?

Are the apostrophes ' in "St. Patty's" and "St. Patrick's Day" problems perhaps? Or are the periods in "St." a problem?

What do you think about this question--"Are the apostrophes ' in "St. Patty's" and "St. Patrick's Day" problems perhaps? Or are the periods in "St." a problem?"

I know you said the ampersand posed a problem in names/titles. What about the ' apostrophes and . periods?

Thanks Sam!

[samwilson]

Yep, it goes in chronological order (of upload date), or at least it goes in whatever order Flickr gives us for the "people.getPhotos" API call. So albums will not be populated in order, but will be by the end.

And no, it's a good thought, but slashes, apostrophes, quotation marks, periods, etc. are all fine in photo and album titles. I've tested with https://www.flickr.com/photos/153362204@N03/39172591000/ with no problems.

Maybe there's some rate-limiting or something going on? :(

[samwilson]

Tags! Maybe it's tags with funny characters. :-)

[mysql error 1064] You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' quotes and \slashes\'' at line 3

SELECT id
  FROM tags
  WHERE name = 'tags 'with' quotes and \slashes\'
;

[samwilson]

Yep, seems to have been one problem at any rate. Could you please test again with the latest patch? :-)

[ElliotMcGucken]

Greetings Sam! Thanks!

I just installed a brand new fresh install of everything, and here is the result: http://66.45.225.82/24pw/piwigo/ http://66.45.225.82/24logs/ (latest log files for today's install)

I didn't see any glaring errors, but when Flickr2Piwigo showed 500 photos imported in the "Processing 500," it hung and stopped importing.

So I refreshed the browser, and it seemed to work fine, skipping all the photos that it had already imported.

But then when it hit "Processing 500" again, it stopped again, so it hangs here at "Processing 500:"

I refreshed the browser one more time, and again, it stopped at "Processing 500."

I just copied the latest log files here: http://66.45.225.82/24logs/

I feel we're getting close!

Thanks so much! Elliot :)

P.S. Have you seen these funny characters in your Flickr2Piwigo install? I've noticed them before too:

[samwilson]

Oops, I was checking "totalPages < currentPage" rather than "currentPage < totalPages"! :) Fixed now. Try again.

I also made the stop button work a bit better (so it gets its own little popup message saying "stopped").

No idea about the funny characters I'm afraid. Do they appear when no plugins are turned on? If so, I reckon you should raise a bug against core Piwigo.

[ElliotMcGucken]

Thanks Sam!

I did a fresh install of everything, and it seems to be working well! http://66.45.225.82/25pw/piwigo/

I did notice just a few errors, but it keeps on going and keeps transferring images and all, despite these errors which seem isolated so far:

Here are the log files after transferring around 2500 photos, and as it is still running: http://66.45.225.82/25logs/

Flickr2Piwigo is still running & transferring--just one issue here:

The Home/Recent Photos pages seems stuck at 2248 photos: http://66.45.225.82/25pw/piwigo/index.php?/recent_pics/start-2235

Even though the photos are continuing to load and it says 2633 under the albums:

(this number under the albums list is growing as I write this, while the 2248 number above in Home/Recent photos is stuck at 2248)

Here are the logs: http://66.45.225.82/25logs/

Thanks!

P.S.: Now and then the counter at the top of the page jumps to 2249 photos, but then it quickly jumps back down to 2248 photos: http://66.45.225.82/25pw/piwigo/index.php?/recent_pics/start-2235

Meanwhile, the number under the albums keeps growing as photos are imported:

[ElliotMcGucken]

Hello Sam!

It seems that the above problem is still centered about the same problematic album from earlier attempts, as Flickr2Piwigo is still unable to create this album: "Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!"

Note that Flickr2Piwigo imports the albums just before and after this problematic album circrled below at my flickr page:

Note how the circled album above is missing here, while the other albums around it are present: http://66.45.225.82/25pw/piwigo/index.php?/categories

And this time, after it hits the album during import, it won't add any more photos to the Home / Recent photos [2248] which is now stuck at 2248, as described above.

Here is the album on flickr: https://www.flickr.com/photos/herosjourneymythology45surf/albums/72157664755226537

Here is one of the photos: https://www.flickr.com/photos/herosjourneymythology45surf/39957959485/in/album-72157664755226537/

Are there any characters in the album title, or photo title, descriptions, or keywords which could cause a bug? Is the album or photo title or description too long? When I run the title through addslashes (https://www.tools4noobs.com/online_php_functions/addslashes/), I get:

Irish Lass St. Patty\'s Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick\'s Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

So are the apostrophes a problem? Does the addslashes make the string too long? Could it be a problem that there are two apostrophes to handle in the above string?

Is the title too long, or does it have too many apostrophes, periods, or exclamation marks? Are the apostrophes the problem somewhere in the code?

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

Is the slash a problem in the keywords in dx4/dt=ic?

Is the album title too long perhaps?

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

This same album seems to keep breaking Flickr2Piwigo, and now it doesn't even look like it creates the thumbnails anymore for the album? So it seems like now the album and all its photos are skipped?

Other than this album, things are looking great! :)

Here are the latest logs: http://66.45.225.82/25logs.2/

Thanks Sam!

Elliot :)

[samwilson]

I think the errors being displayed might be something to do with being rate limited from Flickr's end. I'll add some better checking for that.

The recent photos thing only shows photos posted within a day before the lastest-posted photo. So if you're importing posted dates from Flickr, the recent-photos view should max out at however many you uploaded within a day... but of course, that will vary as the latest-posted date changes (as other photos are uploaded).

I'll look into the album naming a bit more. I think it shouldn't mind the length or funny characters, but will do some better checking.

Thanks so much for helping debug all this!

[biffhero]

I wanted to test what Flickr did when I deleted an album (need to make sure that it does not delete what is in it, even though they tell you that in the help).

1. Uploaded one image.
2. Created an album.
3. Put that image in the new album.
4. Deleted the album. -- At this point, my test with Flickr was finished. So I had a new photo in my Flickr photostream that I needed to import with the import-all branch of Flickr2Piwigo.

When I went to Flickr2Piwigo, the plugin says "No photo in the current set."

Thank you, Rob

[samwilson]

This may be due to the 10 minute cache time. Do you get the same error after either waiting a bit or deleting the _data/flickr_cache directory?

-- Sent from my field telephone with K-9 Mail. Please excuse brevity.

On 26 March 2018 12:44:15 pm AWST, biffhero notifications@github.com wrote:

I wanted to test what Flickr did when I deleted an album (need to make sure that it does not delete what is in it, even though they tell you that in the help).

1. Uploaded one image.
2. Created an album.
3. Put that image in the new album.
4. Deleted the album. -- At this point, my test with Flickr was finished. So I had a new photo in my Flickr photostream that I needed to import with the import-all branch of Flickr2Piwigo.

When I went to Flickr2Piwigo, the plugin says "No photo in the current set."

Thank you, Rob

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/mistic100/Flickr2Piwigo/issues/12#issuecomment-376044818

[ElliotMcGucken]

Thanks Sam!

You write:

I'll look into the album naming a bit more. I think it shouldn't mind the length or funny characters, but will do some better checking.

It is interesting to note that the album title is the same as the photo titles in this case, and that Flickr2Piwigo is both unable to create the album and unable to successfully display the full-size photos.

Album name & photo title:

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

In order to see the thumbnails, do a search on "Irish" here: http://66.45.225.82/25pw/piwigo/index.php?/search/1

This is the page of thumbnails one sees:

Now when one clicks on any one thumbnail, one gets this error:

Hope this helps!

Here are the latest logs again from the most recent import: http://66.45.225.82/25logs.2/

Thanks,

Elliot

P.S. I searched for the word "Irish" in the log file (http://66.45.225.82/25logs.2/), and it does look like the album is being created?

But then there is no such album here: http://66.45.225.82/25pw/piwigo/

I think that the album name is giving Flickr2Piwigo a problem somewhere:

Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!

Perhaps if you created an album and photos with this same title at your flickr, you'd see the error too?

Here is the album on flickr: https://www.flickr.com/photos/herosjourneymythology45surf/albums/72157664755226537

Here is one of the photos: https://www.flickr.com/photos/herosjourneymythology45surf/39957959485/in/album-72157664755226537/

Thanks! Elliot :)

[samwilson]

Ah! Yes, indeed, the apostrophes (or rather, my lack of understanding of various bits of the Piwigo codebase) were to blame. Fixed now, and I've tested with your erroring album title and it's worked.

So, perhaps we're closer now! :-) Give it a test and see how it goes.

[ElliotMcGucken]

Thanks Sam!

I ran a brand new install today, and it now works for this "Irish" gallery too!

http://66.45.225.82/28pw/piwigo/index.php?/category/21

It's working great!

I have seen some random errors, which may be flickr timing out or something, but they seem fairly minor--here are today's logs:

http://66.45.225.82/28pw/piwigo/_data/logs/

And here is what the minor errors look like:

Looking great!

One small request would be to put any photo which is not in an album into an album titled, "Not in an official album."! It's not a huge deal, but it seems it may be simple and useful to people on down the line, as sometimes flickr won't add every photo to an album, even though it makes it look like it did.

Also, if two albums have the same name, can we please make sure it creates both albums? Perhaps it already does this?

Thanks Sam!

I am going to stop Flickr2Piwigo now and restart it from home, and let it run overnight.

Will let you know how it goes.

Thanks for all your work on this! It's looking great.

Best,

Elliot