Import problem when command line import stopped and restarted

whcrg commented 2 years ago

Detailed description of the problem [REQUIRED]

Importing from command line with symlinks gives too vague error message, and situation is not handled well (in my opinion), when there is a problem, for example left overs from previous import process that has been killed mid process because of reboot or other various reasons.

Steps to reproduce the issue

Steps to reproduce the behavior:

Start import of pic folder (nice -n 19 php artisan lychee:sync /storage/NAS/photo/2021/ -vvvv --import_via_symlink=1 --skip_duplicates=1)
stop import process (crtl+C for example)
Start import again to continue the process.

problem: Could not import file, that seems to happen to the one that was being processed when the previous import was terminated.

Command line tool output: /storage/NAS/photo/2021/: 67% /storage/NAS/photo/2021/IMG_20210912_203350.jpg: Skipped duplicate /storage/NAS/photo/2021/IMG_20210912_203408.jpg: Skipped duplicate /storage/NAS/photo/2021/: 67% /storage/NAS/photo/2021/IMG_20210914_085702.jpg: Could not import file /storage/NAS/photo/2021/: 68% /storage/NAS/photo/2021/: 68% /storage/NAS/photo/2021/: 68%

Log: 2022-03-03 11:06:03 -- notice -- App\Actions\Photo\Strategies\StrategyDuplicate::storeFile -- 31 -- Nothing to store, image is a duplicate 2022-03-03 11:06:04 -- notice -- App\Actions\Photo\Strategies\StrategyDuplicate::hydrate -- 79 -- Skipped upload of existing photo because skipDuplicates is activated 2022-03-03 11:06:04 -- error -- App\Actions\Import\Exec::do -- 256 -- Could not import file (/storage/NAS/photo/2021/IMG_20210914_085702.jpg) 2022-03-03 11:06:18 -- notice -- App\Actions\Photo\Extensions\ImageEditing::createThumb -- 165 -- Photo URL is e2a91dad86a3825c5b3af74186dbf97b.jpg 2022-03-03 11:06:28 -- notice -- App\Image\ImagickHandler::crop -- 161 -- Saving thumb to /storage/www/Lychee/public/uploads/thumb/e2a91dad86a3825c5b3af74186dbf97b.jpeg 2022-03-03 11:07:02 -- notice -- App\Image\ImagickHandler::crop -- 161 -- Saving thumb to /storage/www/Lychee/public/uploads/thumb/e2a91dad86a3825c5b3af74186dbf97b@2x.jpeg

This seems to because a conflicting link (and maybe an entry to database, how to check?) was already in place, but database-entry is not complete(?) so it is not recognised as a duplicate(?). Photo is not shown in the gallery. Link to photo is in place in Lychee/public/uploads/big/

What I expected would happen: Lychee either just deletes the offending link/file and restarts that ones import, or at least gives more verbose error message about what is wrong so it can be corrected manually (how?) If latter, there should be some --force parameter to delete offending/blocking stuff out of the way.

Output of the diagnostics [REQUIRED]

(Settings => Diagnostics or https://example.com/Diagnostics or php artisan lychee:diagnostics)

Diagnostics
-----------
Warning: Dropbox import not working. dropbox_key is empty.
Warning: user_agent for PHP is not set. You may experience problems when importing images via URL.
Warning: Using 32 bit PHP, recommended upgrade to 64 bit
Error: PHP zip extension not activated

System Information
------------------
Lychee Version (release):        4.4.0
DB Version:                      4.4.0

composer install:                --no-dev
APP_ENV:                         production
APP_DEBUG:                       true

System:                          Linux
PHP Version:                     8.1
PHP User agent:                  
Max uploaded file size:          500M
Max post size:                   500M
Max execution time:              0
MySQL Version:                   10.5.12-MariaDB-0+deb11u1

Imagick:                         1
Imagick Active:                  1
Imagick Version:                 1691
GD Version:                      2.3.0

Config Information
------------------
version:                         040400
check_for_updates:               0
sorting_Photos_col:              taken_at
sorting_Photos_order:            ASC
sorting_Albums_col:              max_taken_at
sorting_Albums_order:            ASC
imagick:                         1
skip_duplicates:                 1
small_max_width:                 0
small_max_height:                360
medium_max_width:                1920
medium_max_height:               1080
lang:                            en
layout:                          1
image_overlay_type:              exif
default_license:                 none
compression_quality:             90
full_photo:                      1
delete_imported:                 0
Mod_Frame:                       1
Mod_Frame_refresh:               30
thumb_2x:                        1
small_2x:                        1
medium_2x:                       1
landing_page_enable:             0
landing_owner:                   JT
landing_title:                   JT
landing_subtitle:                
landing_facebook:                
landing_flickr:                  
landing_twitter:                 
landing_instagram:               
landing_youtube:                 
landing_background:              
site_title:                      Lychee - moerkoe
site_copyright_enable:           0
site_copyright_begin:            2019
site_copyright_end:              2019
additional_footer_text:          
display_social_in_gallery:       0
public_search:                   0
SL_enable:                       0
SL_for_admin:                    0
public_recent:                   0
recent_age:                      1
public_starred:                  0
downloadable:                    0
photos_wraparound:               1
map_display:                     1
zip64:                           1
map_display_public:              0
map_provider:                    Wikimedia
force_32bit_ids:                 0
map_include_subalbums:           1
update_check_every_days:         3
has_exiftool:                    1
share_button_visible:            1
import_via_symlink:              1
has_ffmpeg:                      1
location_decoding:               0
location_decoding_timeout:       30
location_show:                   1
location_show_public:            0
rss_enable:                      0
rss_recent_days:                 7
rss_max_items:                   100
prefer_available_xmp_metadata:   0
editor_enabled:                  1
lossless_optimization:           0
swipe_tolerance_x:               150
swipe_tolerance_y:               250
local_takestamp_video_formats:   .avi|.mov
log_max_num_line:                1000
unlock_password_photos_with_url_param: 0
nsfw_visible:                    1
nsfw_blur:                       0
nsfw_warning:                    0
nsfw_warning_admin:              0
map_display_direction:           1
album_subtitle_type:             oldstyle
upload_processing_limit:         4
public_photos_hidden:            1
new_photos_notification:         0

Browser and system

Debian stable, current stable version of Lychee (both freshly installed 2 days ago). Browser: current chrome on mac os.

kamil4 commented 2 years ago

In general I agree that the diagnostics you are currently getting is substandard and we should definitely see what we can do to improve it. FYI, there is a pending refactoring patch (#1217) that should improve error notifications in general though I don't know if it would help in this particular case...

As to solving your specific problem:

First try to delete the symlink of the photo causing problems and see if that by itself helps during the next import.

If it doesn't, then, as you say, it's probably the database that has a corrupted record. You'll need to remove that record manually.

Load the DB console by typing php artisan db in the top-level Lychee directory
Identify the existing record that's causing problems by typing:
```
select * from photos where title='IMG_20210914_085702'
```
I'm assuming this will return a record. You can delete it as follows:
```
delete from photos where title='IMG_20210914_085702'
```

whcrg commented 2 years ago

php artisan lychee:ghostbuster seemed to solve the problems without more manual intervention. Similar action in case of problem should be available for imports (for example --ghostbust=1 switch or something)

d7415 commented 2 years ago

Similar action in case of problem should be available for imports (for example --ghostbust=1 switch or something)

Rather than add a flag, if you're already running the CLI you may as well just run the command.

whcrg commented 2 years ago

Horribly inelegant to need to manually run this (I am trying to automate system with tens of thousands of photos, new ones arriving several times a day) and even more inelegant to hardcode it being run twice a day or something. Needs to be possible to automate recovery of such thing.

Other thing I noticed is that there is no way to create new album from command line, as only upload by ID and not album name currently possible. Autocreation of albums based on folder structure is not possible as it takes around a week to run though the photo main storage folder and stuff need to be updated semi real time --> needs to be able to point more exactly which photos to process, where to put them and be able to recover from minor problems.

Unfortunately my time and php-skills are not quite enough to write a patch for these myself :/

ildyria commented 2 years ago

@whcrg https://github.com/Chostakovitch/pychee this might be of interest to you.

d7415 commented 2 years ago

Horribly inelegant to need to manually run this

If you want to run this without human intervention, you may as well run the ghostbuster script automatically. I see no benefit in Adding a command line argument to make Lychee do it for you. It's a whole extra line in your script.

kamil4 commented 2 years ago

Other thing I noticed is that there is no way to create new album from command line, as only upload by ID and not album name currently possible.

Album names don't need to be unique -- that's why we only support ID as an argument.

As to creating a new album from command line, you can create an empty directory in the file system and import it -- that should create a new album. Though it wouldn't be easy to script it to then import into that new directory (if that's what you are trying to do) as again you won't know the ID.

In general, it's true that our commandline interface is limited, but that's simply because our focus is on a web interface. I wonder though if it would be possible to write a CLI client that support most (or all) of Lychee API? Maybe with the FUSE client in the works that's not needed...

LycheeOrg / Lychee