search

LycheeOrg / Lychee

A great looking and easy-to-use photo-management-system you can run on your server, to manage and share photos.
https://lycheeorg.github.io/
MIT License
3.27k stars 295 forks source link

Uploading many (>100) images failes with error 429 when using livewire #2166

Closed Gendra13 closed 7 months ago

Gendra13 commented 7 months ago

Detailed description of the problem [REQUIRED]

(This bug may be related to #2129)

When I try to upload a lot of images in a row (~100, size 400 MiB), the upload starts fine, but after ~ 60 images I get a lot of failed uploads. The tracelog shows a lot of http 429 errors popping up and in the logs I can find the message

Illuminate\Routing\Middleware\ThrottleRequests::buildException:232 Too Many Attempts.
App\Http\Middleware\AcceptContentType::handle:50 Too Many Attempts.; caused by

However, then problem only exists with livewire enabled, when I switch back to LIVEWIRE_ENABLED=false im my .env then the upload is working flawlessly.

My php settings are 

max_execution_time = 300
upload_max_filesize = 1024M
upload_max_size= 1024M
max_file_uploads = 200

and my nginx is set to 

client_max_body_size 512M;
client_body_timeout 300s;

so those should not be a problem (and those very settings are working with livewire disabled). Also the error occours regardless if use_job_queues is set 1 or 0.

Steps to reproduce the issue

Steps to reproduce the behavior:

  1. Create a new album
  2. Click the upload button
  3. Drag and drop a larger selection of images (>100)
  4. After the first few uploads are working fine, suddenly the uploads start to fail

Screenshots Error message upload

Logfile entries log

Output of the diagnostics [REQUIRED]

Diagnostics
-----------
    Warning: /var/www/html/lychee/public/sym/ has permissions 0755, but should have 2775
    Warning: Dropbox import not working. dropbox_key is empty.
    Info: Latest version of PHP is 8.3
    Warning: Default timezone not properly set; you might experience strange results when importing photos without explicit EXIF timezone
     Foreign key: access_permissions.user_id     → users.id             : CASCADE
     Foreign key: albums.parent_id               → albums.id            : RESTRICT
     Foreign key: albums.id                      → base_albums.id       : RESTRICT
     Foreign key: albums.cover_id                → photos.id            : CASCADE
     Foreign key: base_albums.owner_id           → users.id             : RESTRICT
     Foreign key: jobs_history.owner_id          → users.id             : RESTRICT
     Foreign key: photos.album_id                → albums.id            : RESTRICT
     Foreign key: photos.owner_id                → users.id             : RESTRICT
     Foreign key: size_variants.photo_id         → photos.id            : RESTRICT
     Foreign key: sym_links.size_variant_id      → size_variants.id     : RESTRICT
     Foreign key: tag_albums.id                  → base_albums.id       : RESTRICT

System Information
------------------
    Lychee Version (release):                5.0.2
    DB Version:                              5.0.2

    composer install:                        --no-dev
    APP_ENV:                                 production
    APP_DEBUG:                               false
    APP_URL:                                 set

    System:                                  Linux
    PHP Version:                             8.2.14
    PHP User agent:                          Lychee/4 (https://lycheeorg.github.io/)
    Timezone:                                UTC
    Max uploaded file size:                  1024M
    Max post size:                           1024M
    Livewire chunk size:                     12.00 MB
    Max execution time:                      300
    MySQL Version:                           10.6.12-MariaDB-0ubuntu0.22.04.1

    exec() Available:                        yes
    Imagick Available:                       1
    Imagick Enabled:                         1
    Imagick Version:                         1691
    GD Version:                              2.3.3
    Number of foreign key:                   11 found.

Config Information
------------------
    version:                                 050002
    check_for_updates:                       1
    sorting_photos_col:                      taken_at
    sorting_photos_order:                    ASC
    sorting_albums_col:                      max_taken_at
    sorting_albums_order:                    ASC
    imagick:                                 1
    skip_duplicates:                         1
    small_max_width:                         0
    small_max_height:                        360
    medium_max_width:                        1920
    medium_max_height:                       1080
    lang:                                    en
    image_overlay_type:                      none
    default_license:                         reserved
    compression_quality:                     90
    grants_full_photo_access:                1
    delete_imported:                         0
    mod_frame_enabled:                       0
    mod_frame_refresh:                       30
    thumb_2x:                                1
    small_2x:                                1
    medium_2x:                               1
    landing_page_enable:                     0
    site_owner:                              John Smith
    landing_title:                           John Smith
    landing_subtitle:                        Cats, Dogs & Humans Photography
    sm_facebook_url:                         https://www.facebook.com/JohnSmith
    sm_flickr_url:                           https://www.flickr.com/JohnSmith
    sm_twitter_url:                          https://www.twitter.com/JohnSmith
    sm_instagram_url:                        https://instagram.com/JohnSmith
    sm_youtube_url:                          https://www.youtube.com/JohnSmith
    landing_background:                      dist/cat.webp
    site_title:                              Lychee v5
    footer_show_copyright:                   1
    site_copyright_begin:                    2019
    site_copyright_end:                      2019
    footer_additional_text:                  
    footer_show_social_media:                0
    search_public:                           0
    SL_enable:                               1
    SL_for_admin:                            0
    recent_age:                              1
    grants_download:                         0
    photos_wraparound:                       1
    map_display:                             1
    zip64:                                   1
    map_display_public:                      1
    map_provider:                            OpenStreetMap.de
    force_32bit_ids:                         0
    map_include_subalbums:                   1
    update_check_every_days:                 3
    has_exiftool:                            0
    share_button_visible:                    0
    import_via_symlink:                      0
    has_ffmpeg:                              0
    location_decoding:                       1
    location_decoding_timeout:               30
    location_show:                           1
    location_show_public:                    0
    rss_enable:                              0
    rss_recent_days:                         7
    rss_max_items:                           100
    prefer_available_xmp_metadata:           0
    editor_enabled:                          1
    lossless_optimization:                   0
    swipe_tolerance_x:                       150
    swipe_tolerance_y:                       250
    local_takestamp_video_formats:           .avi|.mov
    log_max_num_line:                        1000
    unlock_password_photos_with_url_param:   0
    nsfw_visible:                            1
    nsfw_blur:                               0
    nsfw_warning:                            0
    nsfw_warning_admin:                      0
    nsfw_banner_override:                    
    map_display_direction:                   1
    album_subtitle_type:                     oldstyle
    upload_processing_limit:                 4
    new_photos_notification:                 0
    legacy_id_redirection:                   1
    zip_deflate_level:                       6
    SA_enabled:                              1
    default_album_protection:                1
    album_decoration:                        none
    album_decoration_orientation:            row
    allow_username_change:                   1
    auto_fix_orientation:                    1
    use_job_queues:                          1
    random_album_id:                         starred
    use_last_modified_date_when_no_exif_date: 0
    ffmpeg_path:                             
    ffprobe_path:                            
    layout:                                  justified
    date_format_photo_thumb:                 M j, Y, g:i:s A e
    date_format_photo_overlay:               M j, Y, g:i:s A e
    date_format_sidebar_uploaded:            M j, Y, g:i:s A e
    date_format_sidebar_taken_at:            M j, Y, g:i:s A e
    date_format_hero_min_max:                F Y
    date_format_hero_created_at:             M Y
    date_format_album_thumb:                 M Y
    upload_chunk_size:                       0
    nsfw_banner_blur_backdrop:               0
    search_pagination_limit:                 1000
    search_minimum_length_required:          4
    photo_layout_justified_row_height:       320
    photo_layout_masonry_column_width:       300
    photo_layout_grid_column_width:          250
    photo_layout_square_column_width:        200
    photo_layout_gap:                        12
    display_thumb_album_overlay:             always
    display_thumb_photo_overlay:             hover

Browser and system

Server: Ubuntu 22.04 LTS with nginx 1.18.0 (no reverse proxy) Client: Firefox 121 (64-bit) on Windows 10

ildyria commented 7 months ago

humm probably hitting the throttle.

ildyria commented 7 months ago

Can you try to change line 69 from config/livewire.php from :

        'middleware' => null,  // Example: 'throttle:5,1'             | Default: 'throttle:60,1'

to 

        'middleware' => 'throttle:10000,1',  // Allow 10000 uploads per user per minute.
Gendra13 commented 7 months ago

Can you try to change line 69 from config/livewire.php from :

      'middleware' => null,  // Example: 'throttle:5,1'             | Default: 'throttle:60,1'

to

      'middleware' => 'throttle:10000,1',  // Allow 10000 uploads per user per minute.

👍 Yes, thank you. That seems to solve the issue. I ran a test with 500 images (2,2 GiB) and everything worked well without an error.

ildyria commented 7 months ago

Fix on its way to master.

verymilan commented 7 months ago

Sadly i have this problem now with 20 photos. It just fails and floooods the browser console with this until page reload:

SCR-20240120-omsy
verymilan commented 7 months ago

Update: I managed to place invalid values for

php_admin_value[post_max_size] 
php_admin_value[upload_max_filesize]

which did not cause php (or lychee for that matter) to complain, but triggered this issue. Sorry about the noise.