Lychee Storage on S3

[danbaron-fishwise]

Is there any way to update lychee storage of photos to S3? If I'm on AWS, the only option is EBS storage?

[d7415]

Not yet, no.

[danbaron-fishwise]

So, running on AWS for a lot of storage is still very expensive. The best option is running a local Linux server or trying to run on a Mac?

Are there image editing tools via imagick? I only see resizing options.

[d7415]

So, running on AWS for a lot of storage is still very expensive. The best option is running a local Linux server or trying to run on a Mac?

That's up to you. Some users host locally, others use VPSs or even shared hosting (Not sure if that's the case with Laravel, but there were certainly some with the original Lychee)

Are there image editing tools via imagick? I only see resizing options.

No. It's just a gallery. If you want to edit photos you should do that before uploading them.

[danbaron-fishwise]

I see some options with Laravel storage configuration to use S3. There is no way to create an alias for /storage that points to S3?

[ildyria]

That would probably need some rewriting but I do believe this should hopefully not be that hard to implement. :)

[kamil4]

Are you thinking of using S3 just as the source ("Import from...") or also to store the generated thumbs/small/medium images? I imagine the latter would be considerably harder...

[d7415]

As storage, so probably thumbs/small/medium, but certainly the originals.

[danbaron-fishwise]

I want all uploaded masters and versions stored on S3. I just need to figure out how to create the alias for /storage

[ildyria]

if you submit a PR we will gladly merge it :)

[mannp]

S3 Compatible would be a blast, so I can use my Minio docker as the backend...

[ildyria]

It is actually pretty close to be implemented... the problem is to provide the URLS from the S3 storage... and sending the files there.

[andrewminion-luminfire]

@ildyria Did this ever get completed?

[ildyria]

If the issue is still open, nope. Sorry. And I don't have the time / fundings available to set a S3 storage up and debug it.

[andrewminion-luminfire]

If the issue is still open, nope. Sorry.

I assumed that was the case. What’s left to finish on it?

[reisfe]

Hi I would also love to have this feature. Hope you can find the time or money to implement it. If there is a fund raising I would be willing to contribute to this feature.

[bmalbusca]

@ildyria @d7415 I want to make a contribution to this feature. We (DSI team) are moving to S3 storage, and I've seen that you've already made some progress. Because I haven't seen any documentation about this, I'd like to ask you for some insight / help to get started.

[ildyria]

Good to read:

• https://laravel.com/docs/8.x/filesystem
• https://lycheeorg.github.io/docs/read-more.html
• https://lycheeorg.github.io/docs/structure.html#the-app-directory_1

I would also suggest to have a look at this file in particular: https://github.com/LycheeOrg/Lychee/blob/master/app/ModelFunctions/PhotoFunctions.php especially around line 248.

(I am currently under heavy work so I cannot give you more info but this should give you a head start).

[ildyria]

Also, one of the nice thing is that once s3 is supported it will be easy to add support for #580 due to the Storage facade.

[jstader]

I'm late to the party but I am interested in helping (I am a novice with PHP but I have experience in various other languages and I think I can help contribute to simple fixes).

I too would like to be able to use laravel's native filesystem objects so I can have certain files reside on s3 and certain objects reside on a local filesystem.

It seems that laravel's documentation that @ildyria posted suggests using the putfile/putfileas method for file storage. From reading the various scripts it appears that other methods are currently being used in Lychee. To what extent do we expect these methods to break what is currently written? I've noticed that the put method seems to have limited default options for visibility (public vs private) and I assume that is some of the issue.

Anyway I'd love to coordinate on this so I'm not duplicating efforts or breaking current functionality. @bmalbusca and @reisfe have you made any progress on this?

[bmalbusca]

Hello again, I just wanted to let you know that I'm back on this feature!

[bmalbusca]

Hi @ildyria I need a quick explain:

I'm starting to add 's3' on /app/Http/Controllers/PhotoController.php, specific at getArchive() method. I saw that I can use almost existent code used to sql but, it's required to change/create a class Archive to be compatible. But I face some doubts that you can explain me very quickly:

/app/Actions/Album/Archive.php#L54 :

$photos_sql = $album->get_photos();  

$this->compress_album($photos_sql, $dir, $dirs, '', $album, $albumID, $zip);

Can you describe what $album->get_photos(); and $photos_sql->get() does? I followed the code to the definition and I found at app/SmartAlbums/PublicAlbum.php that method return an array of Photo objects which are provided as result of a query. Can I overload this method? And inside of compress_album($photos_sql, $dir, $dirs, '', $album, $albumID, $zip); How I can access to the photos filename to get from s3? I want to call $file_content = Storage::disk('s3')->get($file_name); foreach album's photos but I'm getting some difficulties to understand how I can achieve that .

[ildyria]

Because we have multiple kinds of albums (smart albums etc.) the ->get_photos() method returns a sql query. $photos_sql->get() is executing the query and returns a Collection of Photo model (defined in App/Models/Photo. That photo has multiple attributes, but the ones that are of interest to you is $photo->url and possibly $photo->medium maybe also livephoto or something but details.

To dev highly suggest you to:

1. composer install instead of composer install --no-dev (obviously)
2. have the following in your .env:

   APP_ENV=dev
   APP_DEBUG=true
   DEBUGBAR_ENABLED=true

   And add the use Debugbar; on top of the files you are working in. This will allow you to have access to the Debugbar at the bottom of the page even during AJAX query and explore the data during execution.

3. To peak at the data, you basically have two ways: a. dd($variable); which will stop the execution where you are and display the content of the variable. You can see it as a echo $variable; on steroids. b. Debugbar::warning($variable) which will send a message to the debug bar in the front-end. info, notice, error are also available in place of warning.

Hope this helps. If you have more questions, you can also poke us on Gitter: https://gitter.im/LycheeOrg/Lobby (you can login directly with your GitHub account)

[sigulete]

Is this now implemented?

I mounted a S3 storage in my server hoping that Lychee would recognize it as a standard filesystem, but unfortunately doesn't work.

I used rclone to create the mount. It looks like any other mount when you navigate the filesystem, but there may be some underlining libraries that clearly don't work well with this virtual file system.

[nagmat84]

What do you mean by you hoped that Lychee recognized it? Recognize how? What did you expect?

Anyway, Lychee still does not support AWS S3 storage natively through PHP and Laravel means. There are still some pieces missing.

However, from the sound of your post, you mounted the AWS S3 bucket into your filesystem. Mounting a filesystem via the OS will always work, because this is fully transparent to Lychee and Lychee does not know whether the underlying filesystem is a local filesystem (such as btrfs, ext2/3/4, zfs, etc.) or something else. If you use AWS S3 use that way, you only need to change the Lychee UPLOAD_PATH setting (may be the name is slightly different) in the .env file.

But beware that there might be some drawbacks. As Lychee does not know that the filesystem is in a remote location, it might not as efficient as it could be because Lychee (or more precisely the Flysystem subsystem) will read files not in one-pass but do arbitrary seek and read operations.

[d7415]

What do you mean by you hoped that Lychee recognized it? Recognize how? What did you expect?

rclone can mount S3 using FUSE. I'm not sure why that wouldn't work though. This may be worth a new issue as most S3 users would not want to pipe all of their files through their server to the user.

[nagmat84]

rclone can mount S3 using FUSE. I'm not sure why that wouldn't work though.

That should work and if it doesn't then the problem is not Lychee related.

As I said above there are two options how to use AWS S3 buckets:

1. mounting the bucket into the filesystem on the OS layer
2. Lychee native support

Option 1 is completely transparent to Lychee, because reading files from the S3 bucket and transferring files to the S3 bucket happens on the filesystem layer of the OS. Actually, Lychee does not even know that the files are stored in S3.

And that is exactly what you are trying to do using rclone and FUSE. If there is a problem there is nothing what Lychee can do about.

This may be worth a new issue as most S3 users would not want to pipe all of their files through their server to the user.

I am not sure what the pronoun "this" refers to, but I assume you mean option 2, i.e. native support for S3. There is no need for an independent, new issue, because this issue here is that issue.

And as I said above, we made huge progress towards native S3 support as I had recently changed all file handling to streams, but there are still some pieces missing.

Native support would exactly give the advantage you are talking about: Hosted photos would be directly delivered from S3 to the requesting client without going through the server which hosts Lychee.

[sigulete]

Thanks for the reply to both of you.

For the records, I appreciate your effort, I'm new to Lychee and I went through a long journey testing various self-hosting software before ending up with Lychee. This is by far the best solution for my needs, the cleanness and simplicity of the interface is quite appealing, and the toolkit being used is efficient and light in resources. Anyhow, I just wanted to leave my appreciation in the records.

To the point of discussion, yes I'm testing mounting a S3 compatible storage in my server and directing the public/upload in .env to this location. I expected to be transparent, but for some reason, it doesn't work. S3 storage is not a standard storage, so rclone or any other tool to mount a bucket creates a VFS layer on top. This layer is different for each implementation, so I will start testing with s3fs now instead of rclone. The latter, implements POSIX much better, so it may work this time.

Still interested to see when the native implementation for S3 is out. I'm running Lychee in my Linode and the S3 storage is in the same data center where I can set up a private channel between them, so the communication is fast, reducing internet latency to just routing latency in a private network. That's why I'm so interested to make this integration possible.

[efharkin]

I also really like Lychee and would love to see this feature! It seems like the easiest way forward would be to transparently mount and s3 bucket somewhere as @DarhonSoftware suggested. I also haven't managed to get this to work, and the weird way it doesn't work makes me think that the problem could be on Lychee's end.

What I tried

I tried mounting a bucket to public/uploads, replacing all the subdirectories/index files inside, then running Lychee as usual. I've mounted the bucket using goofys with the following options: --file-mode 0660 --dir-mode 0770. goofys doesn't fully implement POSIX, but surely basic reading and writing should be enough for a photo app to work?

What works

Dragging photos onto the Lychee UI uploads them to the S3 bucket. I can tell the photos have uploaded successfully because I can see them in my bucket on the AWS console, ls public/uploads/... shows new files, new photos appear in the Lychee UI (sort of, see below), and I can download the photo through the UI.

What doesn't work

Lychee renders the uploaded photos as black squares. Checking the logs shows a bunch of "route not found" errors like the one below.

App\Http\Controllers\HoneyPotController::throwNotFound:66 The route uploads/thumb2x/38/ca/xyzxyzxyz.jpeg could not be found.

If I navigate directly to mydomain.tld/uploads/thumb2x/38/ca/xyzxyzxyz.jpeg, I get "404 NotFoundHttpException"

System info

    Lychee Version (git):                    master (2d899a1) -- Could not compare.
    DB Version:                              4.9.4

    composer install:                        --no-dev
    APP_ENV:                                 production
    APP_DEBUG:                               false

    System:                                  Linux
    PHP Version:                             8.1.2-1ubuntu2.13
    PHP User agent:                          Lychee/4 (https://lycheeorg.github.io/)
    Timezone:                                UTC
    Max uploaded file size:                  50M
    Max post size:                           100M
    Max execution time:                      200
    MySQL Version:                           10.6.12-MariaDB-0ubuntu0.22.04.1

    exec() Available:                        no
    Imagick Available:                       -
    Imagick Enabled:                         1
    Imagick Version:                         -
    GD Version:                              2.3.0
    Number of foreign key:                   11 found.

[Kovah]

Would also love to see this being supported. I would also work on this if I would knew what needs to be done. Laravel supports S3 out of the box when league/flysystem-aws-s3-v3 is installed. Usually, the S3 storage is available under a different domain and setting the url option in the filesystem config should tell Laravel to build the appropriate URL for the photo.

[ildyria]

I would also work on this if I would knew what needs to be done

That would be great.

Here are some hints of what code needs to be modified / investigated: https://github.com/LycheeOrg/Lychee/blob/master/app/Models/SizeVariant.php#L202 https://github.com/LycheeOrg/Lychee/blob/master/app/Image/FileDeleter.php https://github.com/LycheeOrg/Lychee/blob/master/app/Jobs/ProcessImageJob.php https://github.com/LycheeOrg/Lychee/blob/master/app/Image/SizeVariantDefaultFactory.php https://github.com/LycheeOrg/Lychee/blob/master/app/Actions/Photo/Create.php#L61

And probably also: https://github.com/LycheeOrg/Lychee/blob/master/app/Console/Commands/GenerateThumbs.php https://github.com/LycheeOrg/Lychee/blob/master/app/Console/Commands/Ghostbuster.php https://github.com/LycheeOrg/Lychee/blob/master/app/Console/Commands/VariantFilesize.php

[Kovah]

Oh, that's a lot to get into. Didn't expect file uploads to be that heavily customized. Not sure if I can spare enough time for that. 😕

[ildyria]

Oh, that's a lot to get into. Didn't expect file uploads to be that heavily customized. Not sure if I can spare enough time for that. 😕

yeah, there are some processing done on uploads:

• checking duplicates
• parsing exif data
• generating appropriate sized thumbs

[Kovah]

Another idea to prevent reworking all the photo upload workflow: how about processing all photos locally, upload all the final files and thumbnails to the S3 storage and then delete the local files? Then save the new photo with the information it's stored on S3. If enabled, it's just another step in the processing pipeline.

Edit: this would be way easier to implement than refactoring the whole workflow, reduce maintenance because it's decoupled from the rest, would reduce the S3 API actions to a bare minimum, and keeping the files locally until they are uploaded makes it easier to retry if there are connection issues. A win-win for everybody.

Would really love to get this working, Lychee is great.

[ildyria]

I have no preference of implementation design. :) And I quite like your idea of implementation.

[ildyria]

Done.