[Feature] Support for externally hosted content. eg: S3

[stashTX]

I would like to see stash support S3 buckets for content and/or metadata storage.

Perhaps then there could be an option for the client to pull the content from the stash server which would proxy the data from the S3 compatible storage, or there could be an option for the client to request the content directly from S3 if permissions allowed.

This could also be extended to allow the content to point to a hosted m3u file where the client could request it directly allowing for the leveraging off offsite storage and CDN's.

[hamboneZA]

I've got a setup with S3 working well using a local mount with rclone.

[sacredantelope]

In my Stash setup I've noticed the system completely freezes quite often when scrubbing or switching scenes. I think hosting on S3 and using presigned URLs could do a lot to, as @stashTX said, leveraging the offsite storage and CDN. It would also make it a lot easier to spread Stash over a networked setup and not requiring a ton of hard storage on the server.

Is there any interest from the project to potentially receive a PR for this? Is there any knowledge about the complexity of such a feature or known hurdles? @StashAppDev

[DingDongSoLong4]

@sacredantelope Have you tried rclone, as mentioned above? It's very powerful and supports a ton of storage providers, not just S3. Simply mount a remote to a local path and point stash to it, and everything should just work.

Given that rclone exists, if you want this feature in stash itself, you're probably going to have to add it yourself. But even then, since the changes likely won't be trivial, it will introduce some amount of maintenance burden, so it likely wouldn't be merged for that reason anyway.

All that is to say, is there a reason you're not using rclone? Is it deficient in some way? It really is the most sensible solution here, rather than a change to Stash itself.

[stashTX]

rclone does exist and I do use it.

Native support of object storage systems though would be preferred rather than a 3rd party OS level driver/shim. This would allow this feature to work cross platform.

Additionally it would potentially allow the client to stream the video directly off the object store rather than the stash server itself. One could then have a stash server as essentially the metadata provider and user interface to videos/photos stored in other locations. It is an architectural decoupling and perhaps not trivial but does allow a lot of flexibility.

[DingDongSoLong4]

Rclone is quite cross-platform (certainly much more than Stash is), but I see your point.

However, adding support for object stores will most certainly be highly nontrivial. We'd have to implement authentication, caching, probably rate-limiting as well, etc. Essentially everything that Rclone does, we'd have to do ourselves.

The only advantage I can see for native object storage support over e.g. an Rclone mount is exactly what you mention there: to allow streaming directly off the object store rather than through Stash. So I think it would make much more sense, and be much less effort, to rather add something that simply allows you to override the URL used by the scene player, to use a direct link to the object store rather than a link to stream through Stash.

This link overriding should also be possible to do using a plugin, which are now much more powerful and easier to write. If such a plugin gets enough usage and attention, it can eventually be adopted into a feature in core Stash itself as well.

[sacredantelope]

@DingDongSoLong4 @stashTX Thanks for chipping in. I previously did try using rclone mount but to no avail. I have a setup that might differ a little bit from the norm with the following layers:

• Stash
• Docker
• VM
• Proxmox
• Bare metal server

It turned out that there were issues with using an rclone mount on the file system level (VM) and then having that show up properly in a docker volume. I guess it kind of makes sense since there are multiple layers of mounting going on? Either way, it didn't work.

Today I managed to get it to work by using the rclone Docker driver. My docker-compose config in case someone has a similar setup/issues:

volumes:
  my-s3-volume:
    driver: rclone
    driver_opts:
      type: "s3"
      path: "my-bucket-name"
      s3-provider: "my-provider"
      s3-env_auth: "false"
      s3-access_key_id: "my-key (configure from env)"
      s3-secret_access_key: "my-secret (configure from env)"
      s3-endpoint: "my-endpoint"
      allow-other: "true"

This seems to work fine and streaming works as it's supposed to. Maybe in the future I'll try and write a plugin to overlay the CDN URL as @DingDongSoLong4 suggested.

Finally I'd just like to add that I still believe there is a strong argument to be made for Stash generalizing it's file system layer and allowing for generic providers. Mostly for the sake of being able to offload hosting easily and reduce strain on the host machine.