[FEATURE] PutUploader

[Nitschi]

Is your feature request related to a problem? Please describe.

In our organization, due to privacy requirements, we need to keep our video recordings on-premise. Currently, LiveKit Egress supports uploading files to cloud storage solutions like S3, GCP, and others very conveniently. However, this does not meet our compliance needs, as we need to ensure that all recordings are stored within our own infrastructure without passing through third-party cloud providers.

Describe the solution you'd like

We propose the addition of a PutUploader to the LiveKit Egress service. This uploader would use HTTP PUT requests to upload video recordings directly to our on-premise storage solutions or any HTTP server that we configure. This feature would allow organizations with self-hosted recording setups to easily store them on-premise.

The PutUploader should be configurable through the Egress service's YAML configuration file, allowing users to specify the endpoint URL, authentication credentials, and any other relevant HTTP headers required by the receiving server. Additionally, it would be beneficial to support retries and configurable timeout settings to ensure robustness in network variability scenarios.

Describe alternatives you've considered

As an alternative, we have considered using the existing file output mechanism to first save recordings locally and then manually upload them to our on-premise servers based on a small script that checks for new files on a regular basis. However, this approach introduces additional complexity and delay in our workflow, as it requires a separate process to monitor, upload, and verify the integrity of the recordings post-egress.

Another alternative is to modify the source code of LiveKit Egress ourselves to include this feature. However, we believe that contributing this feature back to the community would be beneficial for many organizations with similar privacy and compliance requirements.

Additional context

We are willing to contribute to the development of this feature and look forward to your feedback on the proposal.

[serhiynovos]

Instead of cloud S3 storage on premise solutions also can be used any s3 compatible storage like minio. Other solution I see it's just mount to egress volume like NAS so recordings can be stored on prem on remote server with no egress code changes. Or event to have sidecar container which will be watching folder changes and upload to remote server

[Nitschi]

@serhiynovos, thank you for your suggestions. While integrating an S3-compatible storage solution like MinIO or utilizing NAS for on-prem storage could work, these approaches would introduce significant changes to our existing infrastructure and operational workflows. Our current ecosystem uses an ASP.NET backend for handling various types of recordings, and we aim to maintain consistency in how video recordings are managed and stored.

Implementing a sidecar container to monitor and upload files would require the development of a new service for handling webhook messages, which increases configuration complexity, CI, maintaning work etc.

The idea of deploying a script that continuously monitors a directory for new files and uploads them might seem like a simpler workaround. However, considering the sophistication of LiveKit's existing upload mechanisms, this solution feels more like a temporary patch rather than a robust, long-term strategy.

Incorporating HTTP PUT (or POST) support directly within LiveKit as an uploader option would offer a consistent and maintenance-friendly solution. It aligns with many existing backend services and simplifies the configuration process, eliminating the need for additional components or significant infrastructural adjustments. This feature would not only benefit our organization but also support other users in the community with similar requirements for privacy, compliance, and on-premises data handling.

As a first proof of concept my colleague has written a standalone uploader that works well, even for bigger file-sizes (if the backend can handle it): goPutUploader

Would integrating something like this be an option?

[serhiynovos]

@Nitschi Yes, it could be a cool feature to have file uploader. Here I see one complexity it's how to deal with upload API as it can have different auth mechanisms and egress should support all of them to cover all organisations needs.

But in any case let's wait for Livekit team answer, they have to know much better 😊

[Nitschi]

My guess would be, that 90% of users would be happy with a simple token/secret-based approach. For our case we could use any of the many possibilites that libraries like ASP.NET supports, so less flexibility is needed on livekits side.

[coonmoo]

Great idea! We also need an uploader which can upload to arbitrary HTTP urls. This would simplify retrieval of recordings in an air-gapped environment a lot.

Regarding authentication I think supporting custom headers in the HTTP request should cover most use-cases and would be a good starting point.

http:
  url: https://my-api.com/
  custom_headers:
    Authorization: "Bearer ..."

http:
  url: https://my-api2.com/
  custom_headers:
    X-Api-Secret: "XXX..."