Use distribution API with custom backend

[crankedguy]

Hi, as also sketched out on Discord it would be great if the distribution API would be usable with custom / self-hosted backend. The API for it looks very nice. Although here I must say that I did not look deeper into the code on how much work this would be and I just save it here for food of thought. I imagine something maybe stripped down in a way that an artifact "upload" is actually a link that you post to your custom solution where the files can be downloaded, whatever that is. And the users can just get that link from the API then in order to download their files.

[ezekg]

I think this could be accomplished by setting artifact.backend to CUSTOM or LINK, with an additional artifacts.url attribute. Currently, the artifact backend defaults to whatever the account is set to, i.e. S3 or R2. One to this downside would be that we would not be able to expire links like with S3 and R2. So once a user has a link, they'd be free to share it and it'd work into perpetuity. That's one of the benefits of using an S3-compatible storage system for this — timed links.

I'll think about it more to make sure there aren't anymore big gotchas with that approach.

[crankedguy]

Well re this functionality this would have to be done outside of keygen then I think. I would not see that as a duty of keygen. The file could well be deleted after a certain time or something like that. But that would have to be implemented/realized in-house. I am in the middle of something and would also need some time to think deeper into the howto.

Edit : Question is what do the "others" have from that link. They might have an updated version, but without a working license...

[ezekg]

Edit : Question is what do the "others" have from that link. They might have an updated version, but without a working license...

Well, yes and no. When you request an artifact from Keygen, the API will validate the license before redirecting to the artifact's location within the storage provider (typically, an S3-compatible provider). The link that the user is redirected to is a presigned URL that expires after 10 minutes [^0]. The presigned URL is unique to that request.

With a custom backend like you've outlined, that link that the user is redirected to never expires and is not unique per-request. So they could theoretically share that redirect URL with anybody and it would bypass Keygen entirely. Where as with the presigned URL, they could only share the link for 10 minutes at maximum.

You could, of course, delete the file, but then you break artifact downloads for real customers as well. Deleting the file after a certain period of time also has the same effect — the real customers attempting to download that artifact no longer can.

[^0]: Configurable via the ttl query parameter.

[crankedguy]

Ah, ok. That's a neat implementation detail, and that won't be easily available with a custom backend. That's true. But the question remains, what are the users that shouldn't get access to it in the first place do with the artifact, without a license. Where is the difference if a real customer uploads the artifact anywhere else, or just torrents it, You know what I mean? Well, it might be easier to have client area on the product website then. But it is everywhere the same, if someone want someone else to have something he/she can give him just the credentials, or send it via mail... Or do I miss something crucial here?

[crankedguy]

I now finally had the time to have a look at R2. As far as I understand their scheme if I have a 50MB app to download for a few 1000 customers that won't cost a single cent ever. Is this real??? Can't believe it but according to this it is https://developers.cloudflare.com/r2/pricing

A question here to KG : As I did not take this into account when I setup my self-hosted KG I saw this is set to S3 as default it seems. At least I have an S3 entry for backend in the db as far as I remember (am not at the PC right now) Do I have to rerun the whole setup procedure to switch? Hope not.

[ezekg]

It's real. We reduced our distribution costs from being in the thousands per month to mere cents by switching from S3 to R2.

To switch, update your account's backend to R2:

a = Account.sole
a.update!(backend: 'R2')

You will need to set up the R2_* environment variables, of course.

After that, all new artifacts will be uploaded to R2.

[crankedguy]

It's real. We reduced our distribution costs from being in the thousands per month to mere cents by switching from S3 to R2.

Crazy. But good...

Yes the environments that's clear, it was just the question about the setup. Ok, then I will change this directly and just add the envs to the systemd environments for the containers.
Thanks for the how-to! 🙏

[ezekg]

I'm going to close this, as a custom file-based distribution backend has too many incompatibilities with how the distribution API currently works. File-based distribution can easily be done outside of Keygen by simply sharing the link to the public files, since there's no way to license-gate the final download links anyways without an S3-compatible CDN.