[BUDI-6549] https://cdn.budi.live/app.../attachments/... links from `attachment.url` are not reliable anymore - gives "Missing Key-Pair-Id query parameter" or "Access denied" errors

[sergey-vin]

Hosting

• Cloud

Describe the bug If we upload files via "Attachment" fields, and use URLs of those files in other systems - the URLs will expire over time, and start producing the following errors:

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
<Code>AccessDenied</Code>
<Message>Access denied</Message>
</Error>

If we remove "Expires" and "Signature" and "Key-Pair-Id" parts of the URL, the following error shows up:

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
<Code>MissingKey</Code>
<Message>Missing Key-Pair-Id query parameter or cookie value</Message>
</Error>

To Reproduce Steps to reproduce the behavior:

1. In "data" - create a BB table "pictures", with 1 "Attachment" field
2. In "design" - create an auto screen for this table
3. Also in "design" - add an extra action for the "Save" button - to store attachment.url in some other place (other table, etc.), publish. The URL is like this: https://cdn.budi.live/app.../attachments/......
4. In "published app" - try uploading files (PDF, PNG, etc.) saving each file as a record in "pictures" table
5. See the picture links in that other table where you stored them by automation - all files are accessible fine
6. ... wait for ~1-2 hrs...
7. See the picture links in that other table where you stored them by automation - none of the files are accessible via URLs anymore, they are "expired".

Additional context

Everything used to work before, and now - not anymore.

Let's try to see who broke it.

An observation: old links stored in the "other table" contained only domain, path and filename and that's it. But new links have some "key-pair", "expiration", "signature" and other parts.

If now I try to open the links saved long time ago, here's what I'd get:

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
<Code>MissingKey</Code>
<Message>Missing Key-Pair-Id query parameter or cookie value</Message>
</Error>

I assume that there was some change related to enhanced security for uploaded content, and:

1. it broke compatibility with old links
2. links became not permanent because the "Expires" part was added

This means that the old schema won't work for us anymore - and we'd either remove this extra security somehow or re-save all assets periodically to update the "Expires" flag.

Expected behavior Regardless of who saves the attachments - they should all have the working URLs, to be later used somewhere else.

If there is a setting which kills that extra security added to the bucket - give a hint please, on how to enable it.

Also, will moving to "self hosted" solution help with that somehow? If we host BB, we'll also host the storage for uploaded content - can we turn the extra security OFF there?

---

Old bug description (for history)

~Describe the bug~ ~There are several users uploading files via "Attachment" fields. If I upload it - everything works, but if other user do - then the files are visible in admin budibase app, but the links to uploaded files can't be used anywhere else - they show~

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
<Code>AccessDenied</Code>
<Message>Access denied</Message>
</Error>

~To Reproduce~ ~Steps to reproduce the behavior:~ ~1. In "data" - create a BB table "pictures", with 1 "Attachment" field~ ~2. In "design" - create an auto screen for this table~ ~5. Also in "design" - add an extra action for the "Save" button - to store attachment.url in some other place (other table, etc.), publish. The URL is like this: https://cdn.budi.live/app.../attachments/......~ ~6. In "published app" - try uploading files (PDF, PNG, etc.) by different users, saving each file as a record in "pictures" table~ ~7. See the picture links in that other table where you stored them by automation - only some of them will be accessible, other will show the error~ ~8. What's more, if the user for whom the links are saved correctly opens in the "published app" the records which were wrongly saved before - they will be saved correctly this time. So it seems that it depends on the user who creates/updates the images via the app.~

_BUDI-6549

_BUDI-6563

[shogunpurple]

@sergey-vin what's your tenantId? You haven't included it in the issue.

[sergey-vin]

Hi @shogunpurple . I don't think tenantId is important, since I ran more tests and came up with the fact that it's Expiration that kills everything. I'll update the ticket.

My requests stays the same - is it possible to remove Expiration somehow? We are using images uploaded via Budibase in other places, and now - we can't do it anymore. The URLs expire over time.

[sergey-vin]

hi @shogunpurple , so are you taking this to work? I see that you've modified the ticket title. Thanks.

[shogunpurple]

@sergey-vin no, I was just tidying up the title after an automated tool tagged it twice. We are debugging the issue and finding a way to potentially enable this use case. Will keep you updated

[sergey-vin]

Thanks. If possible, please provide some ETA.

Also, given that this is a new way to represent URLs -

https://cdn.budi.live/app_.../attachments/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.png?Expires=1676373723&Policy=xxx&Signature=xxx&Key-Pair-Id=xxx

instead of

https://cdn.budi.live/app_.../attachments/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.png

• it means that it was recently added as one of your features. Maybe you can just provide a way to disable this feature?

Thanks.

[sergey-vin]

any news, @shogunpurple ?

[shogunpurple]

@sergey-vin as mentioned we will keep you updated as we work through it - it’s a bit of an edge case given the attachments are stored in another table, not being fetched through the API and the security feature is there to prevent totally public, unrestricted access of attachments (which was the case before)

[sergey-vin]

But what if that's what we want - to have public access to those assets? There at least should be an option public vs protected.

[fck1tall]

Any answer here please? I was using budibase too. And i'm quite interested in this case as well!

Any suggestions to solve it from your team? No Any explanation what Budibase suggests in exchange of broken functionality? No Any explanation how exactly Budibase provides ability to use public available CDN for now? No Rollback this feature until you're be able to provide a decent answer? No

Just totally broken for almost a month. Now we're trying to move from budibase to another counterparty because of it. What a shame.

[aptkingston]

Any answer here please? I was using budibase too. And i'm quite interested in this case as well!

Any suggestions to solve it from your team? No Any explanation what Budibase suggests in exchange of broken functionality? No Any explanation how exactly Budibase provides ability to use public available CDN for now? No Rollback this feature until you're be able to provide a decent answer? No

Just totally broken for almost a month. Now we're trying to move from budibase to another counterparty because of it. What a shame.

@fck1tall firstly I'm sorry to hear that this change has caused problems for you. This was a very important change to improve the security of all attachments, as it ensures that only you can access your attachments, rather than having globally public assets simply obscured by UUIDs.

When you retrieve a row from the Budibase API which includes an attachment, it will now automatically include an ephemeral signed link to access attachment fields. Attachments continue to work exactly like they always have, with the exception of links to the actual files themselves are no longer permanent.

Budibase is not intended to be used as an object store for other applications - attachment fields are intended to be fetched using Budibase APIs. If you're uploading attachments inside Budibase DB rows then you need to use the Budibase DB API to access those attachments again, allowing us to properly apply role based access control.

If you want to use Budibase as a public object store for other applications (and are self hosted) then you can probably leverage our S3 connector and connect it to our bundled Minio instance, which I imagine will provide similar functionality to how our previous insecure attachment system worked.

Our main concern is that Budibase apps have secure access to attachments, and that is now the case with this change.

If you want provide more details on how you're storing and using Budibase attachment URLs then it would help us as we decide if additional features are needed for attachments. I appreciate you're frustrated, but you've added no additional context and simply complained at this important security change.

[sergey-vin]

Hi @aptkingston, I share the problems that @fck1tall brought up here. And I'm also searching for alternatives now because of this change you made. If there was a way to roll back the "privacy" change somehow, that would be great, for example:

• ~Downgrade to the earlier version of the app, before the security was improved~. No, it doesn't work. The change is in BB core, it's not in the app. Downgrade doesn't help.
• ~Use s3 instead of Attachments~. No, it doesn't work. Attachments provide multiple files upload, upload and re-upload, preview, and many other things which S3 component doesn't have. S3 only allows to show 1 particular file uploaded to S3. If S3 component provided exactly the same functionality as Attachments - that would have been possible, but unfortunately they are not equal.
• ~Start using automation to re-save all Attachment links to external table every 30min-1hr~. Doesn't work as well, because automation is not great with all of the fields, e.g. "Boolean" or "Relations". It breaks the consistency, rewrites content, etc. Also, it works wrongly or doesn't work well with filters in the first place, which means that not all Attachments will be processed. Example of struggling with Automation:
  • https://github.com/Budibase/budibase/issues/9762
  • https://github.com/Budibase/budibase/issues/9764
  • other issues which I didn't file, but struggled with...
• ~Move to self host, to the old docker image, before you introduced extended security~. Also doesn't work, because of 2 things:
  1. You have changed the export format (from .txt (json) to .tar.gz) so that the imports are incompatible between versions now.
  2. Import doesn't always imports all pictures. And it's random - sometimes it does, sometimes it doesn't. Even looking at db.txt from .tar.gz doesn't help finding a difference between the dump that works, and the one that doesn't. Here are related issues:
     • https://github.com/Budibase/budibase/issues/9739
     • https://github.com/Budibase/budibase/issues/9738
     • Not to mention that the "we do not delete anything" concept of Couch DB makes dump files very hard to analyze.
• The last option that I'm expecting from you - is to create a switch "private" or "public" which we can chose in the Attachment component. That would be the cure of this problem and won't force us to search for alternatives and deal with those other bugs mentioned above.

[shogunpurple]

@sergey-vin @fck1tall just want to clear up a few things here.

@fck1tall maybe you are trying to live up to your github username, but I would imagine you already know that asking for things in that way won't give you the outcome you want. I'm sorry that you are frustrated and you have had problems with the change. Rollback this feature until you're be able to provide a decent answer for example is clearly something that we (or any other company) are not going to do.

The security and role based access control of potentially sensitive documents is more important than allowing budibase to be used as a completely free, fully managed and hosted object store in other apps. Being able to host all attachments publicly in our hosted cloud S3 bucket, with no limits, was always temporary - and we will be bringing in storage and upload limits on our cloud plans for file uploads. However - much better attachment/media features will come along with that. Threatening to switch to alternatives to get something shipped faster is unlikely to impact decision making in open source repositories either. Secured and non-public attachments that must be accessed through authenticated APIs doesn't fit your use case, and that's okay. It does however fit the core use cases of attachments in budibase, or we would not have made the change in the first place.

@sergey-vin has done some (appreciated) research around this issue, and I believe has laid it out quite well. There's 2 core issues here:

• You cannot currently just self host and roll back to the previous version of budibase due to the bugs with exports. These are actively being looked at, and will be resolved in the coming weeks. You can keep up with their status in both of the issues posted.
• Allowing public attachments to be "switched on" - or "removing" the security feature. This is not just about removing security, nor is it that simple; it's about allowing budibase to be used as a public object store for other apps. This is why I cannot give an ETA. It's actually a fairly large piece of work, not a core use case of budibase, and therefore not a priority.

To clarify - we will not provide ETAs on large, non-core functionality like this unless it makes sense.

In the meantime, here are some next steps in order of effort (least to most):

• You can use the budibase API to get the latest ephemeral URLs to access images, as the attachments feature is intended to be used - for private, protected assets and internal applications. You won't be able to use the old functionality for now in cloud.
• Wait until the export bugs above are resolved (soon) - self host, roll back your budibase version to the version without this security and use budibase as your own self-hosted public object store where all attachments are public and never expire. Bear in mind you will need to expose your budibase instance to the internet here, and probably provide HTTPS on top using a reverse proxy or similar, and of course configure your own CDN and pay for your own infra/object storage. Guides on how to do all of these things are available at https://docs.budibase.com/docs
• Write plugins/datasources that allow you to upload, download and expose public files https://docs.budibase.com/docs/custom-plugin
• Fork the budibase repo and make changes to support your use case.

Again I apologise for your issues, but I believe there's not much more to be said here, and if the conversation is not productive I think it would be best to lock to collaborators.

Thank you.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity.

[jasonford1]

I know this comment is closed, but just chiming in that this use case was vital to my app (sales rep uploads quote and then the quote is shared with customer). I'll migrate to S3 as best I can, but the stock budibase attachment module appeared perfect only to have customers reply "access denied" and then find this thread.