Closed BoswellB35 closed 1 month ago
Currently seeing the same issues in my customer base. I believe this issue from the controller logs is related (largely based on timing of log event, I haven't dug into the Controller code to verify)
{
"id": "8fZb3SIAQW",
"source": "https://hashicorp.com/boundary/ip-10-11-67-159.ec2.internal/controller",
"specversion": "1.0",
"type": "error",
"data": {
"error": "pagination.List: pagination.list: unknown, unknown: error #0: Status: 500, Kind: \"Internal\", Error: \"output fields not found when building scope proto\"",
"error_fields": {
"Code": 0,
"Msg": "",
"Op": "pagination.List",
"Wrapped": {
"Code": 0,
"Msg": "",
"Op": "pagination.list",
"Wrapped": {}
}
},
"id": "e_j3SAdXQcIm",
"version": "v0.1",
"op": "handlers.ErrorHandler",
"request_info": {
"id": "gtraceid_m74aXX1xkERqrZdmXOv1",
"method": "GET",
"path": "/v1/scopes?filter=(%22%2Fitem%2Ftype%22%20%3D%3D%20%22project%22)&recursive=true&scope_id=global",
"public_id": "at_mQrSM3rcTg",
"client_ip": "10.11.57.158"
},
"info": {
"msg": "internal error returned"
}
},
"datacontentype": "application/cloudevents",
"time": "2024-02-07T16:03:37.400150659Z"
}
Successfully reproduced this issue via the CLI by having the client issue
$ boundary scopes list -recursive
Error from controller when performing list on scopes
Error information:
Kind: Internal
Message: output fields not found when building scope proto
Status: 500
context: Error from controller when performing list on scopes
The issue seems to be tired to permissions in some way. Administrators with ids=*;type=*;actions=*
are not seeing the issue but people with a smaller permissions set are. Currently my testing user has "ids=*;type=*;actions=read,no-op",
on one of four orgs and no permissions on the others.
Hey folks @zellio & @BoswellB35 -- thanks for using boundary & thanks for the updates here. We're investigating this internally and will provide an update once we have one
An interim solution we have found is to grant broad no-op access to all users on all scopes in the global scope
We use ldap groups for all of our users but roughly we have ended up with three roles
admins => "ids=*;type=*;actions=*"
users => [
"type=scope;actions=list",
"ids=*;type=scope;actions=read",
]
everyone else => "ids=*;type=*;actions=no-op"
I'm also seeing this issue with Boundary 0.15.0 with Boundary desktop 2.0.0.
I'm seeing the same 500 error as @zellio has shown above but I'm not able to resolve the issue by giving the suggested permissions.
I'm unable to get beyond the unsupported controller version so tried to give the "ids=*;type=*;actions=no-op"
to u_anon
in the global scope but still see the same result.
@zellio do you have any more information about how you overcame the docker desktop issue?
@william00179 I haven't tested this issue in Docker. My testing and resolution is from a production deploy on AWS ECS. For reference though I had to make sure that all users had no-op on all scopes in the global scope
We're having similar issues with Boundary (OSS) controller version 0.15.2 and Boundary Desktop version 2.0.0-2.0.1. See attached error message.
The no-op
grant did not make any difference. Boundary CLI works without any issues(including the one bundled with the Boundary.app).
Would be great to have some steps to debug this @AdamBouhmad
EDIT: After updating my grant strings to use ids=*
instead of id=*
and adding the type=scope;actions=list,no-op
to u_anon
at the global scope I'm now getting the same error reported by the author of this issue. 😞
We tried everything we could think of to fix this, only thing that worked was dropping the database and recreating everything (we manage everything via Terraform), after that it is working fine
I suggest starting a local boundary in dev mode and try to replicate your environment: same auth method, same roles, similar number of scopes/projects… we did that and everything worked as expected, that’s where we decided to drop the database
I know it’s not a good solution but that’s what we did
So I think this is related to the recursive
parameter in scopesCheck
below
https://github.com/hashicorp/boundary-ui/blob/0902e42cb05ecbab9bd6549eba1f705c17541541/ui/desktop/app/routes/scopes.js#L26
When recursive
is true
, at least one of your orgs must allow unauthenticated scopes list, otherwise the Boundary API will respond with a 500 status code, which makes the check fail and Boundary Desktop show the modal (thinking that it's actually talking to an older version of the API).
We tried everything we could think of to fix this, only thing that worked was dropping the database and recreating everything (we manage everything via Terraform), after that it is working fine
I suggest starting a local boundary in dev mode and try to replicate your environment: same auth method, same roles, similar number of scopes/projects… we did that and everything worked as expected, that’s where we decided to drop the database
I know it’s not a good solution but that’s what we did
Following up on this I tore down my test environment, database and all, and started fresh on 0.15.2. Can confirm that after I was then able to connect to that deployment via Desktop version 2.0.1.
Obviously this would be a less-than-ideal solution for my production environment, even with things under Terraform control. Hoping we can see a fix that doesn't require dropping the database.
Should be fixed by https://github.com/hashicorp/boundary-ui/pull/2199
This issue is still occurring in 0.15.4
$ boundary scopes list -recursive Error from controller when performing list on scopes
Error information: Kind: Internal Message: output fields not found when building scope proto Status: 500 context: Error from controller when performing list on scopes
$ boundary version
Version information: Build Date: 2024-04-09T14:00:40Z Git Revision: 782dfc0dae0df96b5fe0b8c4497a2d54357b4ed8 Version Number: 0.15.4
$Permissions for u_anon:
ID: u_anon
Version: 1
Name: anonymous
Description: The anonymous user matches any request, whether authenticated or not
Authorized Actions:
add-accounts
set-accounts
remove-accounts
no-op
read
update
delete
@AdamBouhmad Do you have any feedback on this issue?
@wichitalineman what version is your Boundary Controller running on? 0.15.4? If so I can check & see if we can reproduce, as this should have been fixed with https://github.com/hashicorp/boundary-ui/pull/2199
@AdamBouhmad I can confirm I'm still seeing this issue on 0.15.4 controller and workers with the latest Boundary Desktop app also.
Let me know what I can provide so you can reproduce the issue.
@AdamBouhmad Yes, running 0.15.4. We upgraded from 0.14.1.
From the controller we see these logs after anyone logs on the Boundary desktop. The same error occurs with user/pass auth or oidc:
Apr 23 15:50:46 globalops-boundary-con-2 boundary[70721]: {"id":"89PVbwsV4m","source":"https://hashicorp.com/boundary/globalops-boundary-con-2/controller","specversion":"1.0","type":"error","data":{"error":"pagination.list: unknown, unknown: error #0: Status: 500, Kind: \"Internal\", Error: \"output fields not found when building scope proto\"","error_fields":{"Code":0,"Msg":"","Op":"pagination.list","Wrapped":{"Status":500,"Inner":{"kind":"Internal","message":"output fields not found when building scope proto"}}},"id":"e_TUrYFStUhZ","version":"v0.1","op":"pagination.list","request_info":{"id":"gtraceid_qOt9RqYpRFATlABeBvi4","method":"GET","path":"/v1/scopes","public_id":"at_7JXj9GadsT","client_ip":"10.91.204.209"}},"datacontentype":"application/cloudevents","time":"2024-04-23T15:50:46.591500449Z"}
Apr 23 15:50:46 globalops-boundary-con-2 boundary[70721]: {"id":"iIL0F2jQox","source":"https://hashicorp.com/boundary/globalops-boundary-con-2/controller","specversion":"1.0","type":"error","data":{"error":"pagination.List: pagination.list: unknown, unknown: error #0: Status: 500, Kind: \"Internal\", Error: \"output fields not found when building scope proto\"","error_fields":{"Code":0,"Msg":"","Op":"pagination.List","Wrapped":{"Code":0,"Msg":"","Op":"pagination.list","Wrapped":{"Status":500,"Inner":{"kind":"Internal","message":"output fields not found when building scope proto"}}}},"id":"e_TUrYFStUhZ","version":"v0.1","op":"pagination.List","request_info":{"id":"gtraceid_qOt9RqYpRFATlABeBvi4","method":"GET","path":"/v1/scopes","public_id":"at_7JXj9GadsT","client_ip":"10.91.204.209"}},"datacontentype":"application/cloudevents","time":"2024-04-23T15:50:46.591675209Z"}
We tested Boundary Desktop 1.7.1 and 2.0.2 and both had the same issue. Please let me know what further info I can provide.
The fix hasn't been released yet, only merged to main
This is no longer happening with 0.16.0.
Thanks for the fix.
Hello,
Sadly, we bumped into this issue on a production upgrade from 0.14.x to 0.15.4. Since Boundary CLI versions 0.13.x to 0.16.x were working ok with 0.15.4, we thought everything is smooth since the Desktop version essentially uses the CLI behind the scene. We had to revert the database schema migration by creating new database from a snapshot. I am glad OP has opened this issue.
If I understand this correctly, controller version 0.16.x fixes this issue. As unfortunately we have many active users using Desktop v1.6.0 (CLI 0.13.0) , we cannot simply update to controller 0.16.x in production. I appreciate if there is a backport for this fix.
Hello @rfc2119,
Sorry for the delay in response here. We have backported the fix here, and it is now available in Boundary 0.15.5. To resolve the issue, you should be able to upgrade to 0.15.5 or higher. Please see the bottom of the known-issues section in 0.15.5 for more details.
For now, since we've resolved this issue, I will mark this ticket as closed.
Thanks everyone!
Describe the bug I've been testing the 0.15.0 release of Boundary, and have had trouble connecting to it via multiple versions of Boundary Desktop.
I initially tried connecting to my 0.15.0 controller using Version 1.7.1 of Boundary Desktop (which uses CLI Version 0.14.2). This produced the following error after entering my Cluster URL:
I then updated to Boundary Desktop Version 2.0.0 (which uses CLI Version 0.15.0). This produced the following error after entering the Cluster URL:
While the first error is not too surprising given a major version change, the second is perplexing as there is no newer version of the controller for me to use.
To Reproduce
Expected behavior Either version of Boundary Desktop should connect to the controller running Boundary version 0.15.0 and display the usual login prompt.
Additional context I am running a Boundary deployment with a single controller and worker in docker containers on Ubuntu instances, both on version 0.15.0. I've not made any configuration changes to my previous controller or worker deployment outside of updating the version, though I did not see anything in the release notes that suggested I would need to. I am running Boundary Desktop on macOS Sonoma Version 14.3, installed via direct download. My current production deployment of Boundary uses version 0.14.2; Desktop 1.7.1 works fine with that, while Desktop 2.0.0 produces the error in the second screenshot, which I would expect in that case.