🐛 Workspace cannot be updated by TFO to set the workspaceID when a finalizer is added by another controller

[kGarciaSemarchy]

Operator Version, Kind and Kubernetes Version

• Operator version: 2.5.0
• Kind: Workspace
• Kubernetes version: 1.28.9

YAML Manifest File

Output Log

Kubectl Outputs

Question

Hi, In order to automate a process, I have CompositeController (from Metacontroller) which inspect workspaces and call sync and finalize hooks. This controller add a finalizer on the workspace resource. It seems that the resource is updated in the same time that an update from the terraform-cloud-operator to set status with workspaceID because TFO throw an error:

2024-07-18T09:38:33Z    ERROR   Workspace Controller    {"workspace": {"name":"tenant-3bf542be-infra","namespace":"test-ci-tenants"}, "msg": "update status with workspace ID", "error": "Operation cannot be fulfilled on workspaces.app.terraform.io \"tenant-3bf542be-infra\": the object has been modified; please apply your changes to the latest version and try again"} 
2024-07-18T09:38:33Z    ERROR   Workspace Controller    {"workspace": {"name":"tenant-3bf542be-infra","namespace":"test-ci-tenants"}, "msg": "reconcile workspace", "error": "Operation cannot be fulfilled on workspaces.app.terraform.io \"tenant-3bf542be-infra\": the object has been modified; please apply your changes to the latest version and try again"}

It seems that there are no retry about the update on TFO. Is it a bug on TFO implementation or something is wrong on my side ?

References

https://metacontroller.github.io/metacontroller/api/compositecontroller.html

Community Note

• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

[arybolovlev]

Hi @kGarciaSemarchy,

Could you please provide us with more details here?

Thank you.

[kGarciaSemarchy]

Hi @arybolovlev,

Sorry, my question was hidden. Comment updated. Don't hesitate if I can help.

Thank you

[arybolovlev]

That happens because the workspace object gets modified between the time when the workspace controller reads it from Kubernetes API and finishes reconciliation. Since you mention finalizers, I guess that only metadata.resourceVersion gets updated.

Does that happen only for newly created workspaces or do those that exist also fail to update? I can assume that only new workspace objects are affected.

Thanks!

[kGarciaSemarchy]

You right, it's the case only for the newly created workspaces.

[arybolovlev]

Thank you, @kGarciaSemarchy. I was able to reproduce this issue. It seems like we can use the Patch() method instead of the Update() method when updating the status. However, it will take some time to implement this since there is another side effect of having another finalizer attached to the object that we need to take into account.

We expect version 2.6.0 to be released at the end of July and then we can release 2.6.1 which will include a fix for this issue.

[kGarciaSemarchy]

Thanks for the feedback and your time. We will wait the release 2.6.1 to test again !

[arybolovlev]

Hi @kGarciaSemarchy,

I plan to cut a 2.6.1 release next week with a fix for this issue. Please keep an eye on notifications. I will let you know if this changes.

Thanks.