Closed dreemoutloud closed 1 year ago
kOps uses /proc/self/exe
to find the running binary path, which is pretty much what Go uses also for os.Executable.
It seems that Dynatrace intercepts exec calls and instead it runs everything through its agent, which changes the binary path.
That sounds plausible. Does this mean the two programs are therefore simply incompatible? Is there a fix or a workaround?
Also, has this situation changed since v1.22? We did not experience this issue at that level. (Also- thank you)
On 21 Mar 2023, at 06:47, Richard Heasman @.***> wrote:
That sounds plausible. Does this mean the two programs are therefore simply incompatible? Is there a fix or a workaround?
On 21 Mar 2023, at 06:16, Ciprian Hacman @.***> wrote:
kOps uses /proc/self/exe to find the running binary path, which is pretty much what Go uses also for os.Executablehttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcs.opensource.google%2Fgo%2Fgo%2F%2B%2Frefs%2Ftags%2Fgo1.20.2%3Asrc%2Fos%2Fexecutable_procfs.go%3Bl%3D20%3Bdrc%3Dda4687923b8c2d42c23f61fa3db9f4d3ce0c5f54&data=05%7C01%7C%7C7c67b823eefc4e8ae98608db29d3cf3f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638149761913078554%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MWzcgd5%2BQ24IZebdPRDX7Y27dUhzh9Zz9h0bP0urZQs%3D&reserved=0. It seems that Dynatrace intercepts exec calls and instead it runs everything through its agent, which changes the binary path.
— Reply to this email directly, view it on GitHubhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fkops%2Fissues%2F15257%23issuecomment-1477334522&data=05%7C01%7C%7C7c67b823eefc4e8ae98608db29d3cf3f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638149761913078554%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ayxd9ECJ652Bo16FumHo6nxL7gjHKGKrzQ92%2BJIjykM%3D&reserved=0, or unsubscribehttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALBMYULQQKAKP5LV655QEADW5FBTZANCNFSM6AAAAAAWA4EZCU&data=05%7C01%7C%7C7c67b823eefc4e8ae98608db29d3cf3f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638149761913234754%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zwD4Tyi%2FSqdN5WywTp%2Fcdg9nmPVZZkD%2FJinwu0kvJJw%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.***>
In theory, this could be fixed by passing the binary path as arg when installing the kops-configuration.service
. For this to happen, it would have to be agreed by the other maintainers.
As for workarounds, I don't see anything on kOps side. Maybe there's something in Dynatrace that can disable it for specific binaries or services. Can't say that I'm a fan of their approach here.
Good shout. In tandem with this, I have already contacted Dynatrace support - they have highlighted the following: https://www.dynatrace.com/support/help/technology-support/application-software/go/support/go-known-limitations#side-effects At present I think it best we pursuing this solution (configuring a custom exclusion on dynatrace) and will update you as to the results; so no need at present to code a binary path arg. Will keep you in the loop.
Per Office Hours, we don't have a problem with passing this explicitly to the unit file as long as we have the information.
Dynatrace have confirmed this is due to their Static Go application monitoring. https://www.dynatrace.com/support/help/shortlink/go-known-limitations#side-effects We have turned this off and the issue is gone. Thank you for your time anyway.
/kind bug
1. What
kops
version are you running? The commandkops version
, will display this information. Client version: 1.25.3 (git-v1.25.3)2. What Kubernetes version are you running?
kubectl version
will print the version if a cluster is running or provide the Kubernetes version specified as akops
flag.bin/kubectl version --short
Client Version: v1.25.0 Kustomize Version: v4.5.7 Server Version: v1.25.4
3. What cloud provider are you using? AWS
4. What commands did you run? What is the simplest way to reproduce this issue? Created a new instance group using a latest AMI. This AMI happens to have a monitoring solution (Dynatrace) installed on it in /opt, running as a systemd service. When the nodeup.sh is run as part of the ec2 user_data, the EC2 User_data (accessible on the server as /var/lib/cloud/instance/scripts/nodeup.sh): specifies this call:
After this, the created systemd service is called to start nodeup and thus kops. However, it fails.
5. What happened after the commands executed? The startup fails with the following message:
Examining the created system files:
cat /etc/sysconfig/kops-configuration
So the --install-systemd-unit flag is creating files that specify "/opt/dynatrace/oneagent/agent/lib64/oneagentdynamizer" in the unit file rather than "/opt/kops/bin/nodeup".
6. What did you expect to happen? The kops-configuration service will start kubelet and the node will join the cluster; the file /lib/systemd/system/kops-configuration.service should specify ExecStart=/opt/kops/bin/nodeup
7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml
to display your cluster manifest. You may want to remove your cluster name and other sensitive information.8. Please run the commands with most verbose logging by adding the
-v 10
flag. Paste the logs into this report, or in a gist and provide the gist link here.9. Anything else do we need to know? Yes. If you rename the file /opt/dynatrace/oneagent/agent/lib64/oneagentdynamizer to /opt/dynatrace/oneagent/agent/lib64/oneagentdynamizer.hide and rerun the nodeup command, it works... We have run fsck to confirm there are no inode or filesystem corruptions.