It looks like Crossplane eagerly creates new clients for each operation. For this Kafka provider, this is problematic because each client keeps connections open without ever closing them. Over time this accumulates and the total number of sockets and CPU utilization grows until something interesting happens.
My changes utilize the Disconnector interface (which was unavailable on the existing crossplane-runtime library version) in order to actually close the client.
I have one concern in the implementation which I will comment on in the PR changes.
[x] Run make reviewable test to ensure this PR is ready for review.
How has this code been tested
I've ran the unit tests successfully, and I have built the image and deployed it into our testing infrastructure. I've verified that the general topic creation/configuration logic works. I haven't explicitly tested ACLs or any deletion, but I haven't changed any logic there, so I don't suspect there should be any issues.
This graph shows the number of goroutines for version v0.2.0 and my version of this provider side-by-side. The new version is flat as the clients are actually getting cleaned up.
In order to update dependencies, I went through step 1 of this doc, but not the others because I don't really know what it is for and it didn't seem necessary for my immediate goal. If it is preferred that I get through those steps to, I can take another look.
Description of your changes
It looks like Crossplane eagerly creates new clients for each operation. For this Kafka provider, this is problematic because each client keeps connections open without ever closing them. Over time this accumulates and the total number of sockets and CPU utilization grows until something interesting happens.
My changes utilize the
Disconnectorinterface (which was unavailable on the existingcrossplane-runtimelibrary version) in order to actually close the client.I have one concern in the implementation which I will comment on in the PR changes.
Fixes #49
I have:
make reviewable testto ensure this PR is ready for review.How has this code been tested
I've ran the unit tests successfully, and I have built the image and deployed it into our testing infrastructure. I've verified that the general topic creation/configuration logic works. I haven't explicitly tested ACLs or any deletion, but I haven't changed any logic there, so I don't suspect there should be any issues.
This graph shows the number of goroutines for version v0.2.0 and my version of this provider side-by-side. The new version is flat as the clients are actually getting cleaned up.
In order to update dependencies, I went through step 1 of this doc, but not the others because I don't really know what it is for and it didn't seem necessary for my immediate goal. If it is preferred that I get through those steps to, I can take another look.