High memory usage - Githubissues

1Password / onepassword-operator

The 1Password Connect Kubernetes Operator provides the ability to integrate Kubernetes Secrets with 1Password. The operator also handles autorestarting deployments when 1Password items are updated.

https://developer.1password.com/docs/connect/

MIT License

532 stars 60 forks source link

High memory usage #149

Open robb-j opened 1 year ago

robb-j commented 1 year ago

Your environment

Operator Version: 1.5.7

Connect Server Version: 1.5.7?

Kubernetes Version: v1.23.14

What happened?

The memory usage of my onepassword-connect pod was 1097Mi which seems very high, is there a normal amount of memory this container should stay around?

What did you expect to happen?

I thought the container would need a lot less memory

Steps to reproduce

The container had been running for 38 days, after restarting the container it was using around 30Mi of memory. This is on a really small cluster with only 8 OnePasswordItem resources in it.
The memory seems to go up a few Mi every few minute

Notes & Logs

before restarting

after restarting

my values.yml passed to helm:

operator:
  create: true
  watchNamespace: [r0b-system,games,default,tools,hyem-tech]

mhixon4479 commented 1 year ago

Seeing similar here. The pod itself isn't being actively used although memory seems to leak until it's OOM killed.

katherine-black commented 1 year ago

Its a shame there are not default values included for the resources requests/limits in the helm chart.

gladiatr72 commented 8 months ago

hrm. has this been resolved with newer versions? I set up a goldilocks monitor for it yesterday and am getting this as a resources recommendation:

resources:
  requests:
    cpu: 163m
    memory: 2282M
  limits:
    cpu: 163m
    memory: 2282M

The last couple months: (I recently inherited this cluster)

The variations in the graph are indicative of this sort of status message w/in the dead pod's skeleton:

Message:      The node was low on resource: memory. Container connect-api was using 1576284Ki, which exceeds its request of 0. Container connect-sync was using 831820Ki, which exceeds its request of 0.

[...]

  connect-sync:
    Container ID:
    Image:          1password/connect-sync:1.5.6
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       ContainerStatusUnknown
      Message:      The container could not be located when the pod was terminated
      Exit Code:    137

[...]

JaniszM commented 7 months ago

👍 , same here. My pod reaches 1GB in two days.

And finally:

The node was low on resource: memory. Threshold quantity: 100Mi, available: 85788Ki. Container connect-api was using 1671000Ki, request is 0, has larger consumption of memory. Container connect-sync was using 926732Ki, request is 0, has larger consumption of memory.

gladiatr72 commented 6 months ago

Bueller...

(sigh)

Dear 1password:

Hello. You might remember me from January 4th of this year. (I think I tripped over @robb-j's corpse in the parking lot, but, no matter, I'm here now). This isn't an open-source project in so much as anyone using it is a paying customer or works for a paying customer. If this operator should be considered abandoned, please have the courtesy of letting us know so we can make other arrangements.

Thanks

mhixon4479 commented 6 months ago

Second the motion.

samirahafezi commented 6 months ago

Hey @gladiatr72!

Samira here from 1Password 👋🏻 Thanks for bringing this to our attention. We do keep a close eye on all of our repos and this issue in particular hasn't been forgotten. Memory issues tend to be tricky and we've been working on our end to try to figure this one out. Apologies on not keeping the issue updated, but it has not been forgotten.

To alleviate your concerns about this repo being abandoned, I want to stress that this is not the case. We are actively working on it and in fact, we released a new version last month that addressed a few bugs. Here are the release notes on that particular release.

We'll continue making updates to this repo over time and will keep the issue updated with our progress on this memory issue.

Thanks!

gladiatr72 commented 5 months ago

I'm starting to feel a bit neglected again...

gladiatr72 commented 3 weeks ago

@edif2008 what's the word?

AndyTitu commented 2 weeks ago

Hey @gladiatr72, are you familiar with this GitHub issue? There have been a few developments there:

a couple of memory leaks were identified and addressed in the latest versions of the connect-sync and connect-api images >1.7.0.
another cause for Kubernetes reporting a high memory usage for Connect was related to a particularity of the container_memory_usage_bytes metric that includes memory that is used for "caching" and can be reclaimed by the OS. The assumption is that container_memory_working_set_bytes is harmless as long as container_memory_rss does not increase as well. Applying a memory limit for Connect in k8s seems to force the OS to clear and reclaim the memory used for caching which seems to bound the memory consumption. To quickly validate this hypothesis it is possible to use sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches" on the k8s host.

Let me know if any of the above is helpful in your case!