Open hvt opened 3 years ago
FabianKramm
commented
3 years ago @hvt thanks for creating this issue! Mhh thats odd, do you have any large directories that might be excluded only on client side? Usually this high memory consumption can only be found in pods that have a lot of files or directories present that need to be scanned and compared initially, but even then over 700MB seems really extreme.
hvt
commented
3 years ago Hey @FabianKramm , besides --verbose-sync, is there any way to show all local (and perhaps also remote) files the devspacehelper sync considers to sync or not?
Like I said, in this particular project there are no large directories that I know of and can see quickly (running find . locally)...
FabianKramm
commented
3 years ago @hvt you can also use --debug which will give you more information what the sync is currently doing, which might help you finding out what causes this issue
hvt
commented
2 years ago @FabianKramm here were go:
$ devspace dev --debug
...
19:26:07 [0:ports] Port-Forwarding: Waiting for containers to start...
19:26:07 [0:sync] Waiting for containers to start...
19:26:20 [0:sync] Starting sync...
19:26:20 [0:sync] Trying to download devspacehelper into pod hvt/crm-data-main-v1-7ff88b5c75-g98g9
19:26:20 [0:sync] Warning: Couldn't download devspacehelper in container, error: stdout, stderr:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (7) Failed to connect to github.com port 443 after 2 ms: Connection refused
: command terminated with exit code 7
19:26:20 [0:sync] Trying to inject devspacehelper from local machine
19:26:21 [0:ports] Port forwarding started on 60635:8080 (hvt/crm-data-main-v1-7ff88b5c75-g98g9)
19:26:23 [0:sync] Start syncing
19:26:23 [0:sync] Sync started on /home/hvt/dev/crm-data-main-v1/container/server <-> /server (Pod: hvt/crm-data-main-v1-7ff88b5c75-g98g9)
19:26:23 [0:sync] Waiting for initial sync to complete
19:26:23 [0:sync] Initial Sync - Retrieve Initial State
19:26:23 [0:sync] Downstream - Start collecting changes
19:26:24 [0:sync] Helper - Use inotify as watching method in container
19:26:24 [0:sync] Downstream - Done collecting changes
19:26:24 [0:sync] Initial Sync - Done Retrieving Initial State
19:26:24 [0:sync] Initial Sync - Calculate Delta from Remote State
19:26:24 [0:sync] Initial Sync - Done Calculating Delta (Download: 0, Upload: 0)
19:26:24 [0:sync] Downstream - Initial sync completed
19:26:24 [0:sync] Upstream - Initial sync completed
19:26:24 [0:sync] Initial sync took: 702.3774ms
19:26:24 [info] Starting log streaming
...Looking at memory usage again:
$ devspace enter
# selecting the correct container
[info] Opening shell to pod:container crm-data-main-v1-7ff88b5c75-g98g9:crm-data-main-v1-server
/server $ ps axf | grep devspace
PID USER TIME COMMAND
79 www-data 0:00 /tmp/devspacehelper sync upstream --override-permissions --exclude .gitignore --exclude .php-cs-fixer.cache --ex
80 www-data 0:03 /tmp/devspacehelper sync downstream --exclude .gitignore --exclude .php-cs-fixer.cache --exclude .phpunit.result
/server $ cat /proc/79/status
Name: devspacehelper
Umask: 0022
...
VmPeak: 720440 kB
VmSize: 720440 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 18444 kB
VmRSS: 18444 kB
...So the initial sync is nihil, yet the maximum amount of memory used by devspacehelper sync is around 720MB. And that is only the upstream one. Downstream used around 723MB at its peak.
Currently using:
devspace version 5.18.3Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.15", GitCommit:"58178e7f7aab455bc8de88d3bdd314b64141e7ee", GitTreeState:"clean", BuildDate:"2021-09-15T19:23:02Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.15", GitCommit:"58178e7f7aab455bc8de88d3bdd314b64141e7ee", GitTreeState:"clean", BuildDate:"2021-09-15T19:18:00Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
RebliNk17
commented
10 months ago This is happening for me too, i am launching a pod with limited resources, the devspace ssh processes alone are taking about 1.5gb ram
this is too much memory, is it possible somehow to reduce it?
this is the dev section in devspace.yml
dev:
frontend:
labelSelector:
app: frontend
env:
- name: DEVELOPER_SCHEMA
value: ${devspace.namespace}
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: 30m
memory: 500Mi
sync:
- path: ./:/data
excludePaths:
- node_modules
- main/core.*
- main/node_modules
- main/.angular
- main/dist/
- .githooks
- main/.webstorm_nodejs_helpers
- .stfolder
- .vscode
- .idea
- .git
startContainer: true
workingDir: /data/main
# Open a terminal and use the following command to start it
terminal:
command: /data/devspace_start.sh
ssh:
enabled: false
proxyCommands:
- command: devspace
- command: kubectl
- command: helm
- command: git
- gitCredentials: true
SudoMishra
commented
1 month ago Any updates on this issue I have been facing the same. Even after the pod is initialised and I have access to the shell the /tmp/devspacehelper is still consuming memory.
demetris-manikas
commented
1 month ago Same here
What happened?
This is not exactly a bug, since devspace handles it well, I just was amazed by the memory used at some point by the
devspacehelper synccommands and was wondering whether this could be lowered somehow.When running a project using
devspace dev, the devspace sync processes are killed due to a cgroup memory violation. That looks like this:As you can see, devspace handles this well and retries injecting the
devspacehelper syncprocesses after the OOM exit code, and after a few tries, sync works. However, I cannot lay a finger on why it works after a few tries, perhaps memory usage is lower after a few tries, due to less things to sync?The sync configuration of this particular project was low volume, around 10 files of only a few KBs each, without anything to initially sync.
When the sync is running smoothly after a while, looking at the size of the
devspacehelper syncprocesses, memory usage seemed normal to around 15MB per process. However, looking at peak memory usage, I was amazed to see:So both processes used around \~715MB some period in time, which seemed a little excessive to me. It might be that I'm looking at it wrong, so therefore just checking out what you think :].
What did you expect to happen instead?
The
devspacehelper syncprocesses use not that much peak memory.How can we reproduce the bug? (as minimally and precisely as possible)
Limit the container you're developing in k8s to say 50Mi, and configure a sync too.
Local Environment:
devspace version 5.16.2Kubernetes Cluster:
/kind bug