Periodic high CPU usage

[gmta]

When the docksal-vhost-proxy container is running, the CPU usage is peaking periodically (about every 5 seconds) without any activity on the user's behalf.

Easiest way to see this is to start Docksal and run docker stats. The docksal-vhost-proxy container should show a CPU usage of about 20%-50% every few seconds.

[lpeabody]

Can you provide a sample of the vhost-proxy logs as they are written during these spikes?

EDIT: fin docker logs -f VHOST_PROXY_CONTAINER_ID is probably the best way to monitor it. You need to replace VHOST_PROXY_CONTAINER_ID with the container id though, you can find it by fin docker container ls | grep vhost.

[gmta]

Sure; it's just leaving a cron trail behind every minute (so not as often as the CPU spikes):

crond: USER root pid 1656 cmd /usr/local/bin/proxyctl cron
2019-02-15 16:05:00 [proxyctl] [cron]
2019-02-15 16:05:00 [proxyctl] [stop]
crond: USER root pid 1805 cmd /usr/local/bin/proxyctl cron
2019-02-15 16:06:00 [proxyctl] [cron]
2019-02-15 16:06:00 [proxyctl] [stop]
crond: USER root pid 1958 cmd /usr/local/bin/proxyctl cron
2019-02-15 16:07:00 [proxyctl] [cron]
2019-02-15 16:07:00 [proxyctl] [stop]
crond: USER root pid 2097 cmd /usr/local/bin/proxyctl cron
2019-02-15 16:08:00 [proxyctl] [cron]
2019-02-15 16:08:00 [proxyctl] [stop]
crond: USER root pid 2251 cmd /usr/local/bin/proxyctl cron
2019-02-15 16:09:00 [proxyctl] [cron]
2019-02-15 16:09:00 [proxyctl] [stop]

[lpeabody]

I was curious because I ran into this issue with vhost-proxy in CI where I had a BAN request being made to a site, but the BAN request was being made to a host that didn't exist in the environment and so vhost-proxy was just churning constantly. I literally couldn't access anything on the container network because vhost-proxy was so bogged down from bad requests.

At any rate, I discovered what the issue was by inspecting the logs of vhost-proxy, I was curious if there was anything similar happening in your circumstances. Doesn't look like there is. Not sure what would be going on at this point. I'll leave it to the maintainer crew from here :)

[achekulaev]

Yeah it is docker-gen. It's not critical since it's 20-50% on one core, and you have 4 of them (at least), but truly annoying. At the same time it would take a major redesign to change it IMHO

https://github.com/docksal/service-vhost-proxy/blob/develop/conf/supervisord.conf#L27

[gmta]

If I have 4 cores, I do not necessarily assign all 4 of them to the Docker for Mac/Windows VM but probably 2 or 3. Then, periodically the vhost-proxy service starts to chug on CPU cycles which will probably negatively influence the performance of my other containers (i.e. the applications I'm trying to run with Docksal). So fixing this will not only increase battery lifetime on laptops running Docksal, but will probably also increase other containers' performance.

Would it be possible to trigger docker-gen using a signal (TCP/unix socket? Command?) instead of making it "watch" (which I'm guessing is polling instead).

[lmakarov]

I don't think there is much we can do with the CPU spikes from docker-gen in the current iteration of docksal/vhost-proxy.

Everyone is welcome to brainstorm and through ideas for v2 here: https://github.com/docksal/service-vhost-proxy/issues/29

[lmakarov]

Here's a potentially viable stop-gap solution: https://github.com/docksal/docksal/issues/1032#issuecomment-484111688

Please try and report results.

[lmakarov]

Looks like that approach won't result in any CPU cycle/Energy consumption savings, but rather will make the load even. After setting --cpus=".05" for docksal-vhost-proxy the occasional 60% CPU spikes (fin docker stats docksal-vhost-proxy) have been replaced with a more or less steady 5% load for me. This makes sense, since throttling CPU on a process does not mean it will use less of it eventually :)

[achekulaev]

@lmakarov seems like a good candidate for 1.12.2 if it proves to be working fine

[lmakarov]

This has been addressed in docksal/service-vhost-proxy#48 and released as a hotfix in docksal/vhost-proxy v1.5.1.

Run fin update to get the fix.