Open OndrejKu opened 2 years ago
need to assign more cpus to docker container, not sure how though
I have noticed the same behavior. Have you found a solution @OndrejKu ?
We see this too. Specifically: Docker Desktop on OSX configured with 8 CPUs, tests create 8 databases as expected, but somehow the tests are serialized and the total runtime is slower: 45 mins without parallel, 60 mins with parallel. In our case, the PostGres DB is running on the host machine, maybe it is related -- I kind of wonder if the DB calls out of the container are all being serialized.
We see this too. Specifically: Docker Desktop on OSX configured with 8 CPUs, tests create 8 databases as expected, but somehow the tests are serialized and the total runtime is slower: 45 mins without parallel, 60 mins with parallel. In our case, the PostGres DB is running on the host machine, maybe it is related -- I kind of wonder if the DB calls out of the container are all being serialized.
I also use Docker Desktop on macOS... but I have a specific container for the DB and I have the same behaviour.
need to assign more cpus to docker container, not sure how though
@grosser I checked this and it's already the case. In my case, I use Docker Desktop on macOS which by default uses half of the host cpus (5 in my case). By default, these 5 cpus can be used without limit by any container [1]. So I don't think it's the problem.
[1] https://docs.docker.com/config/containers/resource_constraints/ => "By default, each container’s access to the host machine’s CPU cycles is unlimited."
docker info | grep -iE "CPUs|Memory"
CPUs: 5
Total Memory: 5.8GiB
run with rake parallel:spec[8]
?
... or is the bug that it does not detect cpu count inside of docker, that should be easy to reproduce ... maybe hard to fix 🤞
run with
rake parallel:spec[8]
? ... or is the bug that it does not detect cpu count inside of docker, that should be easy to reproduce ... maybe hard to fix 🤞
On my side, with rake parallel:spec[5]
it's exactly le same as rake parallel:spec
. I also tested to run rake parallel:spec
on the host and keep the containers for postgres/redis/... and I have almost the same behaviour as I run it without Docker. It seems something happen only in the container where we launch the command (not related to the others containers).
We are seeing that the serialization/slow-down is only happening on our M1 MacBooks ... tested on 3 different Intel MacBooks and the parallelization inside of Docker Desktop for Mac works great on those. Whereas 3 different M1 MacBooks experience the terrible performance. So we are suspecting the problem is somehow related to Docker Desktop on M1 chips.
Hi @Brad-FDI, I am encountering similar issue and have been trying to find a solution, but haven't had any luck yet. I understand this is an ongoing issue with Docker on M1 chips. Have you been able to find a workaround or solution that improves Docker's performance in this situation? Any suggestions or insights would be greatly appreciated. Thanks!
Hi @mezbahalam, unfortunately we have no workaround, we're just relying on cloud CI more where parallel_tests works well.
I'm running my env in docker-compose. I can prepare the database, I see 8 additional databases were created and schema was loaded. Then I run this:
and what I see is 8 running specs as expected, but sequentially not in parallel. Also the runtime is higher then just using simple rspec.