Adding the Full grid components to the build and deploy process

[jamesmortensen]

Thanks for contributing to the Docker-Selenium project! A PR well described will help maintainers to quickly review and merge it

This adds in all of the other grid components, like node-docker, standalone-docker, sessions, session-queue, distributor, router, event-bus, and it also splits up the build process into multiple separate jobs to enable parallelism. The build process is slightly faster, but is still dependent on about 20 minutes waiting for the Base and NodeBase to build before starting to build the browser images.

Before submitting your PR, please check our contributing guidelines, applied for this repository. Avoid large PRs, help reviewers by making them as simple and short as possible.

Description

• Build and deploy process is split into multiple jobs:

Motivation and Context

• CircleCI Free plan times out at 60 minutes. This limit cannot be increased, and the build/deploy cycle was not only approaching it but also exceeding it.
• Splitting it apart reduces the total time to about 45 minutes and also possibly allows each job to run for 60 minutes each, increasing the amount of breathing room we have for adding more things in the future.

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [X] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• [X] I have read the contributing document.
• [ ] My change requires a change to the documentation.
• [ ] I have updated the documentation accordingly.
• [ ] I have added tests to cover my changes.
• [X] All new and existing tests passed.

[jamesmortensen]

I am doing deployments from this branch for a bit until I'm confident this is a stable way to build the images. One problem is that, since the images are built on different runners, I am exhausting the docker pull queue in some cases.

One workaround is to use different credentials on different runners in order to expand the docker pull queue. For now, I'm using the anonymous pull queue.

• Anonymous users are limited to 100 pulls per 6 hour period.
• Free logged in users are limited to 200 pulls per 6 hour period.

By doing some work anonymously, we theoretically expand this count to 300 pulls per 6 hour period by only logging in when we also need push access.

[jamesmortensen]

There are 2 types of errors that I'm seeing during the build process. The first is a "failed to solve" error that arises when attempting to build a Standalone browser image immediately after pushing a Node browser image. I'll experiment with adding sleep 5 in order to give the system enough time to complete whatever needs to happen for the Node browser image to be available to be pulled again.

The other type of error is due to apt-get install failing due to E: Sub-process /usr/bin/dpkg returned an error code (1) errors when installing packages. I am not sure if it's a CircleCI issue or related to using Debian Sid, but there may be ways to retry the command if it fails, up to a certain number of times: https://unix.stackexchange.com/a/730495/6509

The challenge with retries is if the install command starts from the beginning, the total runtime of the workflow may exceed the 60 minute time limit. Building multi-arch images is expensive due to the need for emulation.

Another possible technique to build images faster, and with more stability, would be to build for each architecture natively and then merge them together using buildx imagetools or docker manifest. The only issue is finding a cloud provider who can run Linux on ARM32 platforms. Building the armv7l images still require emulation since CircleCI only offers amd64 and arm64 virtual machines.