Tests fails when running more containers in parallel

[success-stha]

We are currently using a machine with 16 GB of RAM and 16 cores CPU. When container limit is set to 18 the tests runs fine but when increased to 27 the tests fail. The screenshot shows the tests failing with a loading icon. Here is the browser.json.

{
    "chrome": {
        "default": "78.0",
        "versions": {
            "78.0": {
                "image": "selenoid/vnc:chrome_78.0",
                "port": "4444",
                "volumes": ["/opt/docker/fileDownloadLocation:/home/selenium/Downloads"],
                "shmSize" : 1073741824,
                "tmpfs": {"/tmp": "size=768m"}
            }
        }
    }
}

[vania-pooh]

@success-stha according to our docs we recommend to have no more than 1.5-2 browsers per CPU core. However sometimes only 1 CPU per browser works correctly. With 16 CPUs and 27 parallel sessions you have 27 / 16 = 1.69 browsers per core which seems to be a reasonable ratio. If your tested application has a lot of Javascript - then you could be running out of memory or CPU is getting overloaded by Javascript processing. So take a look at system metrics such as load average, memory consumption and iowait of the host machine to understand the reason.

[success-stha]

[  ~]# uptime
 02:42:31 up 11 days, 19:33,  2 users,  load average: 1.40, 1.63, 1.09
[  ~]# uptime
 02:42:33 up 11 days, 19:33,  2 users,  load average: 1.40, 1.63, 1.09
[  ~]# uptime
 02:42:35 up 11 days, 19:33,  2 users,  load average: 1.37, 1.62, 1.09
[  ~]# uptime
 02:42:36 up 11 days, 19:33,  2 users,  load average: 1.37, 1.62, 1.09
[  ~]# uptime
 02:42:37 up 11 days, 19:33,  2 users,  load average: 1.37, 1.62, 1.09
[  ~]# uptime
 02:45:22 up 11 days, 19:35,  2 users,  load average: 2.86, 2.21, 1.41
[  ~]# uptime
 02:45:23 up 11 days, 19:35,  2 users,  load average: 2.86, 2.21, 1.41
[  ~]# uptime
 02:45:24 up 11 days, 19:35,  2 users,  load average: 2.71, 2.19, 1.41
[  ~]# uptime
 02:45:24 up 11 days, 19:36,  2 users,  load average: 2.71, 2.19, 1.41
[  ~]# uptime
 02:45:25 up 11 days, 19:36,  2 users,  load average: 2.71, 2.19, 1.41

Here's the load average. The memory consumption is also fine. The tests fail right after the setup which takes a lot of time to complete (around 10 mins).

[vania-pooh]

@success-stha check that you have recommended Docker settings. https://aerokube.com/selenoid/latest/#_recommended_docker_settings

[success-stha]

I used the recommended settings. But i was unable to change docker0 mac-address as eth0 was not showing up. I guess this might be the problem. There's one named ens192. I have used its mac-address for now but the problem still exists.

ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.48.23.154  netmask 255.255.255.0  broadcast 10.48.23.255
        ether 00:50:56:bd:de:ff  txqueuelen 1000  (Ethernet)
        RX packets 14399286  bytes 7689433723 (7.1 GiB)
        RX errors 0  dropped 1861  overruns 0  frame 0
        TX packets 2127460  bytes 301368239 (287.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[vania-pooh]

@success-stha what is your Docker storage driver? Do you have overlay2?

[success-stha]

@success-stha what is your Docker storage driver? Do you have overlay2?

yes, the storage driver is overlay2. Currently we are using 13 containers and dockerstats looks like this.

CONTAINER ID        NAME                   CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
3f41d179935f        festive_ride           40.41%              201.8MiB / 15.54GiB   1.27%               15.4MB / 417kB      0B / 359kB          155
f73afff85c93        inspiring_bose         8.06%               222.1MiB / 15.54GiB   1.40%               7.94MB / 363kB      0B / 406kB          159
41ebad27e1fd        dazzling_matsumoto     32.31%              205.3MiB / 15.54GiB   1.29%               17.4MB / 537kB      0B / 507kB          164
432a72f2a199        unruffled_galois       24.38%              201.7MiB / 15.54GiB   1.27%               13.8MB / 486kB      0B / 913kB          162
f3590c869c53        competent_tereshkova   71.28%              202.5MiB / 15.54GiB   1.27%               16.6MB / 497kB      0B / 369kB          163
eb6e6152e6f7        bold_hopper            59.79%              207.2MiB / 15.54GiB   1.30%               15.5MB / 514kB      0B / 341kB          160
bb969a772f9d        inspiring_ramanujan    8.87%               202.3MiB / 15.54GiB   1.27%               9.34MB / 352kB      0B / 368kB          161
15695e8d44cf        quizzical_bardeen      14.78%              191MiB / 15.54GiB     1.20%               10.2MB / 357kB      0B / 439kB          154
e83dfe3510e0        suspicious_khayyam     29.98%              208.2MiB / 15.54GiB   1.31%               13.5MB / 434kB      0B / 500kB          158
1ee04dac7d1a        funny_kalam            76.66%              201.8MiB / 15.54GiB   1.27%               17.4MB / 546kB      0B / 453kB          161
de1f12c01e3f        infallible_shannon     63.01%              202.7MiB / 15.54GiB   1.27%               15MB / 449kB        0B / 462kB          159
b1eb5f9656d1        elastic_easley         25.33%              200MiB / 15.54GiB     1.26%               11.8MB / 413kB      0B / 550kB          155
d645cc78346d        hopeful_dirac          69.07%              197.6MiB / 15.54GiB   1.24%               15.3MB / 516kB      0B / 366kB          158
8214859c4a64        selenoid               1.94%               12.96MiB / 15.54GiB   0.08%               12.5MB / 12.6MB     0B / 0B             28
055c600e4fe8        selenoid-ui            0.00%               22.59MiB / 15.54GiB   0.14%               11.9MB / 14.4MB     0B / 0B             19

[success-stha]

Any updates?

[success-stha]

The app we are testing is made using Angular. I have already shared all the info about cpu and memory consumption above. I am really struggling to run more than 16 containers in parallel. Any help from you would be highly appreciated.

[vania-pooh]

@aandryashin any ideas?

[aandryashin]

What are errors in tests? Could you please try to run synthetic tests in 16 parallel sessions?

пн, 17 февр. 2020 г., 14:43 Ivan Krutov notifications@github.com:

@aandryashin https://github.com/aandryashin any ideas?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aerokube/selenoid/issues/862?email_source=notifications&email_token=AAKY23PJZXBPIM2OWMR6H3DRDJZ45A5CNFSM4KJAT7PKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL6DS3A#issuecomment-586955116, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKY23MDMSDRPR7557XAMOLRDJZ45ANCNFSM4KJAT7PA .

[success-stha]

What do you mean by synthetic tests? The screenshots shows the test failing with a loading icon. The log shows null pointer exception.

[aandryashin]

Please provide us with exact error you are receiving? I cannot see any null pointer error...

вт, 18 февр. 2020 г., 16:23 success-stha notifications@github.com:

What do you mean by synthetic tests? The screenshots shows the test failing with a loading icon. The log shows null pointer exception.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aerokube/selenoid/issues/862?email_source=notifications&email_token=AAKY23LORAM3Z2MUTPDWVF3RDPOO7A5CNFSM4KJAT7PKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMB6LKQ#issuecomment-587457962, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKY23LWXTYHPGAQQ2FB2WDRDPOO7ANCNFSM4KJAT7PA .

[success-stha]

Since the page keeps on loading for unknown reason, No such element exception is shown.

Caused by: org.openqa.selenium.NoSuchElementException: Cannot locate an element using By.xpath: //*[contains(@class,'nodata ng-star-inserted')]
build   19-Feb-2020 04:11:34    For documentation on this error, please visit: http://seleniumhq.org/exceptions/no_such_element.html
build   19-Feb-2020 04:11:34    Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:19:58.91Z'
build   19-Feb-2020 04:11:34    System info: host: 'nvbmblinp6', ip: '10.48.23.150', os.name: 'Linux', os.arch: 'amd64', os.version: '2.6.32-754.11.1.el6.x86_64', java.version: '1.8.0_131'
build   19-Feb-2020 04:11:34    Driver info: driver.version: unknown
build   19-Feb-2020 04:11:34        at org.openqa.selenium.support.ui.ExpectedConditions.lambda$findElement$0(ExpectedConditions.java:896)
build   19-Feb-2020 04:11:34        at java.util.Optional.orElseThrow(Optional.java:290)
build   19-Feb-2020 04:11:34        at org.openqa.selenium.support.ui.ExpectedConditions.findElement(ExpectedConditions.java:895)
build   19-Feb-2020 04:11:34        at org.openqa.selenium.support.ui.ExpectedConditions.access$000(ExpectedConditions.java:44)
build   19-Feb-2020 04:11:34        at org.openqa.selenium.support.ui.ExpectedConditions$7.apply(ExpectedConditions.java:206)
build   19-Feb-2020 04:11:34        at org.openqa.selenium.support.ui.ExpectedConditions$7.apply(ExpectedConditions.java:202)
build   19-Feb-2020 04:11:34        at org.openqa.selenium.support.ui.ExpectedConditions$47.apply(ExpectedConditions.java:1427)
build   19-Feb-2020 04:11:34        at org.openqa.selenium.support.ui.ExpectedConditions$47.apply(ExpectedConditions.java:1421)
build   19-Feb-2020 04:11:34        at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:260)
build   19-Feb-2020 04:11:34        ... 19 common frames omitted

The same test passes when the number of container is reduced.

[vania-pooh]

@success-stha could be because of overloaded tested application server then.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.