Screenshots comparison - running out of memory

[vdekov]

Hi @garris, First of all, thank you for the amazing tool.

But I've a question as I've an issue with some memory leak on the screenshots comparison phase. My machine has 2GBs of RAM and 1GB of swap space. I am with latest version of BackstopJS (v3.0.26) and Node (v7.10.1) running over Ubuntu 16.04 VM.

I am using the following configuration:

{
  "id": "my-project",
  "paths": {
    "bitmaps_reference": "my-project/bitmaps_reference",
    "bitmaps_test": "my-project/bitmaps_test",
    "casper_scripts": "my-project/casper_scripts",
    "html_report": "my-project/html_report",
    "ci_report": "my-project/ci_report"
  },
  "viewports": [
    {
      "name": "laptop",
      "width": 1280,
      "height": 1024
    }
  ],
  "onBeforeScript": null,
  "onReadyScript": null,
  "scenarios": [
    {
      "label": "MainPage",
      "url": "{url}",
      "delay": 500,
      "misMatchThreshold": 0.1,
      "readyEvent": null,
      "requireSameDimensions": true
    }
  ],
  "casperFlags": [],
  "engine": "phantomjs",
  "report": [
    "CI"
  ],
  "debug": false,
  "asyncCompareLimit": 1
}

I executed it over 40 URL scenarios through the CLI. Maybe it is important to clarify that the reference and test screenshots are using different URL-s. Also I've expecting a large percent of content mismatch between them.

I am executing backstop reference without problems. When I run backstop test the screenshots have been successfully created (Bitmap file generation completed.) but the process is killed on the comparison phase (COMMAND | Executing core for report) around the 25-30 screenshot comparison because running out of memory. As a result the report is never generated.

...
compare | ERROR { requireSameDimensions: false size: isDifferent, content: 52.01%, threshold: 0.1% }: HelpAfrica_march-2017 my-project_HelpAfrica_march-2017_0_document_0_laptop.png
See: /var/www/my-project/bitmaps_test/20171009-132204/failed_diff_my-project_HelpAfrica_march-2017_0_document_0_laptop.png
Killed

I tried to set different values of the asyncCompareLimit (between 1 and 100) but it doesn't change anything. The process still being killed. The only workaround which I can use at the moment is to divide the scenarios in a separate batches (i.e. 10-15 links in a batch) and to run them each one of the in a different process... but this seems to be not so comfortable as a flow.

Is there any gap in my configuration and if so - how can I fix it? Just say if there is anything else which I can provide as an information.

Thank you!

[garris]

@vdekov Thank you for posting and thank you for taking the time to detail the issue. Unfortunately, the Phantom option probably requires more RAM (I think Phantom alone uses ≈1.5G -- the comparison may also need >.5G) -- maybe doubling to 4G+ will work better? Also, BackstopJS v3 is has a chrome headless option which I think requires less resources -- maybe this will work better for you?

[vdekov]

Thank you for the fast answer @garris.

I applied your suggestions - to increase the resources (to 4GBs of RAM) and to activate the Chromy engine. Definitely the headless Chrome browser is much faster than the PhantomJS. This is a great improvement!

I've successfully executed the batch of 37 scenarios on the top of the new environment. But I noticed that at the comparison phase the memory is still increasing progressively. I did a screen record during the backstop test execution (you can track the memory at the top): https://drive.google.com/open?id=0B8h5lUofXpg5ajNxOTkzLVkyRjA In my opinion if the batch is larger the process again will force running out of memory as the memory never be released during the comparison.

Here are my values using in the setup about the count of the parallel tasks:

  "asyncCompareLimit": 1,
  "asyncCaptureLimit": 5

In my experience such issue can be observed if there are a big mismatch of the content of the reference and test screenshots (e.g. see in the video content: 33.30%)

Now I'll try to update the NodeJS to the latest version as an attempt to get the things better but I'm not sure this will fix the increasing memory.

Is this normal or is it possible to be related with the VM OS (Ubuntu 16.04) or something else in my environment?

Thank you!

[garris]

There was a point where the memory leak on compare was very bad. This was fixed many versions ago. Since then here have been some speed optimization in that area — but perhaps they may have introduced a memory leak? It would definitely be great is someone could take a look. I mostly use it on MacOS and have never run into a RAM issue (most of my team is on macOS with 16g and our Linux boxes are very powerful — so even with large tests we have not run into an issue)

What screen sizes are you using?

[vdekov]

Yes, I read the topics about the memory leak in the older versions.

The only used viewport is 1280x1024:

  "viewports": [
    {
      "name": "laptop",
      "width": 1280,
      "height": 1024
    }
  ]

I also run the same configuration on my Mac but I think that the memory behavior was better than the Ubuntu VM (still it was high on the comparison phase but not enough to kill the process).

[vdekov]

Hi @garris,

I run a few more tests on different environments to prove myself the memory leak.

I executed the tests on two different Linux distributions - Ubuntu and CentOS with 16GBs of RAM both. The tests are including 110 scenarios executed of 3 different screen sizes (330 screenshots in total). Both of them are running out of memory around the 85-90th scenario on the comparison process. AFAIK BackstopJS is using Resemble.js for the comparison. Is it possible the issues to be coming from there?

At last I tried to run the same configuration on my Mac (8GBs of RAM). This time successfully. I made a small research and the reason is the "compressed memory" in the OSX (it is some kind of memory optimization of the oldest data stored in the memory). But this doesn't prevent the constantly increasing memory...

I have another question - is there a way to skip the comparison process between the "reference" and "test" screenshots but to keep the report generation (the DIFF column will be excluded)?

[garris]

Thank you for looking into this. Very interesting. So, yes — resemble is used for comparisons — however I am confident the leak is not from resemble (we fixed a leak there before). However, there is an optimization step that was added in front of the resemble step — it quickly creates hashes of the reference and test images to determine if the files are the same. Files which are the same are not compared bit-by-bit in the resemble process. I suspect that images are getting cached in this hash process and not getting cleaned up after the hash evaluation is performed.

[garris]

Some ideas..

This file here: https://github.com/garris/BackstopJS/blob/master/core/util/compare/compare-hash.js

Try...

• setting the hashes arr to null after comparison.
• cleaning up the stream var after returning
• the .on() closures could also be an issue — probably need a .off() in there after evaluating.

[garris]

@vdekov can you share a config + link to a project which demonstrates the memory leak? I think I make a fix -- but I want to make sure I can repro first.

[vdekov]

Hi @garris, Thank you for the quick response and the time you've spent.

I share with you a link to my "reference" and "test" configurations (they differ only by the scenario URLs):

• reference: https://drive.google.com/open?id=0B8h5lUofXpg5My1jSE5SM2RyVEE
• test: https://drive.google.com/open?id=0B8h5lUofXpg5OE01bm5CSHRsRFU

I tried yesterday the proposed by you ideas but they didn't solve the issue on my end. But I do some tests and I'm sure the reason for the leak is from the comparing of the images when their hash sums are different i.e. from the resemble comparison process.

I start an implementation with a child processes which will be spawned for each pair comparison. I think in this way the garbage collector will have the time to release the already used resources. Today I'll try to finish and to check the memory behavior.

Do you think this will be an appropriate solution?

Also I've another question related with the used Resemble.js module. I saw that BackstopJS relies on node-resemble.js - a project forked by the Resemble.js repository. Also I noticed that they have differences in the image comparison algorithm - node-resemble-js compares the images as a pictures (pixel by pixel) whilst the Resemble.js draw canvas elements and compares them (in this line of thinking you have to install the canvas package as a dependency). Also the node-resemble-js project is looking for a maintainer at the moment. Is the node-resemble-js used with a purpose (Resemble.js also supports Node.js image comparison)?

I did an experiment to replace the node-resemble-js with Resemble.js package and with a few changes I successfully executed the comparison (there are a few differences in the interface between both of them).

Thank you!

[mirzazeyrek]

@vdekov node-resemble-js maintainer here! Could you please share the changes you have made ?

And Can you confirm when you've changed the comparison method the memory issue is gone ?

[vdekov]

@garris in continuation of my previous comment. I think I solved the memory leak issue which I have following the described above algorithm. I created a child process which compares each pair. This process is killed after the comparison and for the next pair a new process is spawned and etc. Meanwhile the garbage collector does its job.

I made some before and after performance tests executed over 37 scenarios on 1 screen size with the following configuration:

  "asyncCompareLimit": 5,
  "asyncCaptureLimit": 3

Here are the average times needed for the comparison phase:

Now	Forked processes
134.449s	64.008s
134.345s	63.411s
135.271s	62.067s
133.92s	62.636s

The maximum used memory per a subprocess is around 140-160MB. I'm happy also because the memory of the VM never exceeds over 1.3GB for my configuration.

The BackstopJS sanity check was executed successfully so I can prepare a PR if you are agree to review the code.

P.S. I forgot to mention in my previous comment - if you want to be able to reproduce the issue you have to use a Linux based VM. OSXs have a "compressed memory" but their memory also is increasing to infinity.

[garris]

This sounds absolutely great! Yes, please post PR and we can try to get this in today. Thanks for your effort!

[garris]

fixed by #571