Open NassirFfx opened 2 years ago
pedrovgs
commented
2 years ago 🤔 @NassirFfx the sample projects we have are not able to reproduce this error. can you please upload a sample project we can use to reproduce this bug? I'm also wondering if this is related to the number of images you have or the size of some images.
bartsg
commented
2 years ago I'm having the same issue. It seems to happen only when a certain amount of differences are detected (e.g. 250+ changed screenshots or so).
adesentenac
commented
2 years ago I'm also reproducing the issue with Shot 5.14.1. I have 22 screenshots which have a resolution of 1080x2400. If I run them on a different device with a tolerance of 1.0, so they all pass with a warning, I have an OOM exception if Xmx is set at 2G, but pass if I change it to 4G.
karsie
commented
2 years ago I get this when I have a significant number of differences. (And I know tolerance is applied after gathering differences). From looking at the code of the image comparison library, the AWT images are never flushed. Maybe adding this will help?
karsie
commented
2 years ago For me PR #314 helps
mhiew
commented
2 years ago I think the issue is due to parallelism when comparing images and not really a garbage collection issue. (While debugging locally I could see that the total used memory was properly being recycled during execution)
screenshots.par.flatMap(compareScreenshot(_, tolerance)).toListI am not too familiar with scala, but I believe converting the collection to a parallel collection with par will by default set the number of threads to Runtime.AvailableProcesses. I think this matches the number of available cores on your machine.
When there are a lot of image differences the imagesAreDifferent function can potentially take up quite a bit of memory due to allocating an array of pixels for each set of screenshots to compare against.
These two factors can lead to random out of memory issues depending on which set of screenshots are being processed at the same time. If we have 4 threads then we are loading 4 sets of images at the same time. The sets of images that get processed per run is somewhat random as par doesn't guarantee ordering. So my hunch is the inconsistent failures happen due to bad luck of processing a set of multiple images that doesn't fit into memory.
I tried to set the maximum concurrency using -Dscala.concurrent.context.maxThreads=1 but unfortunately that didn't seem to work for me.
To test I cloned the project and locally tested setting a maximum thread pool size which seems to have alleviated my issues.
//set the maximum number of threads to 2
private val taskSupport = new ForkJoinTaskSupport(new ForkJoinPool(2))
def compare(screenshots: ScreenshotsSuite, tolerance: Double): ScreenshotsComparisionResult = {
val screenshotsParallel = screenshots.par
screenshotsParallel.tasksupport = taskSupport
val errors =
screenshotsParallel.flatMap(compareScreenshot(_, tolerance)).toList
ScreenshotsComparisionResult(errors, screenshots)
}Perhaps one solution would be add an optional max threads to the plugin extension so that we can control the parallelism on memory constrained machines (CI executors)?
I don't think this will ultimately solve all issues as even with 2 threads we could run into a situation where the set of images does not fit into memory. However, being able to control it or even set it to 1 max thread would likely reduce the likelihood of this issue happening.
Alternatively, I tried removing parallelism altogether and it fixed my issue as well. Anecdotally the comparison time for my project (1.5k screenshots) took about 6 seconds instead of 2 seconds. This isn't too bad for my use case so I may go with this option as the bulk of the time is spent generating the comparison screenshots.
screenshots.flatMap(compareScreenshot(_, tolerance)).toList
AlmightyCZ
commented
1 year ago For me PR #314 helps
I noticed that you closed it due to it not being considered a "complete solution." Could you please clarify what aspects are still missing?
Currently, I'm encountering "heap space" and "java.lang.IllegalArgumentException: Self-suppression not permitted" exceptions after transitioning to a higher Android version on my testing AVDs. I had to set tolerance = 0.01 due to variations in a few pixels and I usually get one of the exceptions for test runs with 200+ shots. (shorter test are fine) I've already set org.gradle.jvmargs=-Xmx8192M long time ago.
karsie
commented
2 months ago I only managed to fix all my issues by converting the entire thing to Kotlin (mainly due to the lack of Scala knowledge on my part). What I also encountered why converting it, is that there is 1 action where all recorded screenshots are copied using Scrimage (and thus loading every file into an awt image), where this could easily be achieved using regular file copy.
Comparing the screenshot step is failing with an error: Java heap space Further, look into gives:
Snapshot tests on Phone
Version: 5.13.0