search

creativescala / doodle

Compositional vector graphics in Scala / Scala.JS
https://creativescala.org/doodle/
Apache License 2.0
327 stars 75 forks source link

`golden/test` fails on Scala 2.13.6 #103

Closed SethTisue closed 3 years ago

SethTisue commented 3 years ago

++2.13.6 followed by golden/test gives

==> X doodle.golden.Text.text-color  0.045s munit.ComparisonFailException: /Users/tisue/doodle/golden/src/test/scala/doodle/golden/Text.scala:40
39:
40:  testPicture("text-color") {
41:    text[Algebra, Drawing]("Red")
Heights differ
=> Obtained
34
=> Diff (- obtained, + expected)
-34
+33
    at munit.FunSuite.assertEquals(FunSuite.scala:11)
    at doodle.golden.GoldenPicture.assertGoldenPicture(GoldenPicture.scala:32)
    at doodle.golden.GoldenPicture.assertGoldenPicture$(GoldenPicture.scala:16)
    at doodle.golden.Text.assertGoldenPicture(Text.scala:11)
    at doodle.golden.GoldenPicture.$anonfun$testPicture$1(GoldenPicture.scala:59)

this came up in the Scala 2.13 community build

noelwelsh commented 3 years ago

Thanks for the report. I run into this from time to time and I think this is due to differences in Java2D rendering between JVM versions and OS versions. What JVM version does the 2.13.6 build use, and, if Doodle is in the 3 build, does it differ from that? Thanks!

On Sun, Aug 29, 2021 at 4:26 PM Seth Tisue @.***> wrote:

++2.13.6 followed by golden/test gives

==> X doodle.golden.Text.text-color 0.045s munit.ComparisonFailException: /Users/tisue/doodle/golden/src/test/scala/doodle/golden/Text.scala:40 39: 40: testPicture("text-color") { 41: textAlgebra, Drawing Heights differ => Obtained 34 => Diff (- obtained, + expected) -34 +33 at munit.FunSuite.assertEquals(FunSuite.scala:11) at doodle.golden.GoldenPicture.assertGoldenPicture(GoldenPicture.scala:32) at doodle.golden.GoldenPicture.assertGoldenPicture$(GoldenPicture.scala:16) at doodle.golden.Text.assertGoldenPicture(Text.scala:11) at doodle.golden.GoldenPicture.$anonfun$testPicture$1(GoldenPicture.scala:59)

this came up in the Scala 2.13 community build

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/creativescala/doodle/issues/103, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAH5XFNDEGKBNJ2A7W5CCDT7JGRDANCNFSM5DALA4AA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

SethTisue commented 3 years ago

I am not sure if I have seen it fail on our Jenkins (which runs Ubuntu) or not.

But I know I'm able to reproduce it locally on macOS 11.5.2 on GraalVM 11:

~/doodle % java -version
openjdk version "11.0.12" 2021-07-20
OpenJDK Runtime Environment GraalVM CE 21.2.0 (build 11.0.12+6-jvmci-21.2-b08)
OpenJDK 64-Bit Server VM GraalVM CE 21.2.0 (build 11.0.12+6-jvmci-21.2-b08, mixed mode, sharing)

and also AdoptOpenJDK 8:

openjdk version "1.8.0_292"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)
noelwelsh commented 3 years ago

So far I've been treating the CI results (Github actions on Ubuntu) for the golden tests as definitive, and mostly ignoring what happens on my local machine beyond visual inspection of the generated images. I could make the image diffing more robust to differences in size. In general image diffing is a hard problem, and I'm not sure how much effort to invest in this.

noelwelsh commented 3 years ago

I've added some code that might address this.

SethTisue commented 3 years ago

it still fails for me. (after #104, which is needed for the tests to even compile.)

noelwelsh commented 3 years ago

It does fail with a different error message, though, right? I can make the code print out the offending image and the diff, which might go some way to debugging this. I fear writing an acceptable image diff algorithm may become a project in its own right.

SethTisue commented 3 years ago

currently the error is:

==> X doodle.golden.Text.text-color  0.021s munit.FailException: /Users/tisue/doodle/golden/src/test/scala/doodle/golden/Text.scala:44 Height or width differ by more than one pixel
43:      .beside(text[Algebra, Drawing]("Blue").strokeColor(Color.blue))
44:  }
45:}
    at munit.FunSuite.assert(FunSuite.scala:11)
    at doodle.golden.Golden.imageDiff(Golden.scala:74)
    at doodle.golden.Golden.imageDiff$(Golden.scala:13)
    at doodle.golden.Text.imageDiff(Text.scala:11)
    at doodle.golden.GoldenPicture.assertGoldenPicture(GoldenPicture.scala:28)

I think you could reasonably mark the test as to be skipped except on Linux.

noelwelsh commented 3 years ago

I've updated the error message so there should hopefully be much more useful output. If you can paste that in we'll be able to tell if its a spurious error or not, and take action accordingly.

SethTisue commented 3 years ago

AdoptOpenJDK Java 1.8.0_292, macOS:

==> X doodle.golden.Text.text-color  0.039s munit.FailException: /Users/tisue/doodle/golden/src/test/scala/doodle/golden/Text.scala:44
43:      .beside(text[Algebra, Drawing]("Blue").strokeColor(Color.blue))
44:  }
45:}

 The dimensions (width x height) of the actual image are 67 by 34
 The dimensions (width x height) of the expected image are 71 by 33

 The actual image, PNG and Base 64 encoded is

 iVBORw0KGgoAAAANSUhEUgAAAFcAAAA2CAYAAACyYAWTAAACsklEQVR42u2YyytFURTGPUqepTwmZCIipZABhgaUGBhTklIM/AOUgTyKKO+BGJCJEgMMKEOPiBKFIlJEyZtSH6vVaTv3Xvd6ZHDc76vTPfusdfdq/+7e397nBoD6MwUQAeESLkW4hEu4FOESLuFShEu4hEsRLuESLkW4hEu4FOESLkW4hOuqjg5gaclnWlMTsLbmD3Cjot4rvJcIDASys4Gurp/3lZEBtLfbug0PBwoLgelpkxYXB4yO+gPcyEigrw84PdUpJZB3dn4NV7rt7QV2d4GyMiA2Fnh48Ee4g4N6v7mp021uzsS3toDcXJ2CRUXA5aU9lpOj07S8HEhMtMG1um1pAWJigNdXd7hpaXYnke5mZ32Xdg7c/n5gfR0oLVUK1hQTiVXU1gIrK0BmJtDZaWJZWUBxMbCxAXR3AyEhNrh1dUBjI5CQAIyMeLaF4GBgZsbEoqOB8XHfpZ0DV6xAZmx8PHB0ZGInJ/pcRr+6ClRVASUlGru40O993JlcbCElRT9TU+1pX4HrrbSz4DY3AwMDCmtx0cTGxnSEQsm6Cgo0Njmp+c/Pn8IVW3h6AurrFeLZ2dfheivtTM+VJZ6UBNzcaHt+Xkcoy95Vy8saOzw0z8RAPXju/r6mDg97hjs1pfdXV0BoqML1VtqZcI+PgYgIoLpa2/f3apiyHgXi46MZ7cuL5tbUAHt7QEOD0vgAt60NODgAKis1JCcHV7jJyerN29tAfr7mCVxvpZ11zh0aMu2eHh3hwoK2ZTeRNSkWIBtWXp7JbW0FgoL0qqgA0tPdzrkSkh1f3i8sibVbcOUUKKeBsDD9McQWJiZ8l/5fb2jn58DdnftzmWLX17/q+vZWvfm7pfnfAv9bIFyKcAmXIlzCJVyKcAmXcCnCJVzCpQiXcAmXIlzCJVyKcB2lN+t94EgHrIxbAAAAAElFTkSuQmCC

 The expected image, PNG and Base 64 encoded is

 iVBORw0KGgoAAAANSUhEUgAAAFsAAAA1CAYAAAAuyJezAAACPklEQVR42u2XzStEURjGBxtkgw0rGgssfSSixCgLZWVpYydLa5IoKyULrP0BytK/oJQaJIpEsUD5ZsFPb6fbuZjurQl1m+ep09z3vqc57/zOO889N4X0b0oJgWALtiTYgi0JtmALtiTYgi0JtmALtiTYgi0JtmBLgi3Ygi0JtmD/Q7Wpv5yeYNj2S4NRXQ1DQ3B09Kuww0tUVEB/PxwcFCrsQNfXMD0NXV2/DjvQ3R3Mz0NbW6HDNt3fQ1mZj9/fYW4O6uuhshLGxuDx0eff3mBiAqqqoKEBVlYiYZseHqC0NHc+F/jwvbhykgP75gampqCjw99bXIRMBk5O4PYWRkdhctLnZ2ZgYAAuLuD8HPr6ImHbXi4sQE9PfrDjykmOZ9uorYXTU59vaoLDQx9fXUFdnY+tm/f3fZzNRnp24Ns7O/nBjisnGZ398QHHx67lNjZ83iylpMSN4mIoKnKfgcwPXl58bNcRnf387Lqztzc/2HHlJMuzz86gpsb9302NjV87/buss8NHi729WM82jw0/FsJ527unJx9fXn7Nx5WTLNimkRFYW3PXS0vOkw2oUdjehuFhP9dOL4ODzrNt2NyYzravbGnJne/uhtlZtyHmy7ZUOB9XTvJgb21Ba6t//C8vO7MsL4f2dtjc9HNfX2F83J1G0mlYXY30bOtoA7q7m7sEs/zOTrdUczOsr/88jUSVo9d1va5Lgi3Ygi0JtmBLgi3Ygi0Egi3YkmALtiTYgi3YkmALtiTYgl3A+gTbCPZGMfcurwAAAABJRU5ErkJggg==

 To convert a base64 string to a viewable picture, use

 Base64[Png](string).toPicture[Algebra, Drawing]

    at munit.FunSuite.assert(FunSuite.scala:11)
    at doodle.golden.Golden.imageDiff(Golden.scala:127)
    at doodle.golden.Golden.imageDiff$(Golden.scala:13)
    at doodle.golden.Text.imageDiff(Text.scala:11)
    at doodle.golden.GoldenPicture.assertGoldenPicture(GoldenPicture.scala:28)
    at doodle.golden.GoldenPicture.assertGoldenPicture$(GoldenPicture.scala:11)
    at doodle.golden.Text.assertGoldenPicture(Text.scala:11)
    at doodle.golden.GoldenPicture.testPicture$$anonfun$1(GoldenPicture.scala:43)
[error] Failed: Total 13, Failed 1, Errors 0, Passed 12
[error] Failed tests:
[error]     doodle.golden.Text
noelwelsh commented 3 years ago

Thanks. It looks like a small difference in font rendering. I've attempted to make the test more robust to this difference. If this doesn't work I'll mark the test as Linux only.

SethTisue commented 3 years ago

still failing

``` ==> X doodle.golden.Text.text-color 0.039s munit.FailException: /Users/tisue/doodle/golden/src/test/scala/doodle/golden/Text.scala:45 44: .font(Font.defaultSerif.size(24).family("Arial")) 45: } 46:} The dimensions (width x height) of the actual image are 110 by 47 The dimensions (width x height) of the expected image are 113 by 47 The actual image, PNG and Base 64 encoded is iVBORw0KGgoAAAANSUhEUgAAAIIAAABDCAYAAABQt93sAAAFRUlEQVR42u2aaahOTxzHr31J9rJvJWSJN/Z9TbaiSF6RJJGllAgvEEUkXiB7IckSskXZ9yWFsm8hsiR7Wb58m8acc895znPvuc/zd+/5f781L+7Mb+acOfOZmd/vd58cSNIf5egTSAJBEgiSQJAEgiQQJIEgCQRJIEgCQRIIkkCQBIIkECSBIAkESSBIAkESCJJAkASCJBAkgSAJBClxIDx7BixenLosWQKsWgVs3w7cvfvfzfjIEfP8HTv+Vt26lfo1V64Ejh4Fnj9PPSTbabttm0AI6vjxP6Pn5L0MGwZ8+ZL9GY8ebZ7Xvfvfqg0b8vaKTZoAd+4Ehxw3zrS3by8QokFo3Rro399fevcGmjUDSpZ0dm3bAl+//lMQ6tUDmjY1pXFjoE4d/ytWrQpcuyYQ4oGwf39quwcPgM6dne2hQ/8UhJs3g12+fQNmzwaKFTM2Y8cKhMyDYP0Jazt1aqEDwapHD2NTsybw65dAyDwIP34A5coZ2z59gu2PHplt2KYNUKYMUL8+MGRI9Lh0QnnSVKwIVKkCDBoEHDxYIBDsgleqBPz8GQ3CihVAv37AtGnhYy1bZtqnT8/MdJMBwrFjznbChOCCcjFTeXC0925Pau3acNvixc0qxgDh0yegUSNjM3x4+qsh3SlheezSpeDTLfogvHoFbNninzmh8G6NsmVNfcOGwO7dwMuXwJUrJsqwfTZudH3OnwdKlzb17doBZ88Cjx+bUNWeOhEgcKhTp0w5edJEm6tXOwjI0blz2QEhznSLFgjVqgENGvhLhQpB3KdM8Y/Brcd6gvL0qb+N26JXL3dp29Cza1dTV7duMALZsyctCOlK7oghkyDEmW5y8giMzXiX79oVHMPCksqB9K7g5cvma9nThQmrMLVqVSAQWrQALlzIDgj5nW7RA2HyZGDzZmDTJuNFDRjg2jp0AJ48CfZ/8cLZMKAfPDhY+BWtDdN6DEXt30z3hYleWAQIZ84A7965cv++mcr8+e7GoavBGyiTIMSZbjJ8hKVLXTsTS+/f+9u5IvnJTC5cCBw44P6+ejX8vebOjR01HD7s7EaMyCwIcaabHGdx4kRnw2wjw0grrog3t0DPKarcvg3cuOH6nDgR/kzGazFB4M1Tvbq7IvICApOlYRo40A9CnOkmB4TPn00u19otWuRP5/EMZv2CBeH96TFxKz18CHz/bkAqX970WbMmvM/QobFBYO6gcmVjR580CgSGeayjSxImO20LQpzpJiuPcPGi+wKMne7dc23Nm5t6Jvw/fgz2ZZRhn7Fvn6nr1MltWW/Wh6IvYi/6GCCQLWs3Y0Y0CDNnmjo6gPQ1vOJuLlUq6CzGmW6yEko8C61tz57h4R4zjvSo7PZh/sECxKDbLjq9KNuHFzFtrTdGxzRN+Lh8ObB3rys7dwLr1pnQrkQJY8Ns3/Xr0SBs3erGHDkSePPGvCK55+vaNi8IcaabLBCIP3ML1n79etc2apQ/1OS28eYgeBXkzvDMmePaaduypduCBcwjcDFony585OLVqhWe1GSx082dWYwz3cINAtNy9u3pzaeT1+Pn/wa8ZyNTabVrB1elb9/wgJpe3axZ/i/IK2HMGGDevMDJw6g2Ks3Bo5pJykmT/DeX1fjxxrZjR389k0J8jN3JLDVqmFPHZsG7dQuOl9/pFm4QsqHXrw1gly4Bb9+mtydM/HJMM2f7dw4R+vABOH06/z/Eyu90/z8gSAJBEgiSQJAEgiQQJIEgCQRJIEgCQRIIkkCQBIIkECSBIAkESSBIAkESCJJAkASCJBAkgSAlRL8B4nyc/ZY6dt8AAAAASUVORK5CYII= The expected image, PNG and Base 64 encoded is iVBORw0KGgoAAAANSUhEUgAAAIUAAABDCAYAAACya8aVAAAEeUlEQVR42u2ZS0hUURjH7V2aWlFZkGToRsGIICl6EEQECQVCFEGbKNwo0iIQIyqICDe6DCLQKFoU1CZCEaWiRUQUPegNQQ9qUVqalYX/+vN1mel6JmeOMzFX/j+4IPd853rPnN95fTcHQoTI0U8gJIWQFEJSCEkhJIWQFEJSCEkhJIWQFEJSCEkhJIWQFEJSCCEphKQQkkJICiEphKQQkkJICiEphKT4/fScxNekScCsWcDy5cDhw8DAwH9s9Z93SOGV+brz5gFr1wKtrcDQUMqPlhSjShG+KiuBz5+zVorwtXo18O2bpEjrj4+fP4E3b4BTp4A5cyyuqSlrpAjDmeHdO+DCBWDRIos5ckRSpFeKeC5ftrjS0qyVIp6ensSvKynSJUV/v8VNm+Yuf/kS2LsXKC4Gpk4FFiwAdu4EHj5M/Mxnz6xOSQkwfTpQXg6cOJEWKQYHLWbGjOTrj/bcf5X7ND/6UnBOZhw7MMy1a0BBgXthZ2dzlgnT1QXk57vrtLSMWYobNyxm2bLMS+HT/OhK8eUL8OgR0NwM5OVZ3IEDf8d8+AAUFVlZbS3w4IHNKlevArt2ARMnAoWFwNu3sTq9vXZMYJ3t24HHj4Hv34E7d4CNGy3eQ4ofP4D374Hz523EMqa9PbNS+DR/fJ0+1q8Hvn79u/6xY1a2b5/7+Q0NI2U6ftzubd7s7tlVq8Z8+mBnHDqU2jjwkcKn+dGXIjcXWLcOOHnSOiwMz32Me/HC/fz79618xYrYPSYSeI9LiItLl8YsxYQJwJYtNgllUgqf5kdv+RgetnNddbXdr6gAXr9OXH/27OR6iUmwgPnz7R6XERf8/57LB6dzTt3c5DFm7lzg6dPMSeHT/OjuKT59ApYssbKqqsTpQaYRk53PA6ZMsXvMg7hg745xo0n2749tWzIlhU/zo73R5LaarWF5Y6M7Jth2p7KTCnZmHNYu+vrSIgU3nYzjv0tFCk6WYZjIddXzaX70j6R1dTHVb94cWc7vIiy/eDH5/8k9CuskOqt1d6dFio8fLW7mzOTqB6Pe9Ynn7l13PZ/mR18KLiMLF1rMypUjhxHT3sF3EdcSc/q0ldfUxO4FeQhuOF1LCE8laZCCH8WCbyDJ1Of+g/fpZJj6enc9n+aPj+TVmTOxOP4dDzehQV5hwwabTXhs5S+7Z48NP+4huA0P4FBcvNjqbNpkw5B5CuZEtm2L9Y6HFOwYngSOHrXMIuPOnk2u/tatdr+sDLh+3Zrx/LkJwcyoq55P88dPRjOY8vmliUmteLgMxP9q4cuVQbp1K5bACp8lz51Ly1dSXuyYZJvMV2IG0vVKbW2J6/k0f3xIce8eMHmyxR48OLL8yRNg925LJXJocPgwO9nZmfiZr17ZMOQph72xdKml00d5r3/lJphW4UjfsQO4ciX1Jt++bfkNHjWZxF2zBujoGL2eT/OzVwoRSSSFkBRCUghJISSFkBRCUghJISSFkBRCUghJISSFkBRCUghJIYSkEJJCSAohKYSkEJJCSAqRFfwCH5CFrQnIlQQAAAAASUVORK5CYII= To convert a base64 string to a viewable picture, use Base64[Png](string).toPicture[Algebra, Drawing] at munit.FunSuite.assert(FunSuite.scala:11) at doodle.golden.Golden.imageDiff(Golden.scala:127) at doodle.golden.Golden.imageDiff$(Golden.scala:13) at doodle.golden.Text.imageDiff(Text.scala:11) at doodle.golden.GoldenPicture.assertGoldenPicture(GoldenPicture.scala:28) at doodle.golden.GoldenPicture.assertGoldenPicture$(GoldenPicture.scala:11) at doodle.golden.Text.assertGoldenPicture(Text.scala:11) at doodle.golden.GoldenPicture.testPicture ```
noelwelsh commented 3 years ago

Ok. I've made my final attempt. Let me know if it works...

SethTisue commented 3 years ago 
[info] Passed: Total 13, Failed 0, Errors 0, Passed 13

🥇

noelwelsh commented 3 years ago

:joy_cat: