Implement full diff mode in string comparison

[tribbloid]

I have open a StackOverflow bounty for the possibility of this feature:

https://stackoverflow.com/questions/69772985/in-scalatest-how-to-show-full-diff-in-test-result

The idea is to delegate the entire value and groundtruth as the error information. This delegates the diff display to third-party IDE (e.g. IntelliJ IDEA) and makes approval-driven testing much easier.

Its implementation should be much simpler than status quo. If it is missing I'll be interested in submitting a PR, just let me know where to add the extra configuration

[cheeseng]

@tribbloid Unfortunately I don't think there's a flag currently, as far as I know the related code are:

https://github.com/scalatest/scalatest/blob/3.2.x-new/jvm/scalactic/src/main/scala/org/scalactic/Differ.scala#L54 https://github.com/scalatest/scalatest/blob/3.2.x-new/jvm/scalactic/src/main/scala/org/scalactic/Prettifier.scala#L338

We have quite a long list of options currently supported:

https://www.scalatest.org/scaladoc/3.2.10/org/scalatest/tools/Runner$.html

I am not sure what will be good character for the new option, @bvenners do you think it will be a nice new feature to have?

[yawaramin]

@tribbloid what kind of output exactly are you aiming for? Should it be more like a git diff?

[yawaramin]

Bump. I have a working implementation that's good enough to use internally, but maybe can be polished up a bit. Let me give a little context on this.

About a decade ago, @japgolly asked about how to make test failure output easier to read. There was an interim solution, with some discussion about possible more complete solutions.

I came late to the party and suggested snapshot testing/approval testing/'golden master' testing. Since then, for about the past year now I've been successfully using a home-made solution for snapshot testing and it's been working pretty nicely. Recently I made it a bit more general-purpose. It looks like this:

// Foo.scala - live code to be tested

case class Foo(id: Int, name: String):
  def toMap: Map[String, String] = Map("id" -> id.toString, "name" -> name)

// FooSpec.scala

class FooSpec extends RefSpec:
  object `A Foo should`:
    def `convert to a map` =
      assertSnapshot("Foo", "convert to a map"):
        Foo(1, "Bob")
          .toMap
          .mkString("\n") // Important to use newlines for diffing

On the first run, this writes a file src/test/snapshots/Foo/convert_to_a_map with the contents:

id -> 1
name -> Bob

This file is checked in to the repo along with the test.

On subsequent runs, the assertion checks the output of Foo(1, "Bob").toMap.mkString("\n") against the stored snapshot, and if they match, succeeds. If they don't match (comparison as UTF-8 strings), then it fails and prints out a diff.

Here is where it gets interesting. Instead of trying to implement diffing, which is a 'diff'icult job, we just use git which we assume exists on the system. This is a pretty reasonable assumption in any dev and CI environment. So we print out the git diff output:

📸 test failed:
diff --git a/src/test/resources/snapshots/Foo/convert_to_a_map b/src/test/resources/snapshots/Foo/convert_to_a_map
index 1234567...2345678 100644
--- a/src/test/resources/snapshots/Foo/convert_to_a_map
+++ b/src/test/resources/snapshots/Foo/convert_to_a_map
@@ ...
 id -> 1
- name -> Bob
+ nomen -> Bob

💬 If the new version is correct, delete the snapshot file and rerun the test.
[info] FooSpec:
[info] A Foo should
[info] - convert to a map *** FAILED ***
[info]   Foo_-_convert_to_a_map (Snapshot.scala:...)
...

Note, we don't try to provide prettifiers, we leave it to the user to provide diffable strings. This greatly simplifies the implementation. I'm sure it can be improved, but it works pretty well in practice.

If there is interest, I can submit a PR.