Support `anything` in nested data structures

[pjlegato]

Would it be possible to extend anything support so it can be used in maps nested within a sequence?

We often have to check a function that returns a sequence of maps (e.g. JDBC queries that return a sequence of map rows). In many cases, we don't care about the exact value of some map value, such as autogenerated SERIAL IDs.

This test passes:

(fact "this passes"   {:x 1, :y 1} => (just {:x anything, :y anything}))

But this version, where the map is wrapped in a vector, fails:

(fact "this fails"   [{:x 1, :y 1}] => (just [{:x anything, :y anything}]))

[philomates]

An engineer at my company developed a custom Midje checker that does exactly this and a bit more, including what you suggested in https://github.com/marick/Midje/pull/403#issuecomment-333672673.

We've are a bit reluctant to release it into the midje code-base as is; there are some minor rough edges to clean up. He is currently on vacation, but I've asked him for the go-ahead to post a gist with the checker code so you can reuse it. I'll keep you posted

Looking forward, we have hopes to clean up the checker implementation and add it to the core midje code.

[marick]

May not be relevant, but I originally developed https://github.com/marick/structural-typing with the idea it would replace much of Midje's checking infrastructure. It was partly based on my years-too-late belief that testing frameworks erred by starting with equality as the fundamental operation. Instead, for maintainable tests of non-trivial data, the fundamental operation is applying a predicate to the actual value. Midje does that by letting => take a predicate as the right-hand side, but that produces crummy error messages.

The structural-typing library was an attempt to provide predicates that both produce truth values and also provide explanations -- good explanations -- when something goes wrong. It got the wind taken out of its sails when clojure.spec was unveiled, and then I exited the Clojure ecosystem. But I still think replacing Midje's ad-hoc extended-equality with something based on this library would at worst be a useful experiment.

[pjlegato]

@marick I'm sorry to hear you've exited the Clojure ecosystem. Does this mean that Midje is dead and we should not use it for new projects?

[marick]

There are some people from Brazil who are supporting Midje, honestly with more company support than I had. It might be time for them to say Midje is now their project.

That's not advice about new projects. But I'd say Midje is safer to rely on than it was during the last two years of my involvement. It's probably safe to rely on it continuing to work.

For new projects, the decision comes down to this:

1. It seems clojure.test will continue to be a pretty minimal test framework, but...
2. ... with increasingly more third-party tools and built-in support from tools like Leiningen.

Then we could ask:

1. how long will it take the clojure.test ecosystem to include the things you value about Midje?
2. will the clojure.test ecosystem will outstrip Midje in what it allows?

My personal opinion is:

1. the clojure.test ecosystem is hampered because core clojure people neither appreciate nor understand modern testing. So I doubt clojure.test is the route to really great testing.

2. Midje's stability as Clojure versions change shouldn't be a major worry. Midje was first written for Clojure 1.2. Clojure version changes since then required very few Midje changes. Given Hickey's emphasis on backwards compatibility, I expect that will continue to be true.

3. But there are important secondary issues. For example, Midje is weak at describing clearly how a big expected value differs from an actual value. I wanted to improve that. It was a big part of the reason I wrote the structural typing library. My goal was to integrate it into Midje's failure reporting. I gave up on it because clojure.spec, despite having much less of an emphasis on error messages, was the variant everyone would use.

Getting rid of Midje's deficiencies will probably require community involvement. The question of "should we keep using Midje" might turn into "do we value the Midje approach enough to add to it?"

[philomates]

Hey @pjlegato, as @marick mentioned, a few engineers at Nubank (including myself) have taken over supporting Midje. The test infrastructure for our 100+ engineering team is powered by Midje, so we are keen to see that it continues to run smoothly.

We don't currently have plans for any grand new features, but have invested time fixing a few bugs and making Midje more user-friendly where possible. We've published a few alpha releases that we've been using internally, and are hoping to cut a normal release in the coming weeks.

Looking forward I hope to invest some time profiling Midje to see if startup times can be improved. My colleague @rafaeldff has been playing around with better checkers for nested structures and built-in generative testing constructs; hopefully we can get these released in the future too. That said, it would be great to hear other peoples ideas!

Lastly, thanks Marick for continuing to offer background context and insights on PRs and issues; it has proven very valuable as we continue to improve Midje

[philomates]

To follow up on this, we've released a new library for checking nested data-structures that attempts to solve the problem described here: https://github.com/nubank/matcher-combinators

@pjlegato as you mentioned, the following doesn't work with midje checkers:

(fact "this fails"   [{:x 1, :y 1}] => (just [{:x anything, :y anything}]))

With matcher-combinators this can be solved:

(require '[midje.sweet :refer :all]
         '[matcher-combinators.matchers :refer [equals embeds]]
         '[matcher-combinators.midje :refer [match]])
(fact "the nested map can have extra keys we don't ignore during matching"
  [{:x 1, :y 1, :extra :ignore-me}] => (match [{:x anything, :y anything}]))
(fact "the nested map must have exactly the keys specified"
  [{:x 1, :y 1}] => (match [(equals {:x anything, :y anything})]))

In the example above still need to wrap the nested map with the strict equals matcher if you want to check the map exactly because the default matcher for maps is a looser embeds (similar to midje's contains)

matcher-combinators also give you nice diff messages when there are mismatches in results just like you asked about in https://github.com/marick/Midje/pull/403#issuecomment-333672673