--tag option precludes any tests from running

[avescodes]

We have a project utilizing speclj, but can't seem to get the "tags" feature working properly.

When we run lein spec alone, our entire suite of ~170 examples execute.

When we run any invocations of lein spec with a --tag option, however, 0 examples will run. (In our suite, the majority of our tests are (tags :unit), and select few are (tags :integration))

For example, all of these invocations result in 0 examples, 0 failures:

• lein spec --tag integration - Just integration
• lein spec --tag "~integration" - Everything but integration
• lein spec --tag unit - Only unit
• lein spec --tag :unit

The only thing particularly different about our project.clj is that we utilize multiple :test-paths and :source-paths.

Do you have any ideas what might be going wrong? Can I provide you any other information to help diagnose our issue?

[arlandism]

I think I've narrowed this down. It boils down to the logic in src/clj/leiningen/spec.clj in the with-paths function. The function will be truthy anytime it senses an argument with a dash, and thus won't ever add the test paths to the argument list. I would work on a patch, but the context surrounding the with-paths function eludes me; specifically, why does this check need to be performed at all?

[avescodes]

Looking at the commit that that code originated from (d16d38940), it seems like the intent was to allow overriding test-paths. I'd propose (and may submit a PR for in the near future), that the test be tightened to check specifically for the presence of paths in the command-line arguments.

@slagyr: is this something you'd be amenable too? How should replacement test-paths be specified on the command-line?

[avescodes]

Hmm, looking at the code in cli.clj and the like makes me thing it's not going to be quite so easy to fix this up. Since some command-line options are flags, and others take arguments, it is difficult to tell when a user means to provide a path, and when they are just passing an option.

We could resort to some crazy hacks or hard-coded tests to check options, but I'm not super enthused about that. Any comment @slagyr on the direction you'd like to take? I'd gladly help implement, but I don't want to start something without your input.

[trptcolin]

I don't use tags much, but I noticed that unlike with lein test, speclj tags get applied to the describe or context, not to individual tests (it).

@rkneufeld any chance your (tags :unit)/(tags :integration) declarations are directly inside it blocks? I can reproduce this behavior if I do that.

[avescodes]

Unfortunately that is not the case. We're looking at describes with nested tags. The Lein bootstrapping issue above remains the most likely candidate.

On Sep 19, 2014, at 3:26 PM, Colin Jones notifications@github.com wrote:

I don't use tags much, but I noticed that unlike with lein test, speclj tags get applied to the describe or context, not to individual tests (it).

@rkneufeld any chance your (tags :unit)/(tags :integration) declarations are directly inside it blocks? I can reproduce this behavior if I do that.

— Reply to this email directly or view it on GitHub.

[trptcolin]

Ah, OK, I get it now (h/t to @arlandism for setting me straight). Both of my :test-paths were underneath the spec directory, so using the default spec directory worked fine.

I'm thinking a reasonable approach might be to pass the project's :test-paths down via a new option (perhaps --test-paths) and have speclj.cli sort out whether it should use those (by default) or explicit command line arguments. I've started working on a patch for this. Input definitely welcome if that sounds like a good or bad idea!

[trptcolin]

BTW, thanks for uncovering this issue - it affects way more than just --tags. e.g. lein spec -f d for formatting will omit any directories outside of spec.

[trptcolin]

Alright, my repro case is good on master - let me know if this works for y'all?

[avescodes]

This seems to do the trick. Thanks!