Closed SoftwareDreamer closed 10 years ago
darrencauthon
commented
11 years ago I'm not sure I understand... could you give a little bit of code to make it a little more apparent?
I've used SpecFlow to work with an app's state, even altering IoC registrations and the like, but I haven't come across situations like these. I'm not sure I understand.
SoftwareDreamer
commented
11 years ago Ok, assume you have a shared steps assembly. Each steps file in there will be specific to some feature, let's say you have a complex business system, an you have all kind of things, like users, workflows, etc. Now there is the users step file with things like "given the following users exist" etc. These weould just take the table, convert it to a list of users, and add them to a Teststate resolved via e.g. IObjectContainer.Resolve
The problem: this teststate is only known in this "root" assembly, and there is no way to force SpecFlow to go through your Stepfile in the "root" assembly first. What then happens: it uses some Given step from the shared steps assembly, before the "root" testassembly had a chance to intervene, and the concrete TestState isn't registered yet, to attach the users to it -> Exception. [BeforeFeature] doesn't work exclusively, as the IObjectContainer isn't available on it (bound to ScenarioContext). So the solution right now for me is: save the TestState class type on a ServiceLocator-like static class in [BeforeFeature], and then, in every constructor of every shared and nonshared step file, register the TestState class for every interface that it is derived from, via this ServiceLacator class which now knows the TestState type, if not done yet. But that is suboptimal, as it is more verbose than needed, involves locking, and is just harder to remember (one call in constructor.. easy.. 5 lines of code.. harder).
It would be nice if there would be some determinism/composition root in SpecFlow. Right now, there seems no defined point where the SpecFlow IOC container is first available, when given multiple Stepfiles in several assemblies. An optional additional class attribute might be a solution. In my opinion, it should enforce calling the Stepsfile in the "root" assembly first by default, sufficient for me (just one steps file), but there could be multiple steps files there, so certainly not a complete solution.
I think SpecFlow should provide a composition root for IOC containers.. it right now doesn't seem to have one. Am I mistaken?
darrencauthon
commented
11 years ago I think you lost me in the first sentence with this:
Now there is the users step file with things like "given the following users exist" etc. These weould just take the table, convert it to a list of users, and add them to a Teststate resolved via e.g. IObjectContainer.Resolve<IUsersTestState>.Unless your system calls users "IUserTestState," then you are probably not actually testing your system. "Given the following users exist" means exactly that -- those users exist. Not test users, not implementations of some interface, but exactly what it says: Users. It should take the table, convert the text to Users, and then save those Users in the database.
I've written many SpecFlow tests over time, and I'll tell you: I had no idea there was an "IObjectContainer" or this sort of thing. And knowing it now, I'd recommend never using it or making up composition roots in your system or anything of the sort. If the tests are complex, or if they have to be aware of the order in which they are run, they will break down. Keep it simple.
SoftwareDreamer
commented
11 years ago Don't be that guy.. don't tell others what "should" be done. ;-)
Hrm, I think there is a chance you misunderstand. I am testing my system, but I replace my database with a fake in-memory database, it's not called Integrationtest. I know a lot of people do IntegrationTests/BATs, UI tests via SpecFlow, etc. Trust me, it works for small systems, not the big legacy system I have to test, if there are no test below. It's a mess. It is big. I did IntegrationTests of all kinds for years, trust me, on a certain size you begin to hate any out-of-process dependencies. You really want to have the quality of unit tests (fast, stable, trustworthy) with the expressiveness of SpecFlow. Some call them Behaviortests, but as the community can't get clear terms for things like process dependencies vs testframework vs shape of code tested, it is a muddy topic. There is no clear term. Only thing that is sure: not Integrationtests. Up-/Downmigrating a system with huge databases, etc.. Absolutely not neccessary to test business logic. Absolutely awefull to do, there is a testpyramid for a reason.. I could go on.
But that's not even the point here. IObjectContainer exists for a reason. IOC exists for a reason, a composition root in MVC exist for a reason, such a thing can be usefull in SpecFlow, for larger usecases.
Do you know what's the state of the SpecFlow project? Are you the maintainer? What's with the 2 guys who created SpecFlow, and probably the lightweight IOC container to begin with?
SoftwareDreamer
commented
11 years ago just to make clear: IntegrationTests are good and usefull and neccessary, when kept to a small code surface. A few big integrationtest (system tests) might also be usefull (equals "more time saved then spent in maintaining/fixing tests"), but without a broad unittest/componenttest base beneath, it's useless. And I want to create that, but for that I need reusable test infrastructure (The Given steps). And that's what it's about..
darrencauthon
commented
11 years ago But SpecFlow is an integration test framework, just like the Cucumber framework that inspired it.
And it's best used to cover a few larger stories, with smaller unit tests that cover the core components. It's not meant to be used for unit tests or business logic, at least at the level that you seem to be using it to do.
If you want to get a manual as to how to use SpecFlow, I'd suggest checking out the RSpec Book. That book talks how unit testing and integration testing goes hand-in-hand, and almost everything it describes about Cucumber can be ported directly to SpecFlow.
I won't argue with you if you don't want me to talk about how it "should" be used, but I'll have to disagree with how you say SpecFlow "should" accomodate this request. Like in a previous request, I believe SpecFlow should start randomly firing and running the order of these steps, as everybody seems to have an opinion on what order should be applied.
darrencauthon
commented
11 years ago But as for the state of the SpecFlow project: It's stable, and it is always kept up-to-date with the new versions of .Net. It's not active in that it has tons of changes made to it every day, but I think at this point there's not a ton of features that I can imagine need to be put in it. There are periods where all of the pull requests are sucked in and a new release is made, but that happens every few months, not every few weeks.
I'm not the maintainer, but I'm a contributor and a long-time user of the app.
The IObjectContainer object exists, but it may not be for the reason you think it should be used for. MVC needs IoC, and I always used it in every app I've ever written, but SpecFlow/Cucumber are frameworks meant to do something else.
SoftwareDreamer
commented
11 years ago Cucumber was designer to express stories. Business usecases. Since when are business usecases tied to the database? Or a UI? Sure, you could argue that it's not a complete test. But it's not about that, at some project size,. it's about test maintainability. Can't I express in words a service that takes a mail, and emits a message to the message queue? I can. No Ui. Also Integrationpoints, but I'd rather test them separately, and use only very few system tests.
But not everyone agrees, sure, many see 1 class = unit test, and, as I noted, there is a missing understanding of process boundaries vs. code complexity (class, multiclass, component). Here you do the same, arguing that integrationtests are story-based, and as SpecFlow describes stories (or can do that at leats well), it is an IntegrationTest framework. It is not. Maybe designed by someone who doesn't distinct between expressiveness and process boundaries, who uses the word Integrationtest to express "reasonably complex test to be meaningfull for business".. You mix the notion of business-relevance (should I use Nunit or SpecFlow? How do I express my tests?) which is in part tied to code/logic complexity (business isn't usually interested in a single small class) with the notion of inter-process dependencies, like accessing database, system clock, message queue, etc. Unfortunately a lot are doing this, and it is really hurtfull for testing, as people "think" SpecFlow can only and should only be used for IntegrationTests (which I define strictly as process-boundary-breaking). Luckily, there are some who also agree that it should be used for business logic testing (expressing stories).
About the ordering: SpecFlow, by design, as it uses Nunit, doesn't guarantee the ordering of the tests, anyway, as NUnit doesn't guarantee ordering. An IOC container also is not about ordering, it is about untying components. Which again removes ordering issues. So my request is the complete opposite of what you seem to think it is.
Anyway, I give up at that point, but thank you for your efforts in reading my walls of text. :-)
SoftwareDreamer
commented
11 years ago Or to sum up, my point is one of the points in the "busting the myths" talk by Gojo Adzic "acceptance tests have to be end-to-end tests", or called Acceptegration in this talk http://gojko.net/2012/06/18/bdd-busting-the-myths/
Please look at it, it's at around minute 25. And I'm not saying it because he said it (I didn't know the guy when I had to learn testing the hard way).
SpecFlow is an Acceptancetest framework. It is not an Integrationtest framework. Nonono.. Because there are no IntegrationTest frameworks (well.. MSTest is the most Integrationtest friendly framwork out there.. ordered test, heavy multi-data capabilities via ODBC; etc.).
darrencauthon
commented
11 years ago I don't know who you are arguing with... I have not made the claims that you are arguing again. As just one minor example, I'm the guy who has pushed every SpecFlow user away from UI tests.
With SpecFlow, I have tested business logic against the UI and without the UI. I have abstracted the database out and I have kept it in. I have used it for unit tests, "acceptance" tests, and "integration" tests. I have used shared assemblies and simpler ones. I've created test suites that have been so bad, I'd have to delete the entire assembly. I don't think there's much you could suggest to me with SpecFlow that I haven't tried. And after so many failures and successes, I've found the way to get the most bang for my testing buck, and where not to take SpecFlow.
It all breaks down to complexity for me, and that's where I believe you are jumping off the rails. Two main reasons you've described:
1.) Abstracting out the database. The biggest reason I believe you should keep the database in your SpecFlow tests is that having a database is the natural state of your application. When code runs, it probably hits the database. Taking the database out is inherently more complex than just letting it stay. You have to do the additional work to get things set up properly in your tests, when if you just fired up your app with the database it would work.
I've found the speed boost from abstracting it out to be a relative non-issue, as database run on a local computer is pretty fast at making the basic CRUD calls you need for testing.
Plus, having a database imposes restrictions on your application that abstracted stores do not. For example, string columns in a table are usually restricted to a certain length, but an in-memory string is not. Or, the database might not allow certain relationships to be stored without other records happening first, but the in-memory stores will work just fine.
2.) Shared step assembly. This is another source of complexity that I've found people get into too fast. They have these pre-conceived notions about what step files need to go where, and they start organizing things before there's a demand to do so. It gets worse when the assemblies start to tie to each other. Devs start clamoring for certain steps to be run before others, they start creating new schemes for organizing the steps, and before you know it it's too late and nothing works.
The simpler way to do it is to keep SpecFlow as a story-teller, expressing just enough stories to drive development and keep the application development tied to reality.
darrencauthon
commented
11 years ago And one more note in case others read this later:
I agree with what I heard in the video, but I disagree with your interpretation. The overall point about finding the sweet spot with what to target on the tests is important, as some think you have to go end-to-end. I think there has to be a good reason to take something out, though. With the UI, that's easy -- it's a huge pain to test against the UI. The database is a different story, though.
I also think the verbose nature of Gherkin makes it more difficult to test business logic with it. Business logic can sometimes require an incredibly amount of setup and precision, where there could be many different paths down a decision tree. Tracing through each possible enumeration can be difficult with Gherkin, as its "flat" and expressive nature can create scenarios that are either hard to read or too vague that don't get the story across. Code-based tests with tools like MSpec, RSpec, etc. allow the tests to be broken down in hierarchies that produce expressive results and be easier to maintain later.
But as soon as you get into the code-based tests, you lose that connection to reality that you get with SpecFlow forcing you to tie your logic to English. Since it's hard to cover everything with SpecFlow... don't!
SoftwareDreamer
commented
11 years ago Ok I promise myself and the rest of the sorry readers that this will be my last wall of text.
"Since it's hard to cover everything with SpecFlow... don't! " I don't, we have unit tests, too. They cover ~3% of the codebase, a pretty typical situation in several older software companies, unfortunately. But for anything more complex.. well, I have yet to encounter a scenario which I can't express in rel. short terms in SpecFlow, given certain "invisible", yet spectested, defaults.
To move things into a better context: I need SpecFlow, because of its abstraction layer. I have a bigger picture in mind: a website which holds the specs by tags (easily seachable), which lets you execute and play with the tests. Which changes the way devs and business communicate, as well as doc team and tech support. Which increases the number of testing stakeholders in the company. It's a bigger picture, as I said. A usecase for a bigger company, where devs make up just 30% of employees, or lower. Where business just doesn't know what testing does. Etc. etc.
Anyway, to your points.. "I've found the speed boost from abstracting it out to be a relative non-issue" Not in our case. We have a very inefficient architecture, I can't fix that. Each test would take 2 seconds, because a cache is loaded first, and a lot of non-need Sql qeueries run. Spinup-time has unfortunately never been a top priority.
"Plus, having a database imposes restrictions on your application that abstracted stores do not. For example, string columns in a table are usually restricted to a certain length, but an in-memory string is not. Or, the database might not allow certain relationships to be stored without other records happening first, but the in-memory stores will work just fine." Very very good points. it's just that this is the domain of integrationtests. I don't care, for three reasons: a) there are seperate small number of integrationtests which run the database-facing API layer against the DB, to check if the constraints fit b) it is checked at runtime by our DB Access layer, anyway (the tests for that are there); it never happened to us, compared to the hundreds of businessrule-related bugs we have in our bugtracking system c) I can, if neccessary, check that, as I rewire the calls anyway, and the T4 templates allow me to convert the DB types to attributes which are then used for validation. Pretty simple, but as I wrote, really not that neccesary.
"Shared step assembly. This is another source of complexity that I've found people get into too fast. " Again, I agree. only in this case, it grew out of neccessity. I wrote a second testassembly, and saw me rewriting infrastructure code. Now that is important: one doesn't appreciate the usefullness of shared step assemblies if there is no real testing infrastructure code. You can't build up on something and get an acceleration effect, when you start from scratch every time. In Unittesting or component tests, that is ok. Not if you have to rewrite the codebehind, or build up a steplibrary for public consumption on a website.
" Devs start clamoring for certain steps to be run before others, they start creating new schemes for organizing the steps, and before you know it it's too late and nothing works." That's the nice thing.. I stay order independent, because I only share the Given steps, which just build up the testate. There are certain minor ordering issues (a group has to be created before putting users in), but that turned out to be not an issue, we don't have many of these. The harder part is the verbosity.. Scenarios and Backgriounds tend to get very verbose. I solved that by creating a default "invisible" minimal teststate, which again is specified by @default-tagged SpecFlow tests.
" I think there has to be a good reason to take something out, though. With the UI, that's easy -- it's a huge pain to test against the UI." We're on the same boat here.
" The database is a different story, though. " And here not anymore.. ;-) To begin with, my company has no test DB, no Ci system, etc. Every developer has his "homegrown" DB. Sure, shitty situation, but that's what I have to work with (I tried to change that, believe me). Spinup time for the system takes 20 seconds.. a complete "DB restore" time of at least as long..
But that's not even the point. Most people writing tests against the Db use TransactionRollback, thus never hitting the DB anyway. If you run you tests against the DB, how many tests do you run? How long do they take? Are they order-dependant? How often do they fail for reasons other than codechanges (I assume "0 times" in your case, in my case "twice a week") ? So, I need >1000 tests run. As stable as possible, as fast as possible, and containing as much logic as possible that describes the whole usecase. In the end it will be well over 4000 to cover them most important stuff. Against a DB? With restore after every test? .. Haha.. haha.. ha. No, thanks. With no restore, and thus test-crossing teststate, making my testordering important? No, thanks, again, maintenance nightmare. Out of experience, that is. That's my good reason to take the DB out. It's a good one. For me, for my scenario. Practically, every tests starts with a clean state, and still just takes < 20 ms. Backing up the DB by deepcopying an object? 3ms.
" Taking the database out is inherently more complex than just letting it stay." Hehe.. you know, I'm employed solely for testing. My job is to write tests, noone else does. So, for two years I wrote the tests you proclaim are the better ones.. hitting the DB. The problem: it doesn't scale If everybody maintains them, the inefficiency is not even noticed. If one guy maintains them.. and you are that one guy.. believe me, you get very picky about test effiency. What you describe only scales for smaller systems. Just trust me in that. Otherwise, more and more testfailure investigations suck up more % of your daily worktime, hitting 100% pretty quickly. It doesn't scale up.
And the last point: "business logic can sometimes require an incredibly amount of setup and precision, where there could be many different paths down a decision tree." You're right. But I've yet to encounter a dealbreaker when expressing things in Specflow. I'm testing the thing from the outside, basically using the codebehind (WebForms.. oh gosh) logic to create my entities, and then just "running" the method that would be executed on button click. I'm not getting more and more complicated in describing suituations under the hood, because I'm NOT doing "cirtualised component testing". I'm doing "almost" end-to-end testing with the speed and stability benefits of unittesting. And that is, yes, only doable with refactorings, but every minute spent there is oh-so worthy.
So, that's my last point-of-view thing here, after that I think it's up to anyone reading this to make up his own opinions. Just a last sentence: respect other usecases. Even if they seem wrong, they can make sense in a bigger picture.
darrencauthon
commented
11 years ago That was the first post I've read online where someone has called me ignorant twice, then asked that I treat their "usecases" with respect. (I was emailed your post before you edited it.)
I'm not going to go through that thing, except to point out two things to others, as they are common misperceptions:
1.) You don't have to destroy and rebuild your entire database between test runs.
It can be time-consuming to run your migrations on a database over and over. So... don't do it. Just delete the data relevant to the feature you are testing, then reset the data using your Given statements.
2.) You don't have to run your entire test run under a transaction.
I know, I know, it seems neat that you can just rollback a transaction and have the database changes removed. Unless your app runs under that sort of setup, my advice (as always) is to keep it simple and as close to the production setup. Just let your app run against the database and clean up before the next run.
gasparnagy
commented
11 years ago Stepping back to the original problem: in order to "resolve" a dependency (as long as you don't use interfaces), you don't need to register the class, just use it in the constructor of the step class. See example: https://github.com/techtalk/SpecFlow/wiki/Context-Injection
otherwise you configure the dependencies from the app config: add a dependencies element inside the runtime element:
<dependencies>
<register type="MyClass, MyAssembly" as="IMyInterface, MyAssembly"/>
</dependencies>does this solve your problem?
lock[bot]
commented
5 years ago This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Hi, I have a little bit more complicated setup, specifically that I have to use steps from a shared assembly.
My problem is, I need to register my own "shared teststate", which derives from several small testate interfaces, via the built-in IOC (IObjectContainer), so that I can later, within the shared "Given" steps, resolve these small interfaces to my teststae, and add the needed teststate content.
I got it somehow working by pseudo-registering my state class type in [BeforeFeature], and then, in every step file (the shared ones and my own in the real testassembly) I test if my state got initialized. But this means writing coe around the issue, with locking, and I had to squish several bugs in my own code (now sitting on the third). I have a feeling it ought to work better out-of-the-box.
Is there a better way, or somehow a point where I can enforce setting up the (scenariobound) Specflow ObjectContainer, kind of a composition root? It's really painfull to not have one.
Thanks for any feedback, Hannes