Could/should this be integrated into REDCap Core, the module framework, or specific modules?

[mmcev106]

I must admit I forgot this library existed... I'm afraid I might do the same again if I don't stumble upon it again for another year or so, and possibly re-implement parts of it as future needs arise. Could/should any existing tests people have written using this framework be incorporated into REDCap Core, the module framework, and/or specific modules so they are automatically available to all REDCap users (and perhaps even automatically run during updates)? I know none of the above are trivial, but I'm curious to hear your thoughts.

[aldefouw]

@mmcev106

In a word: possibly.

The thought of it is amazing, but I don't know what it would take to implement. There would be a lot of hard technical constraints that we'd have to figure out to make the framework simultaneously flexible enough to meet the needs at an institutional level and consistent enough to be stable to test REDCap versions in general.

What's important to note is that the core tests are being written against LTS versions of REDCap. That's intentionally so because we are mirroring the functionality that is validated by the Regulatory & Software Validation committee.

Talking about making the tests a part of REDCap itself is a bigger change because, if I'm not mistaken, then we're talking about having tests for every version. That would be great, but that's a lot of ground to cover and keep running .. so that would make our UUID discussion / selector discussion all the more important to figure out.

In terms of having every version tested: I'd argue that's the direction things should be moving for the purposes of having confidence that REDCap works as it's stated to ... but that's a Vanderbilt decision, really.

It's also worth noting that there are two facets to testing REDCap (at least as far as I'm concerned):

1) Core Functionality / Validation 2) Institution Specific Functionality

All Vanderbilt could concern yourselves with is the core of course, but administrators like me crave the ability to have confidence moving forward a version of REDCap without having stuff blow up in our face due to customizations.

Thus, I think we would want to leave room for individual institutions to have their own tests for their specific implementation.

For instance, for our institution, I am currently using the framework to do regression testing on some specific hooks that we have implemented. If we use external modules in the future, I'll do the same. (Yes, the EMs have unit tests, but those don't necessarily prove an EM works.)

You'd also want to define why you are testing at both levels.

Would the built in tests be tests for validation (like we are in this project) or are you testing to search out bugs?

There's some overlap present there, but the way that you approach the two are much different. For instance, in validation all we care is that there is someway to do task X ... if there's a bug we catch, that's great - but it isn't our primary concern since we aren't Vanderbilt developers.

For REDCap core, our goal was to have the LTS versions validated and screen recorded (which Cypress does built-in) so that it can be used for audits - either internal or external (like a 21 CFR, for instance). That's at the core level.

For individual institutions, well - they can write whatever test they want.

When I started this project, a lot of people told me that other developers in the consortium had tried automated testing in the past and there were too many technical hurdles and problems to overcome so it probably wasn't worth my time.

Hopefully we can figure out how to overcome some of these hurdles together =)!

Our group would love to talk with Vanderbilt developers and see how we could collaborate on this project. We're aiming for the some of the same goals but maybe have different assumptions ... so it would be good to get on the same page with some of this.

Just drop me a line and let me know.

[mmcev106]

Thanks VERY much for the detailed feedback. Without a large grant specifically for this, I suspect our only option is incremental progress over time. I have a couple of immediate thoughts:

• I wonder if tests for modules themselves would be one of the better places to start. That might be the only area where we have overlap in our current goals. It is rare that my team has a module that can justify investing in unit testing, but can justify time to review any PR sent in from the consortium. Long term I'm envisioning unit tests defined in modules that are run by a continuous integration server immediately when new versions are submitted. I have a lot of ideas about incremental steps toward that goal, but no time to implement them.
• In regard to testing different REDCap versions, I believe that part becomes much easier if the tests and/or the framework supporting them is bundled with each REDCap version. Unit tests would run on a continuous integration server whenever a REDCap commit occurs. This would let us know before a release if an unintended breaking change is introduced, if a test needs to be modified, or if the community needs to be notified of a necessary breaking change.

Let's try to keep talking. I'd really like to take advantage of any shared goals as they arise.

[aldefouw]

@mmcev106

A few thoughts:

1. Tests for External Modules

One of my reasons for suggesting to test EMs in Cypress is because unit tests don't prove that EMs work in the context they exist. Unit tests only prove that a particular method is operating as expected.

This video humorously demonstrates that point and the need for integration tests:

I'd like to see integration tests written in Cypress by the authors of EMs that prove their EMs work in the context that is claimed. The nice part is, these could serve as regression tests for future versions as well.

2. Continuous Integration Tests

You might not realize it, but this is already being done on our framework core tests.

I have a public account through Cypress.io since this is an open-source project so you can see exactly what tests pass / fail: https://dashboard.cypress.io/projects/pvu79o/runs/273/specs

The test workflows are operated through CircleCI.

There's an entire circle.yml setup file here: https://github.com/aldefouw/redcap_cypress/blob/master/circle.yml

The only file that isn't public is the REDCap source code. I have that in a private repository since REDCap's source code isn't available without academic license - but CircleCI workflow can access it because it's within the same Github account.

3. Test Maintenance

Low amounts of Test Maintenance is the key to programmer happiness. Programmers, including me, hate having to constantly make small little changes to tests that broke not because functionality has changed but because they were forced to write something brittle because of the user interface.

The reason I was beating the drum so hard with regard to consistent UUID or data tags is because robust integration tests that don't need a ton of maintenance are predicated upon being able to consistently select elements that appear on screen.

Well-written tests shouldn't break because some subtle button text has changed unless the actual functionality being tested has changed in some way.

REDCap's user interface appears very similar from version to version, but when I started testing it, I quickly discovered how differently it's structured behind the scenes from one version to the next. We don't want to stifle the forward march of functionality, but we also don't want to rewrite tests for every version.

I've tried lots of tricks to make selecting items easier and less brittle from version to version - but haven't really found a silver bullet that will work in all cases.

To make matters harder, our group is not privvy to what is coming down the road, so anything we do is at best a guess.

I truly believe that the silver bullet which would fix most of the brittleness is undoubtedly UUIDs or data tags. If the language file was used in a different way than it is, it would have been sufficient but it's clearly off the table.

The only thing we can do to mitigate the pain currently is to write Cypress helper methods that we can change en masse for each version, but my concern with that is we'll be trading off individual test maintenance for method maintenance instead.

It might be slightly less difficult to update, but on the other hand, one change might fix certain tests and break others ... so it's a delicate balancing act in some ways.

It would really be better if we just had data tags.

4. Bundling REDCap Cypress Test Framework into REDCap

As I mentioned previously, we're already doing continuous integration style tests on each push to the repository. These are intended to land as validation of the REDCap version itself.

Part of the advantage of this test suite is that it attempts takes a black box approach to testing REDCap. It just tests the user interface and tries not to know too much about what's going on behind the scenes.

Our goal is that once we have full test coverage of the core of REDCap, we'll have a suite of test specs that users at institutions can run against their own instance for validation of their own instance.

In your second bullet point, you are referencing being able to run the tests to see if an unintended breaking change is introduced by Vanderbilt. I think that's great - and what you're talking about - I think - is a regression test.

My dream is for these tests to serve not only as validation but also as regression tests. And they can do that without a doubt. But they're going to be a nightmare to maintain if we don't have UUIDs and data tags on our selector elements.

I've seen this over and over again with each new version.

If Vanderbilt wants these to be able to validate more than just LTS versions of REDCap, I really don't see a way out of making the user interface contain stable selectors.

/end of rant about selectors =)

[mmcev106]

Many of the EM framework tests definitely blur the line between unit/integration/regression test...

It's exciting that it sounds like Rob can justify some work toward stable UUIDs. Apart from merging any EM framework PRs, I may not be able help much with that. I completely understand how valuable those could be in some cases, but it's difficult for my team to justify even writing a single test of any kind for many of the projects we do...

I feel like the lowest hanging fruit that I could potentially help with is to provide incentives for module authors to create cypress tests. Streamlined tooling in the EM framwork and automated test execution at module submission time seem like great incentives. Do you have any ideas related to those or other possible action items where our goals might overlap?

[aldefouw]

@mmcev106 - Assuming that this framework becomes the singular vehicle for REDCap integration tests, I could easily see EMs being something an author could write Cypress integration tests for, and REDCap could offer a circle.yml configuration specifically for testing their individual EM to automatically tested against a REDCap instance in the CircleCI environment.

Something great would be if REDCap itself linked into the EM's integration test suite video run to allow people to view the videos of the test functionality passing so you could see the extensiveness of the tests conducted. EMs that had videos proving out functionality through tests would be preferred vs EMs that didn't.

One of the nice things with Cypress is that you can isolate tests to a single folder. Thus, you could use the same framework to test an EM as you would any other part of REDCap.

You could either test their EM in isolation on a test project with the functionality described or test it within the context of other REDCap functionality depending upon where you point Cypress to test from.

Rob has reached out to invite me to a developer's meeting so we can discuss this and more during a meeting. I think one of the things I need to understand is how outside collaborator relationships work with contributing code to REDCap.

Thanks for getting us started!

[mmcev106]

That all sounds great. I probably won't be in that meeting with you since Rob & I are on different teams with different funding sources and rules about how time is spent, not to mention different collaboration standards/goals. I am able to jump in on things relates to External Modules. Feel free to ping me anytime (mark.mcever@vumc.org works).

[aldefouw]

Got it. Thanks for the information and for getting the ball rolling.