Feature UC Request: How do I integrate a Rubberduck unit test module into an automated testing environment through cli?

[HenrikBach1]

How do I integrate a Rubberduck unit test module into an automated testing environment through cli?

/Henrik

[rubberduck203]

You know, that's actually a very good question. It's not something that I had considered previously. Right now, I don't think it's possible, but I think it could be with some changes.

We would need to extract a TestRunner class out of the TestExplorerDockablePresenter to even start thinking about this. Honestly, that's something that should be done anyway. Right now, the presenter is responsible for discovering and running the tests. Which, I don't much care for. Then we would need to create a TestExplorerCommandLinePresenter and a static entry point to that program.

There would still be a few challenges at that point. Because the tests are written in VBA, they're executed within the context of the VBE and Host Application. Minimally, the command line program would need to open the Host Application in order to run the tests.

Yeah. I think this is possible with a little bit of effort. There's just no way to do it as of right now.

[HenrikBach1]

There are two documented ways from MS to write and process commands from CLI:

* /x macro

• /cmd

Sources: https://support.microsoft.com/en-us/kb/209207 https://msdn.microsoft.com/en-us/library/office/aa211469%28v=office.11%29.aspx

[rubberduck203]

@HenrikBach1 that's for running VBA code from the command line. To run the unit tests, we need to be able to run the .Net code that runs the VBA code from the command line. We're not currently set up to do that, but this is a great feature request and I want to see this happen in a future release.

[HenrikBach1]

Me, too :-) Even though I need it, now. You, know ;-)

[rubberduck203]

You're more than welcome to fork us and submit a pull request if you need this sooner rather than later.

[HenrikBach1]

Yes, I know.

But, I don't know anything on how to do it, effectively, even though you have a highly prescribed way to do it.

I'll think about it. But, for the time being, I prefer to use my time on other projects, even though yours are interesting, too.

[rubberduck203]

No worries. I just like to remind people that sometimes the fastest way to get a feature is to implement it themselves. I definitely want this to happen, I'm just not sure when we'll get to it. We're just a handful of guys doing this in our spare time.

[retailcoder]

Oh wow, that's a good one! ...the addin is registered with a CommandLineSafe DWORD, value 0. AFAIK it isn't possible to load VBE addins from a command line; the DWORD can have an effect in the actual VB6 IDE though...

[retailcoder]

At first I thought "VBA can't load VBE add-ins from command-line, this isn't going to be possible".

Then I had an idea. AssertClass is being exposed via the COM API, and whether or not the add-in is loaded into the IDE is irrelevant - as far as VBA is concerned, it's just another library that contains external code.

So I went on to expose a TestRunner class to the COM API, and wrote this method in a standard module (.bas):

Public Sub RunUnitTests()
    With New Rubberduck.TestRunner
        Debug.Print .RunAllTests(Application.VBE)
    End With
End Sub

And it worked (here running the two unmodified unit test templates):

RunUnitTests
Rubberduck Unit Tests - 07/06/2015 3:21:10 AM
VBAProject.TestModule1.TestMethod1  6ms Inconclusive 
VBAProject.TestModule1.TestMethod2  0ms Failed Fail assertion failed. Expected error was not raised.

But that's not the best part.

namespace Rubberduck.UnitTesting
{
    [ComVisible(true)]
    public interface ITestRunner
    {
        /// <summary>
        /// Runs all Rubberduck unit tests in the IDE, optionally outputting results to specified text file.
        /// </summary>
        /// <param name="vbe"></param>
        /// <param name="outputFilePath"></param>
        /// <returns>Returns a string containing the test results.</returns>
        string RunAllTests(VBE vbe, string outputFilePath = null);
    }
}

The API RunAllTests method not only returns a string containing the test results for immediate output... it also takes an optional string parameter specifying the name of the text file to output test results to.

Guess what this did:

Public Sub RunUnitTests()
    With New Rubberduck.TestRunner
        Debug.Print .RunAllTests(Application.VBE, "C:\Dev\RD_Tests.txt")
    End With
End Sub

(commit aa39249b)

[retailcoder]

So basically, if you can run a macro from the command-line, you can run a macro that runs all tests and outputs the results; you can even split the result string on vbNewLine and save the test results into a db table. The only thing is that it requires passing an instance of the VBE, because TestRunner can't use the instance that's provided by the add-in.

[HenrikBach1]

Hi,

Thank you for your fast response.

However, it seems that I'm missing something in my module. How do I declare/reference the missing part?:

/Henrik

[rubberduck203]

You would have to build the project yourself. Otherwise, it won't be available to you until the next time we release a new version. We closed the issue to indicate that it has been resolved in the next version. We've just not released that version yet.

[retailcoder]

@HenrikBach1 the change is sitting in our [next] branch, which we haven't released yet - we'll post on Rubberduck News and on Twitter as soon as we do!

[HenrikBach1]

Oh, I see. Thank you so very much... :+1:

[HenrikBach1]

Hi,

Hmmm, maybe, I'm asking stupid; But, how do I build and install the project, then?

I've downloaded, unzipped a clone of the project. Loaded and builded the solution, successfully. But, how do I go from here? There seems to be no Installers or executables. What should I look for and replace to get everything running?

/Henrik

[rubberduck203]

That should pretty much do it. If you've successfully built the solution (from the next branch), then you should be able to access the library.

One catch is you might have to remove the reference in VBA, and browse to the /bin directory to add a reference. @retailcoder I know you moved the projects around again. Which *.tlb would @HenrikBach1 need to reference?

[retailcoder]

@ckuhn203 I've folded Rubberduck.UnitTesting.dll back into the main Rubberduck.dll library, so the type library will remain Rubberduck.tlb, at least in RD 1.4.

@HenrikBach1 for a dev build to work, you'll need uninstall the release build, and re-register the VBE add-in manually in regedit.exe, under HKCU\Software\Microsoft\VBA\VBE\6.0\Addins\Rubberduck.Extension:

The Rubberduck.Extension key needs these values:

• Description (string, value "Rubberduck")
• FriendlyName (string, value "Rubberduck")
• LoadBehavior (DWORD, value 3)
• CommandLineSafe (DWORD, value 0) is optional (no-op in VBA)

On a Microsoft Office x64 setup you need to create the key under HKCU\Software\Microsoft\VBA\VBE\6.0\Addins64.

This key is where Office VBA looks for installed VBE add-ins; load behavior 3 specifies "load at startup".

A successful build from Microsoft Visual Studio 2013 (we're using Community Edition) should register all the COM types.

In the Rubberduck project properties, make EXCEL.EXE (specify the full path) launch on debug if you want to F5-launch the solution from within Visual Studio - if it works from VS, then launching Excel (or any Office app for that matter) without VS should load your last build.

Let me/us know how it goes!

[HenrikBach1]

Thank you for your fast responses.

However, I found, that I've downloaded wrong branch. Now I've downloaded your [next] branch:

BtW, which configuration should I build - I'm running Windows 7 - x64, VS 2013 Pro, Office 2013?

And should I build solution or Rubberduck?

Just to be sure...

/Henrik

[retailcoder]

Debug configuration should be good; build the solution. You must be running VS as administrator to register the COM types :wink:

[retailcoder]

Also note, we're still working on the next release, so the [next] branch isn't exactly stable (although it's getting there) - feel free to submit issues if you discover any bugs, we'll try to fix it before we ship it :smile:

[HenrikBach1]

I've builded the solution with Debug configuration as Administrator to register the COM types and I've edited my registry for Office 2013 for x64 setup.

Everything seems to be in good condition, when I start my Access database with loaded VBA modules. Even Test Explore shows my two Tests... :-)

However, I get this error when I try to "Run all tests" in the Rubberduck menu:

What did I miss?

/Henrik

[Hosch250]

@HenrikBach1 You are referencing 1.22's TLB. You need to update the TLB in Tools / References.

[rubberduck203]

@retailcoder when you folded unit tests back into the main project, did you revert the changes to the auto-reference?

@HenrikBach1 if you go to Tools>>References, you'll need to make sure you've got a good reference to Rubberduck. I'd recommend browsing to the project's bin folder to add the reference.

[retailcoder]

@ckuhn203 I did. ... ....I think.

[retailcoder]

Didn't need to:

    public static void EnsureReferenceToAddInLibrary(this VBProject project)
    {
        var referencePath = Path.ChangeExtension(Assembly.GetExecutingAssembly().Location, ".tlb");

        List<Reference> existing = project.References.Cast<Reference>().Where(r => r.Name == "Rubberduck").ToList();
        foreach (Reference reference in existing)
        {
            project.References.Remove(reference);
        }

        if (project.References.Cast<Reference>().All(r => r.FullPath != referencePath))
        {
            project.References.AddFromFile(referencePath);
        }
    }

The .tlb location is acquired via reflection magic.

Now, looking at this code just made me realize, if we run this extension method just prior to executing unit tests, we'll always have the correct version referenced, since it removes any "Rubberduck" prior to adding the correct one. Thoughts? Perhaps we could amend it to only remove a reference when its FullPath doesn't match the referencePath?

[HenrikBach1]

After an unregister and then a register did the job. Now, it runs when I start it manually.

[retailcoder]

@HenrikBach1 you mean with the new TestRunner API? Awesome! :hatching_chick:

[HenrikBach1]

Yes, and now I can reference it from an Access procedure as suggested. Now, I've to think a test infrastructure up. Thanks again, guys... :+1:

[HenrikBach1]

Oh, yeah, another thing,

Can you remove the time stamps and running time in the output file. Then, it will make things easier to make a comparison with a baseline file, I think...:

Rubberduck Unit Tests Spartacus.Test_qryGetArtifactTypeTo_prmArtifactID.Test_qryGetArtifactTypeTo_prmArtifactID_1 Succeeded Spartacus.Test_qryGetArtifactTypeTo_prmArtifactID.Test_qryGetArtifactTypeTo_prmArtifactID_0 Succeeded

/Henrik

[HenrikBach1]

And is there a way I somehow can short or make calls to tests get the same appearance:

Rubberduck Unit Tests Spartacus.Test_qryGetArtifactTypeTo_prmArtifactID.Test_qryGetArtifactTypeTo_prmArtifactID_0 Succeeded Spartacus.Test_qryGetArtifactTypeTo_prmArtifactID.Test_qryGetArtifactTypeTo_prmArtifactID_1 Succeeded

/Henrik

[retailcoder]

The output file is merely a convenience. You can take full control of the output by intercepting the String output and writing the file yourself (untested):

Dim output As String
With New Rubberduck.TestRunner
    output = .RunAllTests(Application.VBE)
End With

Dim outputLines() As Variant
outputLines = Split(output, vbNewLine)

Dim lineFields() As Variant
Dim line As Long
For line = LBound(outputLines) + 1 To UBound(outputLines) 'skip 1st line
    lineFields = Split(outputLines(line), vbTab)

    'lineFields(0) contains the QualifiedTestMethodName
    'CLng(Left(lineFields(1),Len(lineFields(1)) - 2) ' gets the number of milliseconds
    'lineFields(2) returns the outcome ....and the output message, if any.
    '''lineFields(3) ...there's currently no tab between the outcome and the message... will change that.
Next

---

Now that I think of it... the release version of TestRunner.RunAllTests might return an ArrayList containing RubberduckTestResult objects instead of a String.

Dim output As Variant
With New Rubberduck.TestRunner
    output = .RunAllTests(Application.VBE)
End With

Dim fn As Integer
fn = FreeFile

Open "C:\Dev\RD_Tests.txt" For Append As #fn
Print #fn, "Unit Test Session: " & Now

Dim result As Rubberduck.RubberduckTestResult
For Each result In output
    Write #fn, result.ModuleName, _
               result.MethodName, _
               result.Outcome, _
               result.Message
Next

Close #fn

Yeah that would be much cleaner.. thoughts?

[rubberduck203]

I don't much like requiring people to use ArrayList, but I think the concept is sound. It would be easier to expose the output as a plain Enumerable. Exposing TestResult to com should be pretty trivial.

[retailcoder]

Actually... @ckuhn203 how about returning a TestSession object that would expose members such as:

string UserName { get; }
DateTime SessionStart { get; }
DateTime SessionEnd { get; }
int SucceededTestCount { get; }
int FailedTestCount { get; }
int InconclusiveTestCount { get; }
TestResult[] Results { get; }

And then we just make TestResult COM-visible?

[rubberduck203]

I think that's a very good idea, but let's make Results Enumerable so it can be iterated over with a For Each loop. Also, I can't shake the feeling that the managed code should also be making use of TestRunner.

[HenrikBach1]

Something strange has happened with my VBE so I can't debug (the debugger seems to ignore my break points) anymore - Any suggestions?

[retailcoder]

@HenrikBach1 if your breakpoints are hollow it means the code in VS isn't for the build that's running in the VBE (assuming the VBE has the add-in loaded). Try right-clicking the solution and making a clean+rebuild.

Otherwise, you have a Stack Exchange account? Join us in Code Review's VBA Rubberducking chatroom anytime! (we can authorize you if you don't have the 20 rep points normally required to talk in chat).

[rubberduck203]

@Henrik the VBE or VS?

[HenrikBach1]

It is in the VBE. I'll contact you the SE account.

[HenrikBach1]

I did a clean+rebuild as administrator and it did the job. However, now the Rubberduck.TestRunner class isn't recognized anymore...

[rubberduck203]

Do you have RD installed side by side the development version? I imagine that could cause problems.

[HenrikBach1]

Yes, I had. Now removed.

Made a clean+rebuild, again, as Admin, re-referenced RD.tlb from: C:\Users\sad\OneDrive\Projects\Private\Rubberduck\Rubberduck-next\Rubberduck-next\RetailCoder.VBE\bin\Debug

However, the problem still persists and the RD menu is gone, too, after I removed the RD install...

[rubberduck203]

Ahh. Yes. You'll need to manually register the addin by adding the registry keys found here. https://github.com/rubberduck-vba/Rubberduck/wiki/Building-&-Installation

[HenrikBach1]

Now, my RD menu is back and debugging works with RD "installed".

But, the VBE is still annoyed not to find the RD.TestRunner class, when I try to run macros...

[retailcoder]

@HenrikBach1 make sure you are referencing the .tlb and that the reference isn't broken - it's possible it breaks every time you rebuild RD. Once the reference is ok, verify that the TestRunner class is exposed in the object browser (F2 in the VBE).

[HenrikBach1]

Yes, it is exposed in the object browser and the reference to the .tlb is not broken. I'll try a reboot...

[HenrikBach1]

No effect. The VBE can't somehow not see/reference the .tlb, even if I can look it up in the object browser.

[rubberduck203]

Are you connected to the right *.tlb? You'd want the one under ./Rubberduck/RetailCoder.VBE/bin/Debug/ not the one that's whereever you installed.

Or... Wait... The installer should have removed that. COM registration issue maybe??

[HenrikBach1]

I was thinking of that, too, and I'm trying to build an install. But, I ran into problems, because I can't figure out Inno Setup's error messages and why they appear...

On Sun, Jun 14, 2015 at 9:06 PM, Christopher McClellan < notifications@github.com> wrote:

Are you connected to the right *.tlb? You'd want the one under ./Rubberduck/RetailCoder.VBE/bin/Debug/ not the one that's whereever you installed.

Or... Wait... The installer should have removed that. COM registration issue maybe??

— Reply to this email directly or view it on GitHub https://github.com/rubberduck-vba/Rubberduck/issues/585#issuecomment-111864855 .

Med venlig hilsen/Kind regards Henrik Bach

Civilingeniør/Cand. Polyt./M.Sc.E. JTI: ENFP

Email: bach dot henrik at gmail dot com M: +45 20 87 10 35 Fight back globalism - Start locally: Educate your self!

[HenrikBach1]

Ok.

Now, I've build my own Install (from the .iss file) and am able to use it. However, it seems that the original install (1.3.0.1) of RD isn't able to run macros, too, because, I've made a clean and fresh install on a new fresh computer and it wasn't able to run macros, too.

Everything else seems to work...

/Henrik

On Mon, Jun 15, 2015 at 7:49 AM, Henrik Bach bach.henrik@gmail.com wrote:

I was thinking of that, too, and I'm trying to build an install. But, I ran into problems, because I can't figure out Inno Setup's error messages and why they appear...

On Sun, Jun 14, 2015 at 9:06 PM, Christopher McClellan < notifications@github.com> wrote:

Are you connected to the right *.tlb? You'd want the one under ./Rubberduck/RetailCoder.VBE/bin/Debug/ not the one that's whereever you installed.

Or... Wait... The installer should have removed that. COM registration issue maybe??

— Reply to this email directly or view it on GitHub https://github.com/rubberduck-vba/Rubberduck/issues/585#issuecomment-111864855 .

Med venlig hilsen/Kind regards Henrik Bach

Civilingeniør/Cand. Polyt./M.Sc.E. JTI: ENFP

Email: bach dot henrik at gmail dot com M: +45 20 87 10 35 Fight back globalism - Start locally: Educate your self!

Med venlig hilsen/Kind regards Henrik Bach

Civilingeniør/Cand. Polyt./M.Sc.E. JTI: ENFP

Email: bach dot henrik at gmail dot com M: +45 20 87 10 35 Fight back globalism - Start locally: Educate your self!

[HenrikBach1]

Hi

After having played around a lot, I think the issue is, that the TestRunner class isn't somehow correctly registered. It was my Access database that luckily brought a reference to the Rubberduck COM object with it and accidentally the compiled source code was brought to the clean computer, too. That way everything else seemed to work when peeking with the Object Browser and References... :-(

Now, I've released the reference to the Rubberduck COM object in VBE. Uninstalled RD. Installed the original RD (1.3.0.1), which now shows no reference to the Rubberduck COM object! in VBE.

Any ideas?

/Henrik