Recommendations for an automatic testing environment

[dirkwhoffmann]

VirtualC64 has become so complex that the introduction of an automated regression test system is long overdue.

Does anybody perhaps know a decent tool for running automated tests?

Basically, such a tool needs to do the following:

For each VICE test case ...

• Run VirtualC64 for a certain number of seconds
• Take a screenshot
• Exit and proceed to next test

To verify the result, the screenshots could be viewed manually, uploaded to some web space, or compared automatically with some reference pictures.

Every company uses such tools, so the questions really comes down to what is best suited for us. From my working experience in industry, I know that the amount of time needed to maintain such tools is often underestimated. As man power in this project is very limited, we need something lightweight.

[dirkwhoffmann]

There is a well written tutorial about making an app scriptable here:

https://www.raywenderlich.com/1033-making-a-mac-app-scriptable-tutorial

I wanted to learn this anyway. I plan to make the emulator scriptable (to some extend), so it will be possible to control it remotely by a JX script which could run the VICE test cases for us.

[dirkwhoffmann]

Now as I know how Apple's scripting API works, I have decided not to implement it. To be plainly bold: Apple's API (which goes back to OS 9) as well as the existing documentation is a big piece of 🤬.

I'm looking into Xcode's build in testing capabilities now (which include UI tests). Hope they did a better job there...

Again, there is a nice tutorial on that for those who are interested:

https://www.raywenderlich.com/972-unit-testing-on-macos-part-1-2

[dirkwhoffmann]

There is a nice wizard in Xcode for enabling UI testing in Xcode. When I apply it to the VirtualC64 project, it creates some classes (stubs) automatically which looks good at first glance. However, when I run the test environment, I get this:

😲

BTW, Apple didn't use their slogan "It just works" for a longer time, did they?

UI testing in Xcode seems to be a dead end for now. Maybe Apple can fix it in Xcode 11.

On the positive side, I did manage to add some (limited) scripting support (yes, using this "API" from the Middle Ages). Now, the emulator can be controlled remotely by a JXA script (Javascript for Automation). JXA scripts can be written and executed in Apple's script editor which is part of macOS. E.g., the following script shows how the Errata demo can be loaded and run automatically. The script first configures the hardware configuration. After that, it emulates a drag and drop operation and clicks through the demo until the picture drawing code is executed. It then waits a couple of seconds, takes a screenshot, and terminates:

You can watch the execution of the script inside script editor, too:

After running the script, there is a png image at the specified location:

Using JXA scripts might help us to test the emulator more efficiently. I'm not sure if it's best approach, but for me, it seems to be superior to the shell script approach used for testing VICE.

[dirkwhoffmann]

As I finally got the commands from the Standard Suite to work, I'm beginning to like app scripting. E.g., it's now possible to create new documents remotely:

Before:

JXA script, executed in Script Editor:

c64 = Application('VirtualC64');
newDocument = c64.Document();
c64.documents.push(newDocument);

After:

😎

Although the underlying API is 🤢, it provides a tremendous amount of flexibility.

[dirkwhoffmann]

I've implemented another remote control API using Distributed Notification Center. I'm not sure yet about the best way to go. Still in experimental phase...

For those who are interested: I've written down my findings in the forum:

https://gitq.com/dirkwhoffmann/virtualc64/topics/11/automating-virtualc64-an-interim-report

[dirkwhoffmann]

Has someone any experience with CI systems such as Travis CI? I managed to make Travis CI grab the contents of the master branch from the Github repro on every commit and trigger a build on its own servers via:

xcodebuild -project V64.xcodeproj -scheme VirtualC64 -sdk macosx10.14 build test

It works fine to build the app this way on my local machine and Travis CI seems to be able to build the app, too:

Linking VirtualC64
▸ Generating 'VirtualC64.app.dSYM'
▸ Copying /Users/travis/build/dirkwhoffmann/virtualc64/OSX/Sparkle.framework
▸ Touching VirtualC64.app (in target: VirtualC64)
▸ Processing Info.plist
▸ Compiling VirtualC64Tests.swift
▸ Linking VirtualC64Tests
▸ Touching VirtualC64Tests.xctest (in target: VirtualC64Tests)
▸ Build Succeeded

But when it runs the app in test mode, it stalls and times out after 10 minutes.

No telling error message, of course 😖.

[dirkwhoffmann]

Findings (summary):

• Xcode build in unit test support:

Gives access to all internal classes, but does not offer the ability to take screenshots of the app window (there is no such window when running this kind of test). Hence, not suited for testing VirtualC64.

• Xcode build in UI test support:

Offers the ability to take screenshots of the app window, but gives no access to internal classes. Hence, not suited for VirtualC64 testing.

• Continuous Integration (CI) support:

Premature technology yet, at least for macOS apps. Although there are a couple of providers offering free CI service for Linux based apps, most of them charge for macOS (e.g., CircleCI). More severe: It's hardly impossible to debug server-side issues, because the providers don't grant easy access to their macOS containers. Linux users are better off: E.g., Travis CI lets you download their docker images and therefore replicate their server settings locally. Not possible for macOS, so if Travis CI stalls (as in my case), you are stalled, too.

[dirkwhoffmann]

I have modified the .travis.yml file such that it triggers the build phase only and skips the test phase (there are only dummy tests at the moment, so nothing is really missed out). Now, it works as expected. When a commit is pushed to GitHub, Travis CI gets triggered to build the project automatically. The build status (success or error) is reported back and shown in form of a GitHub batch on the main screen: