Add functional test framework for programmatic detection of behavioral regressions

[eanders-ms]

Is your feature request related to a problem? Please describe.

@darzu brought up the topic of automated regression testing. I'm in agreement this is something we should do. One category of testing is that of looking for behavioral regressions. An example of a recent behavioral regression is the change to return floats rather than ints for sprite position. This change surprised us with the number of regressions that came with it! Here are a few:

• In the game Snail Hike, the snails were no longer able to jump onto the puzzle block
• In Tiny Soccer Cars, the soccer ball started vibrating rapidly up and down

We should be able to write automated tests to catch regressions like this early.

Describe the solution you'd like

• Add an API to register functions as tests. Jasmine might be a nice model to follow (https://jasmine.github.io/).
• Tests can be embedded in MakeCode projects and extensions, defined in tests.ts so they aren't always visible.
• Tests run asynchronously, in the context of a running game. A test is not complete until:
  • It calls the done callback (a function provided as a parameter to the test).
  • It times out.

Additionally:

• We would develop a set of core functional tests that run on every commit to master.
• We would write a backend job to crawl over a list of projects and extensions, running the tests and reporting failures.
• New features would include associated functional tests.

Bonus:

• In the editor, we have a new panel for running tests, similar to Visual Studio's Test Explorer.

[pelikhan]

Recommendation: build the test runner outside the editor and leverage the editor/simulator as widget itself. Allows for out-of-band shipping and don’t bloat editor with testing support. We also use headless chrome in our tests to get the browser/dom context.

---

From: Eric Anderson notifications@github.com Sent: Thursday, July 23, 2020 7:01:38 PM To: microsoft/pxt-arcade pxt-arcade@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [microsoft/pxt-arcade] Add functional test framework for programmatic detection of behavioral regressions (#2257)

Is your feature request related to a problem? Please describe.

@darzuhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdarzu&data=02%7C01%7Cjhalleux%40microsoft.com%7C9f078b4d36dc4152f3cf08d82f7580fd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637311529004885647&sdata=Izhp3HWCHUxmwYBr%2Bywe94lQ9QvnsbjyQJZ4%2BzHhcJA%3D&reserved=0 brought up the topic of automated regression testing. I'm in agreement this is something we should do. One category of testing is that of looking for behavioral regressions. An example of a recent behavioral regression is the change to return floats rather than ints for sprite position. This change surprised us with the number of regressions that came with it! Here are a few:

• In the game Snail Hike, the snails were no longer able to jump onto the puzzle block
• In Tiny Soccer Cars, the soccer ball started vibrating rapidly up and down

We should be able to write automated tests to catch regressions like this early.

Describe the solution you'd like

• Add an API to register functions as tests. Jasmine might be a nice model to follow (https://jasmine.github.io/https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fjasmine.github.io%2F&data=02%7C01%7Cjhalleux%40microsoft.com%7C9f078b4d36dc4152f3cf08d82f7580fd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637311529004895603&sdata=X8ZOKHIR62COr7CeFAofnalzQD7LZIQkR3kSE%2F%2FklJg%3D&reserved=0).
• Tests can be embedded in MakeCode projects and extensions, defined in tests.ts so they aren't always visible.
• Tests run asynchronously, in the context of a running game. A test is not complete until:
  • It calls the done callback (a function provided as a parameter to the test).
  • It times out.

Additionally:

• We would develop a set of core functional tests that run on every commit to master.
• We would write a backend job to crawl over a list of projects and extensions, running the tests and reporting failures.
• New features would include associated functional tests.

Bonus:

• In the editor, we have a new panel for running tests, similar to Visual Studio's Test Explorer.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fpxt-arcade%2Fissues%2F2257&data=02%7C01%7Cjhalleux%40microsoft.com%7C9f078b4d36dc4152f3cf08d82f7580fd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637311529004895603&sdata=7WatclqjPiSFznTtNOV40kUrfh6OlALS6fYntCHbjUk%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAA73QKLFXWKV2EVI7MK2YYLR5DTQFANCNFSM4PGJCECQ&data=02%7C01%7Cjhalleux%40microsoft.com%7C9f078b4d36dc4152f3cf08d82f7580fd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637311529004905559&sdata=zNjB3tP93p12z%2BV8k7JfCJqkhwmsjkR2DShf9sb%2FVUA%3D&reserved=0.

[eanders-ms]

Agreed it would be good/necessary to be able to run tests from a cli. Exposing it in the editor has a few advantages:

• Makes developing, running, and iterating on tests easy
• Users can write and run tests for their own games and extensions in a familiar way

They don't have to be mutually exclusive

[pelikhan]

I would try to make the editor more modular and instead try to break it down. A test runner would not be needed for most of the users.

[darzu]

I agree that exposing it in the editor would be very helpful. TDD is a powerful tool and, like we've one with debugging, it could be one that we slowly figure out how to expose it to users.

For our CI, the steps are probably: enumerating and downloading projects we care about (probably "feature-game" tagged ones), compiling them including test.ts, running test.ts with the runner context.

[pelikhan]

I already have automation that dowbloads and compiles all approved packages (hint: they don’t all compile).

---

From: Daryl Zuniga notifications@github.com Sent: Monday, July 27, 2020 9:53:22 AM To: microsoft/pxt-arcade pxt-arcade@noreply.github.com Cc: Peli de Halleux jhalleux@microsoft.com; Comment comment@noreply.github.com Subject: Re: [microsoft/pxt-arcade] Add functional test framework for programmatic detection of behavioral regressions (#2257)

I agree that exposing it in the editor would be very helpful. TDD is a powerful tool and, like we've one with debugging, it could be one that we slowly figure out how to expose it to users.

For our CI, the steps are probably: enumerating and downloading projects we care about (probably "feature-game" tagged ones), compiling them including test.ts, running test.ts with the runner context.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fpxt-arcade%2Fissues%2F2257%23issuecomment-664513973&data=02%7C01%7Cjhalleux%40microsoft.com%7Cf7605a756756440bd7bf08d8324d92af%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637314656039580021&sdata=ZSzRh5Eaut5sVQfYabem2IudUTf%2Fitdk5L2okpdWX68%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAA73QKPELXWQ5GFN2AKR6UDR5WWIFANCNFSM4PGJCECQ&data=02%7C01%7Cjhalleux%40microsoft.com%7Cf7605a756756440bd7bf08d8324d92af%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637314656039589975&sdata=lPBgrl0C0icUfPTm2OE4FAKUAxvhZ3EYNmf2oAH1xRU%3D&reserved=0.

[darzu]

I already have automation that dowbloads and compiles all approved packages (hint: they don’t all compile).

Where at? Seems like a perfect starting point

[pelikhan]

pxt testghpkgs