Figure out how to handle test plans on github

[PeterWone]

I normally do test plans and tests using Azure DevOps. I moved the code to GitHub as a courtesy to the Linux people but now we have to figure out how to manage testing.

Right at the moment I am super busy with another project so I have to throw this at you and run away for a while. Please everyone grab the code, build it and generally let loose the dogs of war.

Off the top of my head I'd say we need regression tests in code for each of the ways the extension calls arduino-cli because arduino-cli is under active development and we might get breaking changes.

You can also be rude as you like about my UX so long as you have a better idea.

[RBM-PITT]

Peter, I'm a part time programmer, and my experience with most of these tools and development processes is limited. With that said, any reference to training/guidance on what you need /expect would be greatly appreciated.

Thank you for the opportunity to participate.

[PeterWone]

Do you know how to clone the repo to your own machine? Start by doing that. Then rummage through extension.ts and find everywhere it calls child_process.spawn(...) or cli(...). These are the calls to the CLI and each one will require a test case. Gather up these into a list from which we can create our test plan.

Manual test cases suck because they're manual. So we'll automate them. I haven't done this with GitHub before but chances are good at least one of our merry band will know how to set up CICD and if not we'll figure it out together. Start by searching for "CICD github" and "test plan github" or possibly "automated testing github" and read up on it.

In addition to automated checks for CLI breaking changes, we also need to figure out how to test whether paths are being set up correctly for each toolchain/OS combination.

To be honest there will be more complexity in the tests than the actual extension if we do a thorough job. Three things will come out of this:

1. An Arduino CLI integration extension that just works.
2. Fast recovery when breaking changes occur.
3. You'll be able to truthfully claim you have experience with formal testing and CICD.

[RBM-PITT]

Yes, I can clone and build the list!

[MrC74rk]

Howdy y'all! I'm checking in. Not sure what help I can be until testing begins. But I saw you looking for other platforms and I'm on OSX 10.15.7, I can follow a readme, and I'm exceptional at breaking things.

[PeterWone]

Excellent, a Mac user. Welcome!

Somebody play with this https://github.com/marketplace/testquality and work out whether it's useful?

This is a public repo so we qualify for free use.

[PeterWone]

Looks like there is guidance on testing extensions. https://code.visualstudio.com/api/working-with-extensions/testing-extension

Has anyone used Mocha?

[PeterWone]

Mocha

I have figured out how to set up tests for an extension and the latest commit includes some sample tests, one of which fails so you can learn what that looks like. Get the commit, run npm i to pull down the necessary, then switch to debug and run the extension tests as illustrated here.

After the tests run the debug console should look like this.

The sample tests look like this

suite('Extension Test Suite', () => {
  vscode.window.showInformationMessage('Start all tests.');

  test('Sample test', () => {
    assert.strictEqual([1, 2, 3].indexOf(5), -1);
    assert.strictEqual([1, 2, 3].indexOf(0), -1);
    assert.strictEqual([1, 2, 3].indexOf(2), 1);
  });

  test("Sample test 2 which should fail", () => {
    assert.strictEqual([1, 2, 3].indexOf(4), 3);
  });

  test("Sample test 3 which should pass", () => {
    assert.strictEqual([1, 2, 3].indexOf(3), 2);
  });
});

and you can define other suites of tests in the same folder using the pattern *.test.ts. I'd copy the file and rename it, then change the suite and test names and implementations.

CI/CD

For those who don't already know this is Continuous Integration/Continuous Deployment. The idea is that when there's a commit to the repo, the tests all run and if they all pass then the version gets bumped and a package is prepared and pushed to the marketplace. I've found the necessary hooks at https://github.com/PeterWone/vscode-arduino-cli/settings/hooks but first tests need to be set up. I'll create a separate issue for this.

[ghost]

Folks, at the risk of being ridiculed for "not already knowing", what are the steps to install this extension in VSC so that I can actually use it and potentially contribute to the testing? I've tried a few different things and apparently I'm not doing it correctly and I don't see a VSIX.

Never mind...my typical luck, I find the answer five minutes after asking...I located the extensions folder for VSC-Insiders and simply moved the source folder there then re-loaded VSC...

[PeterWone]

@HEMISixty that's the story of my life. It has been my experience with other projects that people who'd like to participate but lack the skills remain that way due to lack of help. So I plan to teach instead of doing wherever possible so I don't end up doing all the fishing.

[ghost]

Well, I hate to admit it, but I've had a few classes, I'm just not a programmer by trade. However, I am no slouch when it comes to technology and figuring things out. Just as well, I spent a few years in a "Test Engineer" capacity creating and executing test plans that included Cisco IOS as well as other vendor firmware. So, long story short, I'm here because I want to hang Christmas lights on my house and it's a PITA to screw around with a json file just so I can program a chip and play with LEDs...

I want to contribute to this because I will benefit from it and I'm more than happy to give up my time to help. I just don't know all the "ins and outs" of VSC or extensions. But, as this example shows, I am capable of figuring things out...lol

Anyway...happy to see you're putting this together and, if I can help, I'm happy to...

[PeterWone]

Well... good. I am a programmer by trade, but I've never been a pro tester.

The way you installed the extension is creatively wrong even if it works. I can see I better write a guide and soon. There's tooling you need to package the extension before you can test after installing from a VSIX. I'll get on that soon. I think we should use the repo wiki for this rather than issues.

One of the things with which I'm still coming to grips is administration of an OSS project. I don't yet know what permissions people need or how to manage them.

[ghost]

Yeah, I figured that out after plugging in a WeMos chip and watching nothing happen... :)

We’ll get through it...

[PeterWone]

@HEMISixty @RBM-PITT @MrC74rk I've made a start on the wiki. This material is pitched at contributors rather than end users. Have a look, I could use some feedback. Are these instructions you can follow?

[RBM-PITT]

Yes, I can follow these instructions.

Should I get this warning when I start the tests?

[PeterWone]

@RBM-PITT er, that doesn't happen on my system. Probably I've missed a dependency. I'll spin up a clean VM and dogfood the instructions. It's late here so I probably won't get to that for a day.

[ghost]

Ok, I've successfully built the vsix and installed it...the tests run as expected and displayed above. So, I think I'm caught up with the rest of you...

So, not being very experienced with GitHub, do we each create our own forks so that we can work with copies of things without having to 'discard' all our local updates each time we mess with the code, or what?

[PeterWone]

That's a common approach. Fork, modify, test, submit pull request. Then someone responsible for main repo merges your changes. That would make you contributors. Collaborators use a similar model but with branches on the same repo. They can accept PRs from contributors.

I suggest beginning with forks, experimenting with creating, modifying and merging from branches within your fork and then once you know what you're doing we can look at moving to collaboration.

It would be a really good idea to get a strong understanding of git and github. There's a free ebook on the git website. You can also buy it as a physical book. Maybe you're already good to go but I find the book very helpful.

[ghost]

Cool, that works, I get the idea and I understand the logic behind version control. I don't know the ins and outs of git and I just don't want to annoy folks. :)

I'm looking at this from a user of the extension and I have questions. First, I don't know if I have correctly installed everything and when I open a workspace, I don't see anything that leads me to believe it is working correctly.

My sketch works just fine with the Microsoft extension. I have to update the json file to get around the bug. But, I can verify my code and deploy it to a controller.

I've removed the Microsoft extension and created a VSIX for this extension and have installed it. I've updated the cli path to include the exe and not just the folder.

So, when I open my sketch, I expect this extension to do something. I don't see anything that leads me to believe it's working and I don't see any way to select the controller I'm using or the serial port it's connected to.

So, how exactly does this extension operate?

[PeterWone]

@HEMISixty

OK let's start with a sanity check. Are you able to invoke arduino-cli on the command line using the path you gave to the extension in the settings. (Did you give it a path? set up instructions not yet provided so perhaps you didn't)

Sorry about abruptly going dark but the power went out six hours ago and it wasn't till dusk that I ran out of patience and set up my big generator.

I wrote you epic instructions here but they are now in the wiki

[ghost]

@PeterWone Yep, sanity checks are good! I am able to run arduino-cli.exe from the command line. Your epic instructions are what I was doing. However, when I attempt to select my board, I don't see a list of boards when I search for wemos or nano. In your instructions, I see you have devices in your list. So, perhaps that indicates I haven't installed the extension properly? I'll rinse and repeat the instructions to create a new VSIX and install it again. I do see 'compile' and 'deploy'. Plus, my status bar does include the 'M:undefined' and 'Choose a board'. So, maybe you'll recognize this description if I'm missing something else...

Thanks for putting up with this...I'll keep working on it...

[PeterWone]

Not at all — thank you for alpha testing.

Here's the code that gets the list from arduino-cli. This is currently lines 334-339 in extension.ts

function loadQuickPickBoards() {
  availableBoards.length = 0;
  cli("core search").then((x: Core[]) => {
    x.forEach(c => c.Boards.forEach(b => availableBoards.push(new QuickPickBoard(b, c))));
    availableBoards.sort(byLabel);
  });

Whenever you see code like cli("core search") that's equivalent (I assume you put arduino-cli.exe on your path as advised) to

arduino-cli core search --format json

The --format json option makes it emit json (SURPRISE!) which is for ease of parsing. For manual sanity checks you can leave it off and the output will be formatted for reading (though sometimes info is less complete).

And finally I ramble to my point, which is that you can see what arduino-cli is telling the extension.

Run this

arduino-cli core search

and you should see this

PATH> arduino-cli core search
ID                   Version   Name
Arrow:samd           2.1.0     Arrow Boards
Intel:arc32          2.0.4     Intel Curie Boards
Intel:i586           1.6.7+1.0 Intel i586 Boards
Intel:i686           1.6.7+1.0 Intel i686 Boards
Microsoft:win10      1.1.2     Windows 10 Iot Core
arduino-beta:mbed    1.2.2     Arduino mbed-enabled Boards
arduino:avr          1.8.3     Arduino AVR Boards
arduino:mbed         1.1.6     Arduino nRF528x Boards (Mbed OS)
arduino:megaavr      1.8.6     Arduino megaAVR Boards
arduino:nrf52        1.0.2     Arduino nRF52 Boards
arduino:sam          1.6.12    Arduino SAM Boards (32-bits ARM Cortex-M3)
arduino:samd         1.8.8     Arduino SAMD Boards (32-bits ARM Cortex-M0+)
arduino:samd_beta    1.6.25    Arduino SAMD Beta Boards (32-bits ARM Cortex-M0+)
atmel-avr-xminis:avr 0.6.0     Atmel AVR Xplained-minis
emoro:avr            3.2.2     EMORO 2560
industruino:samd     1.0.1     Industruino SAMD Boards (32-bits ARM Cortex-M0+)
littleBits:avr       1.0.0     littleBits Arduino AVR Modules

In this case the JSON version contains rather more information so pipe it into a file and open that in VS Code.

arduino-cli core search --format json > coresearch.json

Search it and see whether your board is present.

I have a hunch your WEMOS is from one of the licensed third party Arduino makers. There's a way to supply URLS for getting those boards but I haven't yet implemented it. Nevertheless you should have been able to find the Nano so we have some troubleshooting to do.

[ghost]

Sure, here's the output of that command:

PS.v7.0856:> arduino-cli core search
ID                   Version   Name
Arrow:samd           2.1.0     Arrow Boards
Intel:arc32          2.0.4     Intel Curie Boards
Intel:i586           1.6.7+1.0 Intel i586 Boards
Intel:i686           1.6.7+1.0 Intel i686 Boards
Microsoft:win10      1.1.2     Windows 10 Iot Core
arduino-beta:mbed    1.2.1     Arduino mbed-enabled Boards
arduino:avr          1.8.3     Arduino AVR Boards
arduino:mbed         1.1.4     Arduino nRF528x Boards (Mbed OS)
arduino:megaavr      1.8.6     Arduino megaAVR Boards
arduino:nrf52        1.0.2     Arduino nRF52 Boards
arduino:sam          1.6.12    Arduino SAM Boards (32-bits ARM Cortex-M3)
arduino:samd         1.8.6     Arduino SAMD Boards (32-bits ARM Cortex-M0+)
arduino:samd_beta    1.6.25    Arduino SAMD Beta Boards (32-bits ARM Cortex-M0+)
atmel-avr-xminis:avr 0.6.0     Atmel AVR Xplained-minis
emoro:avr            3.2.2     EMORO 2560
esp8266:esp8266      2.7.2     esp8266
industruino:samd     1.0.1     Industruino SAMD Boards (32-bits ARM Cortex-M0+)
littleBits:avr       1.0.0     littleBits Arduino AVR Modules

PS.v7.0856:>

And, I see that I should probably be using the ESP8266 boards...

[PeterWone]

See my late edits

[ghost]

See my late edits

Yep, I think I see what I can use, at least to test...thanks!!

[PeterWone]

The reason your code fence didn't work is it needs to be three backticks on a separate line. You can follow the opening three with ts on the same line to indicate Typescript, that's how I got the syntax colouring.

[ghost]

Well, crap, I don't get it. I see the 'esp8266' in my command line output. But, when I try to select that from within VSC, no list is seen.

And, I see your comment about Typescript. I'll follow that from now on.

[PeterWone]

Maybe we can set up a mutually suitable time to jointly debug on your system using Live Share. Would you be up for that?

[PeterWone]

Also, the way the extension is written, you can manually install cores and it should detect them. Have a look at these.

• https://arduino.github.io/arduino-cli/getting-started/#install-the-core-for-your-board
• https://arduino.github.io/arduino-cli/getting-started/#adding-3rd-party-cores

[ghost]

Maybe we can set up a mutually suitable time to jointly debug on your system using Live Share. Would you be up for that?

Of course, I'm up for sharing screens and working on things, no worries at all. I have a WebEx account for stuff like that. But, it's not necessary now, I realized where I screwed up.

I rebuilt the VSIX and installed it, I neglected to immediately update the path in settings. I had a brain cramp...I've updated the path and restarted VSC. I now see the drop-downs... So, I don't know why it wasn't working previously, I must have missed something somewhere. The path was defined in settings previously. So, the only difference I can think of is adding the arduino-cli folder to my environment path. I didn't have that previously.

So, is it necessary to add that folder to the system path or is it just necessary to specify it in VSC settings?

[PeterWone]

Webex is good for screen sharing, but VS Code Live Share extension lets you host a shared debugging session where the code is on your computer, the code is running in your debugger on your computer, and we can both see the code and both inspect variables and both step the debugger. It's incredible for this sort of situation, although in this case you already figured it.

It also does voice and text chat. Screen share is just about the only thing it doesn't do.

[ghost]

Yeah, you're right, I forgot about that. I do have the Live Share extensions installed. Thanks again for being attentive and working through this.

[PeterWone]

All of these woes are the reason I went to the trouble of working out how to install cores and libs, and once I get auto-installation of arduino-cli set up the whole thing should be way less painful for new users.

One lesson from today's debacle is that I should set up the third party URLs.

[ghost]

Well, from a user perspective, I now have a few things to add to documentation so that future users will have a smoother experience... So, if I may say, I'm a good representation of part of the user-base for this extension. I just want to use VSC to deploy code to controllers for holiday lightshows. The IoT user base is much like me. We aren't programmers and won't know all the 'ins and outs' of getting things to work. Anyway, you get the point, I'm sure. Those are just my thoughts...you might see a different place for my contributions.

Lastly, I accepted the invite and I won't do anything that I'm unsure of.

I'm off to church now...I'll start messing with things again in a few hours...cheers!!

[ghost]

Well, the output of the core search is a short list. When using the extension in VSC (Choose a board), I saw a message about libraries being updated and the search list in app is much longer. It looks to include the third-party libraries. I could be wrong, but it does look to specify many more than the short list.

[MrC74rk]

Sorry I’ve been MIA. I usually work on this stuff during the week. This is good stuff! I’ll start from scratch on Monday following your new instructions and see how it goes.

You guys are awesome

On Sun, Oct 25, 2020 at 08:44 HEMISixty notifications@github.com wrote:

Well, the output of the core search is a short list. When VSC, I saw a message about libraries being updated and the search list in app is much longer. It looks to include the third-party libraries. I could be wrong, but it does look to specify many more than the short list.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/PeterWone/vscode-arduino-cli/issues/1#issuecomment-716150714, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK2FQ7SSQEOOJV6N2O53COLSMQTSRANCNFSM4S3FHAGQ .