Overall Testing Story for Office Add-ins

[LanceEa]

This potentially is a User Voice but thought I would start here

TL;DR

• Challenge 1: No documentation, or direction on mocking Office-Js Api's for testing. I know there are many permutations and opinions on tooling and testing frameworks so documenting them all is not feasible. But, potentially providing a mock or at least a small write up showing some samples I think would help.

• Challenge 2: No easy way to test against multiple platforms/versions of Excel It would be nice if there was an easier way to test against multiple platforms. Similar to BrowserStack, SauceLabs and Xamarin Test Cloud or at something like what the Edge Team did with Virtual Machines. This is probably more of a User Voice but also could benefit from a short write-up giving some guidance.

Issue: There is no documentation on how to best test add-in code. I understand that is fairly broad so let me explain.

I'm developing my add-in with Angular which has a great Unit and E2E integration testing story. Since the add-ins is web based I can leverage the Angular teams opinions on testing and if someone was using some other framework they could follow the guidelines set by that framework. But, where it gets a little bit dicey is when you start interacting with Office-js API's.

So, what I did was first focus completely on the UI and aspects of add-in that do not require office-js. This way I can follow my normal developer workflow (Chrome Debug tools, HMR code refreshes, automated unit testing,etc...). Then I introduced the Office API's and at this point I have avoided writing any tests for those functions. To me that is the first challenge because there is no examples, no mocks and in general no direction on how to best test these. I recognize I could mock/spy on the office api's but the setup to do that would be fairly painful (I recognize someone has to feel the pain at least once even if a mock was provided 😄 ).

The other challenge is when we need to actually test the add-in within Excel. I have to physically test on my Mac, my Windows 10 PC and with different version of Excel. The number of combinations of OS/Excel version is definitely not controllable, especially with external users (outside our company). So, I would love to get some direction here.

[lindalu-MSFT]

Hi @LanceEa! I'm on the Add-ins team and want to apologize for the delayed response. We got buried preparing for and then attending Ignite. It's a little late tonight to ask a content developer on my team to repspond, but I saw this and wanted to thank you for taking the time to write and point out our content gap when it comes to testing add-ins. I'm going to pass this on to our manager @Reezaali to address. Cheers, Linda

[lalitkapoor]

Thanks @lindalu-MSFT. Can’t wait to hear more on this topic from @Reezaali!

[lindalu-MSFT]

Hi @LanceEa Have you had a chance to look at the topics in https://aka.ms/Kqj3uz? This documentation node has several topics on testing and debugging. Outlook has some topics as well here: https://docs.microsoft.com/en-us/outlook/add-ins/testing-and-tips.

I noticed someone asked this question in Stack Overflow: https://aka.ms/B1o3qn

linda

[LanceEa]

@lindalu-MSFT - I figured most people were busy with Ignite :) so no worries.

I look forward to hearing more on this because it definitely is still a challenge and I believe it has an opportunity to increase developer adoption of Office Add-ins if these challenges can be addressed.

As for the links you sent, most of them are just how to do manual testing by side-loading the add-ins. This works fine for manual testing an add-in and for early distribution but isn't quite what I was referring to above.

Challenge 1 above is more about automated testing and continuous integration scenarios where when I commit changes in my code repository, I then need to run automated unit test and integration test, etc... before my changes can be deployed. You can mock/create spies for office-js in unit tests but there is a lot to mock so this becomes tedious. It is also very difficult to run end-to-end/integration testing.

Challenge 2 goes along with the last point about difficult to do end-to-end/integration testing because there is no easy way to test on various versions of excel (i.e. Office 2013, Office 2016 Slow Channel, Office Online, Office for mac, etc...). The best we have is the requirement sets which is more similar in my mind to feature detection. Kind of like what Modernizr(https://modernizr.com/) has offered web developers for a few years. But, if a user comes to me and says I'm seeing issue "XYZ" and I'm on Office 2013. There is no easy way for me to test this because my Mac & PC both have the latest versions. Which means I need to have other PC's or virtual environments installed with various versions. Also, that user might not be on the latest patch release. Our support response could be update Office then come back to use but unfortunately reality is that won't fly :). So, I would love to see a better overall story for this.

Let me know if anyone wants to kick around ideas. More than willing to go into more details.

[kbrandl]

@LanceEa -- thanks so much for the providing such a thoughtful and thorough explanation of the challenges you're encountering re the testing of Office Add-ins. I understand completely and wanted to let you know that the Office Add-ins documentation folks are aware of this documentation gap and are actively working to address it (although the timeframe for publication of these docs is still TBD). In the meantime, I'll see if I may be able to identify someone on the product team that would be able to provide you with some guidance re some of the challenges that you've encountered.

[joekrump]

@kbrandl Any update on this?

I'm also curious to know if there a recommended end-to-end testing tool that works for testing the web version of Outlook Add-ins?

[lindalu-MSFT]

@LanceEa @joekrump We do have a few new topics in our documentation:

Under the Office Add-ins test and debug section, here's a new topic:

Troubleshoot user errors with Office Add-ins

There's also Debug Office Add-ins in Visual Studio and Debug add-ins using F12 developer tools.

[kbrandl]

@LanceEa @joekrump apologies for the slow response on this; unfortunately I don't yet have any new information to share. I'll follow up again with the folks that I initially pinged about this issue, and you should see this thread updated with new information as it becomes available.

[sureshjoshi]

For what it's worth (and admittedly, might not be worth a lot), I've had decent success in using Bob Martin's Clean architecture (https://8thlight.com/blog/uncle-bob/2012/08/13/the-clean-architecture.html) for developing my plugins - in order to make them somewhat testable.

This is entirely commentary, not really solving any of @LanceEa's problems - just might be useful for someone else down the line.

I won't re-hash the details, other than to note that most of my business logic (80+%) occurs in API/Mock-free code (use cases) which are easily tested with the standard JS unit test tools (ie. Mocha-ish) in a 'domain' layer.

Then I have a set of tests and mocks for my 'data' layer, which contains database-esque components - and these are also unit testable and injected into higher layers. Many 3rd party dependencies which might need to get flipped back out can go here.

The next component is the UI layer, which is UI-testable in any headless web-browser (interactions with the Word document are spied out).

Lastly comes the interaction with the Word document itself, which I haven't solved - and is kinda the crux of this issue :)

I've played around with automated testing that simulates clicks, then exports documents to be compared offline - but it's kinda meh, as it falls prey to the classic problems with click-centric testing.

[kbrandl]

@LanceEa thank you for your patience as I've tried to track down more information for you regarding the testing-related challenges that you've described in this thread.

• Re challenge 1 (no guidance available about end-to-end testing of Office Add-ins), I've confirmed with the Office Add-ins documentation team that there's already a work item in their backlog to address this documentation gap. Timeline for publication of this guidance is currently still TBD, but it's in the queue.

• Re challenge 2 (no easy way to test against multiple platforms/versions of Excel), you were correct in your initial instinct that this suggestion would be a good candidate for UserVoice. Can you please post it there (if you haven't already done so), so that other customers can add their votes for it in the days ahead?

Thanks again for your patience and for providing such a detailed write-up of the testing-related challenges that you've been facing with Office Add-ins. I'll leave this issue open to track the request for additional documentation about end-to-end testing.

[maxpaj]

+1

Need to be able to test our Outlook Add-in on Outlook 2013 for customers that have slow moving IT departments. At the moment a customer is experiencing issues and I can't test anything on my end. What do I do?

[carlosconnected]

@lindalu-MSFT @kbrandl is there a way that you folks could provide office.js as a module that could be mocked through webpack's NormalModuleReplacementPlugin. The purpose would be to run an Add-In as a standalone webapp, so standard integration tests could be run using frameworks like Cypress or Selenium?

I'm working on a Add-In implemented on angularjs and I have not been able to achieve what I just proposed.

[carlosconnected]

I think if the restriction of having the office.js coming from a CDN is changed and it's instead modularized, it will be possible to run integration tests by having Add-ins running as standalone apps just by replacing the module with an office mock module.

[LanceEa]

@kbrandl - Sorry for the slow response on this one.

Is there any update on Challenge 1?

As for #2 I have entered a UserVoice. https://officespdev.uservoice.com/forums/224641-feature-requests-and-feedback/suggestions/34345651-add-in-developers-automated-testing-platform

[kbrandl]

Transferring this issue from OfficeDev/office-js-docs into OfficeDev/office-js-docs-pr.

[sancarn]

Open office has javascript capability Might be possible to build a OfficeJS wrapper for open office. That'd provide a pretty good testing ground for developers at least. Would also allow OfficeJS addons to run on both platforms which could be nice... 😛

[r0bb077]

I see people have mentioned that they have been mocking the Office.js file, I don't suppose they'd care to share some examples on how they did this as I'm struggling currently?

[mattcopas]

+1 - some guidelines/examples on unit testing would be much appreciated :)

[shirbr510]

I tried mocking Office with some sort of an office-js wrapper I've written but so far have failed in doing so (had my whole mocks working but my app couldn't work with Officejs properly)

Would also love to see some sort of example or documentation

[shirbr510]

I have ended up doing the following @ setupTests.js (I'm using jest if it matters):

import OfficeMock from './__mocks__/Office'

const Office = global.Office;

beforeEach(()=>{
    global.Office = OfficeMock
})

afterEach(()=>{
    global.Office = Office
})

the OfficeMock is a simple object mapping that fits the properties I need from Office.js

I do believe that a proper solution including some guidelines and examples is still necessary though...

[oleksii-pi]

Test:

/* eslint-disable @typescript-eslint/no-angle-bracket-type-assertion */
/* eslint-disable @typescript-eslint/no-object-literal-type-assertion */
...

test("getCustomRange", async() => {
  const expectedRange = <Word.Range>{text: "TT"};

  const paragraph: Word.Paragraph = <Word.Paragraph>{
    text: "textTTtextTtextTTtextTTtext",
    search: function(_text) {
      return <Word.RangeCollection>{
        load: function() {
          return this;
        },
        items: [null, expectedRange]
      };
    },
    context: {
      sync: function() {
        return new Promise(accept => accept());
      }
    }
  };

  expect(await ...).toBe(
    expectedRange
  );
});

[oleksii-pi]

const context = {
      sync: () => {new Promise(accept => accept());},
      document: {
        body: {
          paragraphs: []
        }
      }
    };
    (global as any).Word = {run: async callBack => await callBack(context)}; // rollback if needed

[Alex-ray]

Hello @kbrandl , I also need support for this.

My team is building a word add-in that helps lawyers create templates for their documents. Naturally reliability is our number one priority here. End to end integration tests are critical in our ability to ensure our add-in's reliability.

The actual work this might take from your team feels low in relation to the potential increase in quality and reliability of your add-ins.

[kbrandl]

Thanks for reaching out @Alex-ray and I can certainly appreciate the perspective you've shared. This work item is in the documentation backlog; @davidchesnut is there any update you can provide regarding status of this work item?

[davidchesnut]

Hi @kbrandl @LanceEa and folks, The engineering team is looking into solutions in this space, but of course it needs to be general and scalable for multiple testing frameworks. So that effort will take some time. On the documentation side I believe there are some patterns we can share to make it easier to test. I'm working to get those documented and will provide an update once we have something available. Thanks!

[phlppnhllngr]

If anyone is interested, I created a small demo/repository where I wrote down some instructions on how to implement e2e tests for TaskPane apps (in this case the sample Contoso app for Word created by yeoman generator office). I was using Appium and WinAppDriver. The link is: https://github.com/phlppnhllngr/office-addin-integration-testing

[kb339]

How to integrate Office.js with UFT/Selenium for Excel online (Office 365) automation testing?

[seifeldeen92]

@kb339 Have you reached any progress in that? i am stuck in the same spot

[RohilaDasari]

Is there any update on this?

[JensMadsen]

Any updates? We just deployed our add-in with a huge bug in it ... We need automated tests

[celownick]

Hi @davidchesnut , @kbrandl and folks, can I kindly ask for any updates on this ? We are preparing strategy for add-in testing and answering questions mentioned by @LanceEa wound have a crucial meaning for our team. Thank you

[JensMadsen]

Any updates? We need to know the timeline for this "feature". If you know that this will happen in 2025 let us know. In that case, we will invest time in developing e.g. Selenium click e2e tests. If you, on the other hand, are actually working on this an expect a release in 2020 we will probably wait for it :-) But without an update, it is difficult to make a strategic decision on what to do (we must have test for CI/CD and QA). Best Regards

[ludmilanesvitiy]

Hey team! We also waiting for some updates with examples for e2e testing of word add-in. Please, post some updates - when do you plan to create this documentation? Is it possible at all to make this type of automated testing to work with word add-in. If I can help you with something relative to it to speed up the issue resolving - just let me know.

[ElizabethSamuel-MSFT]

Related issue: https://github.com/OfficeDev/office-js-docs-pr/issues/1611

[JensMadsen]

Any news? @ElizabethSamuel-MSFT

[JensMadsen]

We have started building ui tests using Selenium. We currently only test Word online. We plan to use Winium to test Word desktop Windows. The tests run on Github Actions and run headless in incognito mode. Currently, we only have a setup for Chrome (chromedriver). If you plan to create a similar setup please note that:

1. In order for the addin to load and function properly you have to change the default Selenium cookie settings when running in incognito mode:

   chromeOptions.setUserPreferences({ 'profile.cookie_controls_mode': 0 })

   We had some problems before we realized this because Word Online loads fine and you will be able to login, but the addin does not show up anywhere (ribbon, appsource, edit->my addins, etc).

2. For stable tests you have to fix the window sizes by running:

   await driver
     .manage()
     .window()
     .setSize(3000, 2133)

   (setRect for newer versions of Selenium).

3. The addin runs in a designated iframe which is nested in the application iframe. In order to interact with the addin you have to switch to the parent iframe and then locate and switch to the addin iframe:

   // switch to top iframe
   await driver.wait(until.ableToSwitchToFrame(0))
   // locate addin iframe
   const addinIframes = await driver.findElements(
     By.xpath('//iframe[contains(@title,"<name of your addin>")]'),
   )
   await driver.wait(until.ableToSwitchToFrame(addinIframes[0]))

Hope this can be of some help. If you need more details please ask :-)

[piemart]

Hi @JensMadsen, Thanks for the info. I would like to setup UI testing with Selenium on an Outlook Desktop add-in with edge chromium, but I'm struggling. The add-in starts the localhost:3000/taskpane.html and after some user inputs, the taskpane window opens an external appdomain (listed in the manifext.xml file). Here I would like Selenium to take over, so I can select the desired elements by their css_selector. Do you think this is possible? Some more details would be appreciated :-)

[JensMadsen]

Hi @piemart

No quite sure what you wnat to achieve, but it sounds similar to what I do. In metasteps:

1. start in incognito mode (I use Chrome). Incognito more stable
2. login to office.com (make sure you have a test user). you have to pass username and password
3. I open an existing test document (I test Word)-> office opens in a new tab
4. go to new tab
5. Do your tests (I test a taskpane addin; the one that load on the side )

[abdulkareemnalband]

We would like to use https://github.com/microsoft/playwright to automate the testing of desktop addins. If word addin will expose WebView2 instance over Chrome DevTools Protocol , then we could use https://playwright.dev/docs/api/class-browsertype#browser-type-connect-over-cdp to attach to that and use playwright to write tests

[mitoihs]

My coworker has recently set up Selenium for desktop Excel add-in. That's what he said:

Since a lot of people are struggling, I guess I can give some information. Please note that this is a whacky way of doing it, but AFAIK only way for now.

Webview2 (chromium edge) has support for Selenium: https://docs.microsoft.com/en-us/microsoft-edge/webview2/how-to/webdriver It is possible to instrumentate desktop Office (in my case Excel) to start Webview2 and connect with Selenium. To instrumentate Excel, I've used WinAppDriver. It is hard to set up (notable problems are WinAppDriver being slow and not suitable to drive Excel over simple clicks, WinAppDriver requiring Appium requiring Selenium v3, but Webview2 needs Selenium v4 - I've overcome this by using multiple projects that only interface externally). If all goes correctly you will end up with WinAppDriver application that can start Office and connect Webview2, and Selenium that can drive add-in. It is hard to control the actual application, but for that purpose, I use OfficeJs via Selenium function ExecuteScript.

TL;DR: Possible to use Desktop Office with Selenium, very hard to set up.

[ferrykranenburgcw]

We would like to test our add-in in Outlook web using Playwright. But it looks very hard to just activate an e-mail and then activate our add-in inside Outlook. We also tried another way to test our code in Javascript only (using the Jest test framework) but we are missing bits an pieces from the Office.js framework to make it all work.

Any documentation about unit testing and integration testing add-in would be very welcome and helpful.

[AlexJerabek]

Hi everyone,

We appreciate all the discussion and feedback requesting testing solutions for Office Add-ins and Office JS. Unfortunately, there are no automated or unit testing solutions that we provide at this time. We're working on a solution, but have no concrete implementation or release details to share at this time. We realize this is an unsatisfying answer, but we feel it’s better to be clear about the support story, rather than to create false expectations.

We are moving the discussion to M365 Developer Platform Ideas. For the primary concern around standardized testing, please upvote and comment on this idea: Standard way to unit test and mock office-js API. For the other related requests that have accumulated on this issue, we also ask that you take your requests to the M365 Developer Platform Ideas and submit them as ideas (or upvote existing ones). This will help us prioritize things in an orderly fashion.

If there are other issues about the documentation that have been raised in this thread but were not addressed, please create a new issue to track each specific problem.

Thank you.

[lafe]

@AlexJerabek I think, this is a very unsatisfying approach on how to deal with this issue. It has 42 comments and has been open for almost four years. By creating a new issue on a different platform, the complete history is gone (the new idea does not even reference this GitHub issue) and it appears as a new request. However, it is not a new customer request but an RFC from Microsoft to the community. Instead of posting it here and engaging directly with the affected developers, it is moved to a different platform and poses some questions. To me it appears, that Microsoft does not want to deal with this issue and moving it to a different platform buys time. Furthermore, if there is no feedback, you can internally point to the new idea and say "we tried but there seems to be no interest".

Furthermore, the new platform seems to be one where ideas go to die: currently, there are 217 ideas with the state "New" but only 4 ideas that have a different state (2 in "Working on it" and 2 in "Completed"). The ideas with different state where updated by their creators, so those are not community requests that were introduced through this platform. Even off-topic ideas like this one are not moderated. An actively used platform that is used for product planning and is actively monitored looks different...

I can somewhat understand, if you do not want to deal with this issue, as it is complex and might only be relevant for a subset of users. However, please be honest with it instead of pulling moves like this.

[giulianozor]

@lafe I agree with you. IMHO the project is (almost) dead and I see very little chances of it being maintained unless some major security issue is found. Unfortunately there are no alternatives AFAIK

[AlexJerabek]

Hi @lafe and @giulianozor,

I'm sorry that you felt unheard and disrespected by this action. We'll try to be better about how we handle these long-running issues in the future.

The goal of moving this issue and discussion was to be honest about the current state of testing Office-JS frameworks. The Developer Experience team (@igor-ribeiiro, @millerds, and @LouMM) is tracking this issue and trying to deliver a robust testing framework. However, there are no unit test frameworks or "mock" Office-JS frameworks that currently exist. There's nothing functional, but undocumented. Switching the issue from the documentation repo to Ideas is an effort to send the correct message about where the work is in the product lifecycle. We've added links to this thread (and the related unit testing thread) to the Idea, so everyone's contributions here are not lost or forgotten.

Thank for you also sharing the concerns about Tech Community. For the sake of transparency, I'll say the move to that platform was recent and sudden after needing to move from UserVoice. There are internal habits and processes that need to be strengthened. We need to do better as a team in monitoring Tech Community Ideas if we are going to recommend our developers go there to share their needs. Tagging @ElizabethSamuel-MSFT to help figure out how our team can increase their efficacy there.

Please know that your needs are heard and advocated for, beyond the vote count in Tech Community. I can say on behalf of myself and the other documentation writers that we want you to succeed in building on the Office Platform. The moment this testing work is in a usable state, we'll share it with everyone following this thread, demo it on the community call, and work with the community to ensure the documentation is helpful and useful.

[Rick-Kirkham]

@lafe @giulianozor We now have documentation of unit testing and mocking the Office.js.

Unit testing in Office Add-ins

[rootlocus22]

Is there a Cypress like solution for Office Add In, we need to prepare an end to end integration testing framework to automate our flows?

[igor-ribeiiro]

Is there a Cypress like solution for Office Add In, we need to prepare an end to end integration testing framework to automate our flows?

Unfortunately, we currently don't have any end-to-end testing framework. So we don't have a Cypress-like solution at the current time.

[Rinzwind]

As a few others have commented here on using Selenium WebDriver for UI testing add-ins: one problem we’re bumping into is that Excel on office.com pops up a dialog saying “Looking to share a section of your workbook?” (see screenshot below) which covers the top part of our add-in’s task pane, so when a test tries to click an element there it gets an ‘element click intercepted’ error. As the dialog doesn’t seem to pop up consistently at the same point in the tests, I’m not quite sure yet how to deal with it. I’m wondering whether others have bumped into this and can offer advice? More generally: is there a more suitable place than this issue to ask questions about UI testing add-ins?

[AlexJerabek]

is there a more suitable place than this issue to ask questions about UI testing add-ins?

@Rinzwind, Stack Overflow is the best way to get answers from others in the same situation. Stack's great for the intersection of technologies, as opposed to pure Office.js questions.