Add support for sharing the pass/fail results via VSCode Live Share

[orta]

Issue description or question

It would be awesome if the red/green states of a test can be passed across sessions using Live Share. If you've not used it, it's like pairing programming on steroids.

Potentially, you could even have that as a separate extension that anyone could install - so anyone can participate (without paying for a wallaby license) and then be like damn, ok, I'm gonna buy this after they've used it via someone else in pairing (sort of like Mario Kart on the Nintendo DS)

Code editor or IDE name and version

Visual Studio Code v1.x

OS name and version

n/a

/cc https://github.com/jest-community/vscode-jest/issues/369

[dandevs]

Passing the test state is a good workaround for https://github.com/wallabyjs/public/issues/1653#issuecomment-388699692

I've been really wanting to use wallaby with other people to hack something together or show what wallaby can do. I've thought of using Live Share to showcase what wallaby is all about to my friends but that sadly doesn't work.

If Live Share support gets implemented, that'd be super awesome.

[lostintangent]

@dandevs The Live Share team would love to support this 😃 We released our extensibility API to enable scenarios like this, where an extension needs file access, but it may actually be more efficient for “guests” to simply synchronize minimal state with their “host”.

We recently worked with the author of the VS Code Test Explorer extension, and we’d love to work with the Wallaby community to light up support for them as well. It’s been a common request from our users.

[smcenlly]

@lostintangent Can you please provide details of the extensibility API? We'd love to be able to provide this functionality for Live Sharing sessions without having to install another plugin, is that possible today?

[lostintangent]

@smcenlly The Live Share extensibility API is available here. That module handles detecting whether the end-user has the Live Share extension installed/enabled, and among other things, allows VS Code extensions to exchange RPC messages between the host and all guests. For example, this would allow the guest to pull test status from the host when they first join, and then synchronize test changes that occur within the collaboration session.

Does that seem sufficient/interesting? We’re working with a handful of other extension authors to make their experiences collaborative, and it would be great to get your team’s feedback as well. This would also address a feature request of ours, and allow us to recommend folks check out Wallaby 👍

Note, in order for an extension to be able to make use of the Live Share RPC channel, you have to set the “liveshare.features” setting to “experimental” in VS Code. If you decide to look into this, you’ll need to set that while developing. You won’t need this setting in order to ship publically though.

[smcenlly]

@lostintangent, thanks for the quick reply. This does look interesting. We'll take a look at this over the next week.

[lostintangent]

@smcenlly Sounds good! Let me know if you run into any questions or issues.

[lostintangent]

@smcenlly Hey! Did you get a chance to look into this yet? Would love to hear your thoughts/feedback. We had a few other Live Share users this week ask about Wallaby support, so I'm keen to discuss it further.

[smcenlly]

Hi @lostintangent, we haven't had a chance to investigate yet, but it's on our list (3rd in our queue right now). I'll make a note to get back to you by the end of next week on where we're at re: investigation/progress.

[smcenlly]

Hi @lostintangent, we started looking at the Live Share extensibility API yesterday. We're going to focus on our other product, Quokka.js first as it'll be a little easier. We will learn from that before starting on Wallaby, which has more complex use cases.

Having spent 2 days on the integration so far, there are a few general existing feature requests that would be nice to have for our users: (1) Allow sharing unsaved files, (2) Force users to be pinned, (3) Allow hosts to share files outside of folder/solution root.

As far as the extensibility API goes, there's only one item I've been looking for so far. Is there a way to identify whether a user is currently following the host? We're thinking we might like to only display the IDE hints when a user is in the same file as the host. We can manage that ourselves but would be nice if there was a first class feature for that.

[lostintangent]

@smcenlly That sounds great! As you probably saw, one of the top reasons we want to allow sharing unsaved files is to better enable Quokka 😁 We’ll get to this soon, but it would be great to work with you in the meantime.

Regarding the ability to detect if someone is following the host: we don’t currently expose that via the extensibility API, but this is something we plan to address. If you can work around it in the meantime, that would be great. And we can work with you on the first-class support, to ensure we meet your requirements.

[smcenlly]

Hi @lostintangent, a couple more questions from today. Going to continue on this thread if you don't mind. If you'd prefer for us to communicate over email/slack or raise issues in the live-share docs repo, just let us know.

1) We're concerned about security, especially for Quokka and Wallaby where other users who have edit ability can easily edit and run arbitrary code on the host computer. We've implemented security warnings and will ask the user to allow before we execute our plugin. It would be nice to suppress these if the session is read only. We would like to be able to query the session's read-only state (i.e. access IsReadOnly property) and be able to subscribe to change of IsReadOnly.

2) We provide links in our output window that allow users to navigate to the offending file:line:column where their tests are breaking, or where errors occur. We tried to launch using the vsls://filePath that we get from document.uri but received an error that we need to provide an absolute path. What's the best way to go about doing this?

[lostintangent]

@smcenlly Chatting here works for me! Though feel free to ping me over e-mail as well (joncart@microsoft.com). In particular, I'd love to help dogfood the experience if that would be valuable :)

Regarding your questions:

1. This isn't possible today, but it's something we plan to address. I'll file a tracking issue on our GitHub repo, and we can try to address is soon. It sounds like it isn't immediately blocking you? But would optimize the experience for read-only workspaces?

2. Are you using the Live Share APIs for converting URLs, or are you contsructing the vsls URI manually? If you do the former, we should produce an absolute path that can be used with any VS Code API. Let me know if that isn't the case, and which VS Code APIs you're getting errors with, and we can investigate.

Thanks!

[smcenlly]

@lostintangent

RE: #1 - This is not blocking. We just display a warning and force user to allow every time they launch instead of just read-write mode.

RE: #2 - Haven't been able to get this working:

Background Quokka/Wallaby output to the console on the host computer. We add a new output window on each of the peers as part of the Live Share integration.

Use Case As a peer, I want to be able to use a hyperlink from the console output to navigate and open one of the live-share project files.

Error details The Live Share APIs appear to only be available on the host. When we use them on the client, we receive an error: "Only the host role can convert shared URIs." at LiveShareApi.convertSharedUriToLocal.

Happy to share the experience with you before we release it for some feedback. I'll shoot you an email.

[joshsten]

+1 This would be a significant improvement for both products.

[jackwkinsey]

I'd love to know if there's been any more work done on this feature. I reached out yesterday via Twitter and was directed here.

Our company provides Wallaby licenses for our developers and many of them use the Live Share functionality to pair program on a near daily basis. This allows whoever is hosting the session to have Wallaby output, but the partner is essentially not getting the benefit of their license at all.

The workaround we've been using is to throw up the host's screen on a large TV so others can see, but this isn't always feasible.

[smcenlly]

Hi @jackwkinsey - thanks for contacting us.

We scoped this work quite some time ago and it will require us to almost rewrite our VS Code extension and make large-scale changes to Wallaby. Since that time, we've been focused on other features that we feel will have a broader impact (e.g. automatic configuration, time travel debugger, and others). While we added VS Code live-share support for our other product (Quokka), that was relatively simple in comparison to adding support for Wallaby. Our Wallaby VS Code integration tracks open files in your editor and only communicates the relevant changes in your tests/code files to/from VS Code as your tests run (effectively a client-server model). To support Wallaby live-share, we'd need to implement another client-server model on top of that for communicating to the live share clients which will involve some large scale changes to how Wallaby operates.

Right now we don't have immediate plans to implement this feature but will re-evaluate during our next feature planning session. In the meantime, please ask your team to like the first comment in the issue so we can gauge interest. Our question on twitter (re: use case/frequency of use) is for us to try and understand customer demand as input to our planning (e.g. how many hours a day across how many of our users).

Thanks for your feedback so far.

[jackwkinsey]

Hello @smcenlly,

Thanks for the update! I understand the undertaking this would be as it definitely isn't insignificant. We have a decent enough workaround by having the host air play their screen to a TV so the partner can see the test output in real time. It's not super ideal, but I think it is good enough for now.

We do hope that this will eventually be implemented, but I definitely understand that it is lower priority. I assume if any progress is made, it will be reflected here?

Thanks for the great product and the quick response!

Cheers 🍻

[smcenlly]

@jackwkinsey - yes, the issue will be updated as/when things progress

[mlewando]

Hi guys :) We're evaluating wallaby right now in our company. Overall the developer experience with your tool is awesome :) but we'd love to have live share integration to fully benefit from your product. We're doing pair programming on daily basis, that's actually our default way of working. During COVID pandemic, almost all pair programming in the company is done via live share. So live share support is rather crucial for us.

I get from the discussion, that it's quite an investment for you, but I'd like to ask, what's the current priority of this feature (if any)? Any plans of implementing it in the foreseeable future?

[smcenlly]

@mlewando - we did a bit of research about 6 months ago and it looks like there would be fairly limited number of our users with Wallaby and Live Share compared to our overall user base (remember, we support other editors and also, not everyone is using live share) so we decided to prioritize some other features. Would you mind emailing us and telling us a little about your team size, etc. (hello@wallabyjs.com). If you have a large team, it may make sense for us to add sooner rather than later. Also, please have your team thumbs up the first post in issue; when prioritize we do consider how many likes / +1s an issue has.