How to begin extending scrcpy features in C#?

[elhaix]

I am ultimately trying to extend the scrcpy functionality through code in VS, C#.

The issue is that when I run the solution, it opens up a new instance of VS and I'm not quite sure how to make use of this extension.

It's supposed to be a wrapper around scrcpy and I see the C# code that does the work. Ultimately I would like to begin extending its current features/functionalities with C# which is what I require assistance with please.

[pvdstel]

The goal of this extension is not to extend the functionality of scrcpy. Its purpose is to run scrcpy within a running Visual Studio instance. I created it so that I could control my phone while debugging and developing Xamarin apps.

By this extension being a wrapper I mean that it wraps the binary that is released by the scrcpy project. It does not provide access to functions and/or features that scrcpy provides.

[elhaix]

Really appreciate the quick response. What's the process to get it running in VS please? When I build/run the code it opens a new VS window.

Curious as to the reason for wanting to control your phone from within VS, when using scrcpy does that already. Do you get data communication between your new apps under development and your phone using your wrapper? Looking for the use cases.

I'd like to wrap scrcpy inside a new desktop application that I'm building. Is that possible without using their C source code?

[pvdstel]

When I build/run the code it opens a new VS window.

That's by design. When you build an extension for Visual Studio (or any IDE really), you need the actual IDE to test your extension. So it opens an experimental Visual Studio instance. You can open scrcpy there by navigating through View -> Other windows -> scrcpy, or Ctrl + Q and typing "scrcpy". You should get something like this then:

Curious as to the reason for wanting to control your phone from within VS, when using scrcpy does that already. Do you get data communication between your new apps under development and your phone using your wrapper? Looking for the use cases.

Okay so there's not much to this, frankly. The reason I built this is so that I didn't have to Alt + Tab between scrcpy and Visual Studio, and have the ability to see my phone's screen while debugging. So basically laziness ¯\_(ツ)_/¯

I'd like to wrap scrcpy inside a new desktop application that I'm building. Is that possible without using their C source code?

I suppose so. The technique I'm using is launching scrcpy, taking the main window that the application produces and embedding that main window in a control that my code owns. It may not be optimal, but it works well enough I guess. If you decide to replicate this, you may want to look at WindowHost for some inspiration. Note that the code is subject to the Mozilla Public License 2.0, should you copy it.

[elhaix]

So within the window your code owns, you can apply whatever layers you want to interact with the device. For example, what if I wanted to record a functional script of scrolling up and down, ensure certain items exist on the screen, etc. I don't need to modify in any way the way the application currently works for that. So embedding it... would that do the trick?

Again, not optimal, but one option, right?

[elhaix]

In WindowHost.xaml.cs, where/how would I go about adding say a layer or buttons (just simple to begin with) to the interface?

[pvdstel]

So within the window your code owns, you can apply whatever layers you want to interact with the device. For example, what if I wanted to record a functional script of scrolling up and down, ensure certain items exist on the screen, etc. I don't need to modify in any way the way the application currently works for that. So embedding it... would that do the trick?

Again, not optimal, but one option, right?

Yes, you can add layers on top of the application in theory. I wonder how useful that would be though. Although the application window is now embedded in a control that the extension creates programmatically, it is still running in a separate process. That means you won't be able to do anything interesting with the scrcpy process.

The current setup doesn't allow for controlling Android from .NET (and it doesn't need to). It might be possible to send keyboard/mouse input events to the scrcpy process, but that's really not a good idea so please don't.

In WindowHost.xaml.cs, where/how would I go about adding say a layer or buttons (just simple to begin with) to the interface?

Check out WindowHost.xaml (without the extra .cs). It's a WPF user interface. You should be able to add stuff there.

[elhaix]

What would it take to add a menu bar to the extension that I can extend?

I'm thinking more for functional testing. Let's say we had a menu bar option to begin recording, and you go through your activities on the phone, end recording, then replay, so the extension would essentially replay the activities on the underlying scrcpy app, which is running some app on the phone that I'm testing. Then I can track say, database hits, etc., on the app server side. Make sense?

[pvdstel]

I see where you're going with functional testing. However, recording such events at the extension level is very likely to be difficult and unreliable. Recording and then replaying would require support from scrcpy itself.

Frameworks and tools for automated testing in this fashion already exist -- it might be useful to check those out.