C++/CLI - C# Support

[d8ahazard]

Hey there!

I'm trying to figure out a way to wrap this library for use in C#, but having a lot of trouble.

Are you at all familiar with C++/CLI, SWIG, or c# wrappers? If so, would it be possible to provide some kind of extension for use of this in a c# (.net core, specifically) application?

[smasherprog]

Yeah, its not too difficult to create a c# wrapper so that this is available in c#. You'd have to create a c API to call between. This is old, but its still the same way https://www.c-sharpcorner.com/UploadFile/tanmayit08/unmanaged-cpp-dll-call-from-managed-C-Sharp-application/

If you want to do this as a pull request ill work with it on you.

[smasherprog]

Anything on this?

[smasherprog]

Gonna close this if we cant work on it together..??

[rotolonico]

Hey there, not sure if you are still up for it, but I would love to be able to wrap this library in C# because I am trying to set up a screen recording system in Unity using this external cpp library for all standalone builds. Thing is, I know nothing about cpp; I have read online how to incorporate a .dll into Unity/C# but what I would need is a way to set up listeners for screen updates and send the pixels back to C#. Could you get me a .dll that has extern functions that can set up those listeners or just point me in the right direction? That'd be very appreciated.

[smasherprog]

If you are trying to capture the current unity window, theres probably a more efficient way to do that within the unity system itself. This library will likely do the job, but if you are trying to do what I think you are, this library will be less efficient.

[rotolonico]

No, actually I need to capture the entire desktop, not just the game window. I have tried using Unity's ScreenCapturer but that only captures the game window.

[EinarMU]

Hi, I am having the same issue as @rotolonico. I am trying to create a screen capture system inside unity in order to share the desktop or other apps that are open. @rotolonico did you manage to solve this or you still struggling with this?

[smasherprog]

What sort of problems are there? Is it just black always? Have you checked for any hidden errors that could be occuring?

[rotolonico]

Hi, I am having the same issue as @rotolonico. I am trying to create a screen capture system inside unity in order to share the desktop or other apps that are open. @rotolonico did you manage to solve this or you still struggling with this?

Nope, didn't get any luck yet, but I am still looking at it; maybe during the holidays, I'll have more time to work on it. What I tried doing (and failed) was to create a bridge between the .dll in the release and Unity with external functions, because the ones exposed already in the release don't get read by Unity. That said I could be doing something wrong I never did this sort of thing before.

[rotolonico]

What sort of problems are there? Is it just black always? Have you checked for any hidden errors that could be occuring?

It's just making the library work in a Unity environment. Unity currently only officially supports capturing the game window, not the entire screen.

[rotolonico]

Hi, I am having the same issue as @rotolonico. I am trying to create a screen capture system inside unity in order to share the desktop or other apps that are open. @rotolonico did you manage to solve this or you still struggling with this?

Hey @EinarMU , just wanted to update you on this just in case you were still struggling with the problem. I ended up using a different solution by opening a Python server that used the "mss" library for screen capturing and then sending that information to Unity via socket. Perhaps not the solution you are looking for, but it does the job and it should work for all standalone platforms (haven't tested for MacOS tho). I have pushed this code to a public repository and I have added the package I used to the README if you want to take a look.

[ludos1978]

Hi,

I have tried looking into this topic as well. I am not into c++ anymore (20years ago i worked with it) and i struggeld understanding the implications of what is needed.

As you said, it's needed to get the C API working. Also im on osx and i need to generate a dylib. Currently only a ".a" is generated. I was able to create one with "g++ -dynamiclib -undefined suppress -flat_namespace .o -flat_namespace ./ios/.o -o something.dylib" manually right now. It worked with some small example functions i added. However now i am blocked with for example the GetWindows function. When trying to call the function GetWindows Unity crashes with the following message:

0   libscreen_capture_lite.dylib    0x000000015145afd5 std::__1::__vector_base<SL::Screen_Capture::Window, std::__1::allocator<SL::Screen_Capture::Window> >::__vector_base() + 37
1   libscreen_capture_lite.dylib    0x000000015145afa8 std::__1::vector<SL::Screen_Capture::Window, std::__1::allocator<SL::Screen_Capture::Window> >::vector() + 24
2   libscreen_capture_lite.dylib    0x0000000151459c45 std::__1::vector<SL::Screen_Capture::Window, std::__1::allocator<SL::Screen_Capture::Window> >::vector() + 21
3   libscreen_capture_lite.dylib    0x0000000151459831 GetWindows + 945

i assume it's because of the warning

 warning: 'GetMonitors' has C-linkage specified, but returns incomplete type 'std::vector<Monitor>' which could be incompatible with C

after i added "extern "C" {" to the most important parts (Window & Monitor structs and functions).

Do you have any hints where to proceed?

[smasherprog]

You have to create a C interface for everything. Because your are creating a dynamically linked library you cannot pass c++ objects. It must be only basic types: char*, ints. Strings, vectors are not allowed to be passed through the C API.

[ludos1978]

i got this far, but i'm unsure wether i can continue on it. It can read the windows & monitors informations, but it's missing the important and more difficult callback things (screenshots). Remember to restart unity on all changes to the dylib/dll.

// ScreenCaptureInterface.h
// included in ScreenCapture.c & added to CMakeLists where include/ScreenCapture.h is included as well
// also add extern "C" { ... } around all types in ScreenCapture.h from Point until ImageBGRA

#pragma once
#include <assert.h>
#include <chrono>
#include <cstring>
#include <functional>
#include <memory>
#include <string>
#include <thread>
#include <vector>
#include <stdio.h>

#if defined(WINDOWS) || defined(WIN32)
#if defined(SC_LITE_DLL)
#define SC_LITE_EXTERN __declspec(dllexport)
#else
#define SC_LITE_EXTERN
#endif
#else
#define SC_LITE_EXTERN
#endif

#define C_INTERFACE

extern "C" {
namespace SL {
namespace Screen_Capture {

    SC_LITE_EXTERN int C_GetWindows_Count () {
        std::vector<Window> cpp_windows = SL::Screen_Capture::GetWindows();
        return cpp_windows.size();
    }

    SC_LITE_EXTERN void C_GetWindows_Index_Out (int index, Window* window_out) {
        std::vector<Window> cpp_windows = SL::Screen_Capture::GetWindows();
        if (index < cpp_windows.size()) {
            Window w = cpp_windows[index];
            memcpy(window_out, &w, sizeof *window_out);
        }
    }

    SC_LITE_EXTERN int C_GetMonitors_Count () {
        std::vector<Monitor> cpp_monitors = SL::Screen_Capture::GetMonitors();
        return cpp_monitors.size();
    }

    SC_LITE_EXTERN void C_GetMonitors_Index_Out (int index, Monitor* monitor_out) {
        std::vector<Monitor> cpp_monitors = SL::Screen_Capture::GetMonitors();
        if (index < cpp_monitors.size()) {
            // monitor_out = cpp_monitors[index];
            Monitor m = cpp_monitors[index];
            memcpy(monitor_out, &m, sizeof *monitor_out);
        }
    }

    // typedef int ( __stdcall *ANSWERCB )( *ImageBGRA );

    // SC_LITE_EXTERN int TakesCallback( ANSWERCB fp, int n, int m );

    // SC_LITE_EXTERN class C_ICaptureConfiguration {

    }

} // namespace Screen_Capture
} // namespace SL
}

a small build script i used to generate the dylib placed in the screen_capture_lite-master folder. this is specifically for osx.

make && g++ -dynamiclib -undefined suppress -flat_namespace ./CMakeFiles/screen_capture_lite.dir/src/*.o -flat_namespace ./CMakeFiles/screen_capture_lite.dir/src/ios/*.o -o libscreen_capture_lite.dylib && cp libscreen_capture_lite.dylib '/MY_PROJECT_ASSETS_PATH/Plugins/ScreenCaptureLite/libscreen_capture_lite.dylib'

// ScreenCaptureLite.cs 
// added to unity along with the dylib / dll (dylib/dll must be in Plugins)

using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using System.Runtime.InteropServices;

public class TestScreenCaptureLite : MonoBehaviour {

    [DllImport("libscreen_capture_lite")]
    public static extern int add(int a, int b);
    [DllImport("libscreen_capture_lite")]
    public static extern int subtract(int a, int b);

    [StructLayout(LayoutKind.Sequential)]
    public struct Point {
        public int x;
        public int y;
    }

    [StructLayout(LayoutKind.Sequential)]
    public struct Window {
        public uint Handle;
        public Point Position;
    }

    [StructLayout(LayoutKind.Sequential)]
    public struct ImageBGRA {
        public char B;
        public char G;
        public char R;
        public char A;
    }

    [StructLayout(LayoutKind.Sequential)]  
    public struct Monitor {  
        public int Id;
        public int Index;
        public int Adapter;
        public int Height;
        public int Width;
        public int OriginalHeight;
        public int OriginalWidth;
        // Offsets are the number of pixels that a monitor can be from the origin. For example, users can shuffle their
        // monitors around so this affects their offset.
        public int OffsetX;
        public int OffsetY;
        public int OriginalOffsetX;
        public int OriginalOffsetY;
        // https://www.mono-project.com/docs/advanced/pinvoke/
        [MarshalAs (UnmanagedType.ByValTStr, SizeConst=128)]
        public string Name;
        public float Scaling;
    }

    [DllImport("libscreen_capture_lite")]
    public static extern int C_GetWindows_Count();

    [DllImport("libscreen_capture_lite")]
    public static extern void C_GetWindows_Index_Out(int index, out Window window);

    [DllImport("libscreen_capture_lite")]
    public static extern int C_GetMonitors_Count();

    [DllImport("libscreen_capture_lite")]
    public static extern void C_GetMonitors_Index_Out(int index, out Monitor monitor);

    void Start() {
        int wCount = C_GetWindows_Count();
        Debug.Log($"number of windows: {wCount}");
        for (int i=0; i<wCount; i++) {
            Window window = new Window();
            C_GetWindows_Index_Out(i, out window);
            Debug.Log($"Window: handle:{window.Handle} posX:{window.Position.x} posY:{window.Position.y}");
        }

        int mCount = C_GetMonitors_Count();
        Debug.Log($"number of monitors: {mCount}");
        for (int i=0; i<mCount; i++) {
            Monitor monitor = new Monitor();
            C_GetMonitors_Index_Out(i, out monitor);
            Debug.Log($"Monitors Id:{monitor.Id} Index:{monitor.Index} Adapter:{monitor.Adapter}");
            Debug.Log($"- Name:'{monitor.Name}'");
            Debug.Log($"- Height:{monitor.Height} Width:{monitor.Width} OriginalHeight:{monitor.OriginalHeight} OriginalWidth:{monitor.OriginalWidth} Scaling:{monitor.Scaling}");
            Debug.Log($"- OffsetX:{monitor.OffsetX} OffsetY:{monitor.OffsetY} OriginalOffsetX:{monitor.OriginalOffsetX} OriginalOffsetY:{monitor.OriginalOffsetY}");
        }

        // C_CaptureCallback ();
    }
}

[ludos1978]

Hi again, i've still found some more time before starting at my new job. I've gotten it to work so far that i can get an image and store it in a texture in unity. It's only a proof of concept. The Texture is upside down, it's only implemented to get one callback for one window with a very bad indexing access. i think i could push it with the access, but i am not sure wether i could mess up something in doing so. Can send you a zip as well.

[smasherprog]

This is a good start. In order to make the code better, why dont you add an global vector for the windows and monitors so that these can be returned in your function call without needing to get the count seperately.

Ill merge this into the codebase as well and work with you to create a C API. Fork the repo and start work on it and ill work on it with you if you like.

Storing this information globally is so small its worth it because it will make the code simpler.

[ludos1978]

I've added you to the forked repo.

https://github.com/ludos1978/screen_capture_lite

A thing that i didnt figure out yet is how to switch upside-down in the function C_ExtractAndConvertToRGBA in include/ScreenCaptureInterface.h . That would make the screengrabs much more meaningful.

[smasherprog]

I see that.. Ill take a look at the repo today.

[smasherprog]

I did glance at the code so far and if this is going to be merged into this repo, the code must be either: generic for everyone, or specific to unity. I prefer to first try and make it generic, and if its too different or too complicated as a result, then we can make a more specific codepath.

For example, the code to convert the image from or to different formats is beyond the scope of this library. There are simply too many variations that would need to be supported: texture coordinate differences between directx and opengl, little endian vs big endian, and all the different ways that colors can be arranged, RGBA, ARGB, BGRA... an on and on.
Right now, the texture format is the same across all platforms which is good

Also, Id like it if we can use cmake to build the library so its cross platform. So far you have done a great job thought! its really almost done

[d8ahazard]

I did glance at the code so far and if this is going to be merged into this repo, the code must be either: generic for everyone, or specific to unity. I prefer to first try and make it generic, and if its too different or too complicated as a result, then we can make a more specific codepath.

For example, the code to convert the image from or to different formats is beyond the scope of this library. There are simply too many variations that would need to be supported: texture coordinate differences between directx and opengl, little endian vs big endian, and all the different ways that colors can be arranged, RGBA, ARGB, BGRA... an on and on. Right now, the texture format is the same across all platforms which is good

Also, Id like it if we can use cmake to build the library so its cross platform. So far you have done a great job thought! its really almost done

+1 for "Universal". I can help test for non unity.

[ludos1978]

I can fully agree to to cross plattform and generic. But being generic also meaning to support unity (so it runs, not optimized for it). For example some c# Marshaling method is not supported in unity. Converting to different image formats is out of the scope and i dont see any reason to support it. One thing i think is useful is flipping images upside-down, at least if every byte of the image is handled on the c++ side already before sending it to the c-interface.

[smasherprog]

Work has begun on C# and Unity support here https://github.com/ludos1978/screen_capture_lite/pull/1

[smasherprog]

I merged into master my work in progress. I have paused currently on this so anyone who wants to take a stab at implementing more can do so. Ill post again when I resume

[d8ahazard]

I merged into master my work in progress. I have paused currently on this so anyone who wants to take a stab at implementing more can do so. Ill post again when I resume

Tried integrating the dotnet library into my application and mostly using the sample code to create an instance of the Manager, but it just hangs at the createframegrabber() method.

I compiled the example app "screen_capture_example_cpp", which threw an error because the "shared" dlls are missing.

Grabbed the latest release and added all the files to the project dir, and then it compiles properly. But, on execution, I get the following message:

"PS E:\dev\screen_capture_lite\Example_CSharp\bin\Debug\net5.0> .\screen_capture_lite_example_csharp.exe Starting Capture Demo/Test Unhandled exception. System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (0x8007000B)"

A quick google says this could potentially be related to a mismatch between 32-bit and 64-bit libraries. Does this mean I need to compile the shared dll as a 64-bit version, or the dotnet native lib?

[d8ahazard]

Update: Compiled the demo for x86, that got it to run, but it threw this error. It should be noted, this is being run from an RDP session, which is probably the issue...

PS E:\dev\screen_capture_lite\Example_CSharp\bin\Debug\net5.0> .\screen_capture_lite_example_csharp.exe Starting Capture Demo/Test Id = 0 Index = 0 Height = 1080 Width = 1920 OffsetX = 0 OffsetY= 0 Name= \.\DISPLAY561 Fatal error. 0xC0000005 at SL.Screen_Capture+NativeFunctions.isMonitorInsideBounds(Monitor[], Int32, Monitor) at SL.Screen_Capture.isMonitorInsideBounds(Monitor[], Monitor) at screen_capture_lite_example_csharp.Program.Main(System.String[])

[smasherprog]

0xC0000005 is generally a permission error of some sort.

[smasherprog]

This work is about 90% done. The API works exactly the same as c++. The only remaining calls are to mouse capturing which should be done this weekend

[smasherprog]

@ludos1978 Since this is almost done, if you want to work on the unity example, so it can be merged in that would be awesome. I dont use unity. If you do this, please try to make it generic so it can be used by anyone to get started... so Bare bones as possible!

[smasherprog]

All work done.. Fully functional now... Closing . Please open new issues if any bugs are found!