How to get the raw bytes from Texture2D

[keytrap-x86]

Hi,

I would like to try to send the captured Texture2D over the network. For this, I need to send a byte array;

Could you provide an example of how could I get the Bitmap from :

using (var backBuffer = swapChain.GetBackBuffer<SharpDX.Direct3D11.Texture2D>(0))
using (var bitmap = Direct3D11Helper.CreateSharpDXTexture2D(frame.Surface))
{
    d3dDevice.ImmediateContext.CopyResource(bitmap, backBuffer);
    //How to convert the backBuffer to Bitmap or byte[] ?
}

[keytrap-x86]

Okay so I have found a way of converting the backBuffer to Bitmap (or byte array) :

//...
// Get the desktop capture texture
var mapSource = device.ImmediateContext.MapSubresource(screenTexture, 0, MapMode.Read, SharpDX.Direct3D11.MapFlags.None);

// Create Drawing.Bitmap
using(var bitmap = new Bitmap(width, height, PixelFormat.Format32bppArgb)) {
  var boundsRect = new Rectangle(0, 0, width, height);

  // Copy pixels from screen capture Texture to GDI bitmap
  var mapDest = bitmap.LockBits(boundsRect, ImageLockMode.WriteOnly, bitmap.PixelFormat);

  var sourcePtr = mapSource.DataPointer;
  var destPtr = mapDest.Scan0;
  for (int y = 0; y < height; y++) {
    // Copy a single line 
    Utilities.CopyMemory(destPtr, sourcePtr, width * 4);

    // Advance pointers
    sourcePtr = IntPtr.Add(sourcePtr, mapSource.RowPitch);
    destPtr = IntPtr.Add(destPtr, mapDest.Stride);
  }

  // Release source and dest locks
  bitmap.UnlockBits(mapDest);
  device.ImmediateContext.UnmapSubresource(screenTexture, 0);

  using(var ms = new MemoryStream()) {
    bitmap.Save(ms, ImageFormat.Bmp);
    ScreenRefreshed?.Invoke(this, (Bitmap) bitmap.Clone());
    _init = true;
  }
}

The problem is that all this conversion is very costly. My CPU get's up to 30%

Any ways of improving this ?

[robmikh]

It looks like you may be doing multiple copies. Here's the approach I would try:

1. Create a staging texture with the same dimensions as the buffers in the frame pool. Make sure to keep this the same size if you chose to change the buffer size when calling Recreate.
2. After calling TryGetNextFrame from the FrameArrived event handler, copy the frame to the staging texture.
3. Map the staging texture.

From here you can copy out each line to either a new allocation or something you reuse (depending on how you're transmitting the frame). Take a look at this comment from another issue/sample for some sample C++ code.

At the end, unmap the texture.

[keytrap-x86]

@robmihk, thank you for you answer. Although, for me, what you are advising me to do, is already what I am doing. Here's the full sample :

public void Start()
        {
            _run = true;
            var factory = new Factory1();
            //Get first adapter
            var adapter = factory.GetAdapter1(0);
            //Get device from adapter
            var device = new SharpDX.Direct3D11.Device(adapter);
            //Get front buffer of the adapter
            var output = adapter.GetOutput(0);
            var output1 = output.QueryInterface<Output1>();

            // Width/Height of desktop to capture
            int width = output.Description.DesktopBounds.Right;
            int height = output.Description.DesktopBounds.Bottom;

            // Create Staging texture CPU-accessible
            var textureDesc = new Texture2DDescription
            {
                CpuAccessFlags = CpuAccessFlags.Read,
                BindFlags = BindFlags.None,
                Format = Format.B8G8R8A8_UNorm,
                Width = width,
                Height = height,
                OptionFlags = ResourceOptionFlags.None,
                MipLevels = 1,
                ArraySize = 1,
                SampleDescription = { Count = 1, Quality = 0 },
                Usage = ResourceUsage.Staging
            };
            var screenTexture = new Texture2D(device, textureDesc);

            _ = Task.Factory.StartNew(() =>
              {
                  // Duplicate the output
                  using (var duplicatedOutput = output1.DuplicateOutput(device))
                  {
                      while (_run)
                      {
                          try
                          {

                              // Try to get duplicated frame within given time is ms
                              duplicatedOutput.AcquireNextFrame(500, out OutputDuplicateFrameInformation duplicateFrameInformation, out SharpDX.DXGI.Resource screenResource);

                              // copy resource into memory that can be accessed by the CPU
                              using (var screenTexture2D = screenResource.QueryInterface<Texture2D>())
                                  device.ImmediateContext.CopyResource(screenTexture2D, screenTexture);

                              // Get the desktop capture texture
                              var mapSource = device.ImmediateContext.MapSubresource(screenTexture, 0, MapMode.Read, SharpDX.Direct3D11.MapFlags.None);

                              // Create Drawing.Bitmap
                              using (var bitmap = new Bitmap(width, height, PixelFormat.Format32bppArgb))
                              {
                                  var boundsRect = new Rectangle(0, 0, width, height);

                                  // Copy pixels from screen capture Texture to GDI bitmap
                                  var mapDest = bitmap.LockBits(boundsRect, ImageLockMode.WriteOnly, bitmap.PixelFormat);

                                  var sourcePtr = mapSource.DataPointer;
                                  var destPtr = mapDest.Scan0;
                                  for (int y = 0; y < height; y++)
                                  {
                                      // Copy a single line 
                                      Utilities.CopyMemory(destPtr, sourcePtr, width * 4);

                                      // Advance pointers
                                      sourcePtr = IntPtr.Add(sourcePtr, mapSource.RowPitch);
                                      destPtr = IntPtr.Add(destPtr, mapDest.Stride);
                                  }

                                  // Release source and dest locks
                                  bitmap.UnlockBits(mapDest);
                                  device.ImmediateContext.UnmapSubresource(screenTexture, 0);

                                  using (var ms = new MemoryStream())
                                  {
                                      bitmap.Save(ms, ImageFormat.Bmp);
                                      ScreenRefreshed?.Invoke(this, (Bitmap)bitmap.Clone());
                                      _init = true;
                                  }
                              }
                              screenResource.Dispose();
                              duplicatedOutput.ReleaseFrame();
                          }
                          catch (SharpDXException e)
                          {
                              if (e.ResultCode.Code != SharpDX.DXGI.ResultCode.WaitTimeout.Result.Code)
                              {
                                  Trace.TraceError(e.Message);
                                  Trace.TraceError(e.StackTrace);
                              }
                          }
                      }
                  }
              });
            while (!_init) ;
        }

This works of course, but as I said, the CPU gets up to 30%. I'vre tried commenting all ou excep the

var mapSource = device.ImmediateContext.MapSubresource(screenTexture, 0, MapMode.Read, SharpDX.Direct3D11.MapFlags.None);
...
device.ImmediateContext.UnmapSubresource(screenTexture, 0);

But even then, it's quite CPU intensive. Is this the only way of getting the data from the CPU ?

[robmikh]

It still looks like you're making multiple copies to me. I would try this in C++ and see if you get the same result. I'm not sure what some of those helper methods are doing under the covers.

[keytrap-x86]

Could you elaborate ? In which way is this making multiple copies ? When I look at SharpDx's example : This is how a single screen capture works (basic screenshot). I'm just trying to make multiples screenshots and convert them to byte array for sending on the wire.

Also, in this example, everything works fine and there's no performance issue. Now of course, for me, there's no point of just showing the captured screen in a form. I don't understand how this works fine and as soon I try to get a Bitmap from it, the cpu goes crazy

[robmikh]

When you call bitmap.Save(ms, ImageFormat.Bmp), I'm guessing that's making a copy of bitmap into your MemoryStream. I'm also a bit surprised this works... to my eye you're mixing pixel formats (B8G8R8A8_UNorm vs Format32bppArgb).

Keep in mind that the WPF capture sample you're pointing to doesn't try to access the bits in system memory. Everything is kept in video memory, so it will have different performance characteristics from what you're trying to do.

Does this project have the same issue for you? https://github.com/robmikh/ManagedScreenshotDemo

[TKGNET]

Since the OP seems to have abandonded the thread, I wanted to leave my regards for robmikH sample solution. Way faster and definitely more elegant than my own LockBits solution:

• converting stagingResource -> byte array ~ 20ms per screenshot
• BitMapEncoder -> file takes ~ 80ms per file on top

I just wish there was a pure in-memory solution to convert stagingResource to drawing.bitmap , other than file export/import. But I guess writing to an InMemoryRandomAccessStream might do the trick.

[EricBallard]

+1 thanks for the demo