How to handle MemoryData images?

[mikebm]

Hello,

There is support for NativeImageStreams, using GetNativeImageStream, but there doesn't seem to be any logic around how to handle MemoryData. I have a device that is sending a TIFF according to the MemoryInfo, but am not sure how to parse the bytes as the bytes is not in what appears to be a TIFF format.

Side topic - I am also seeing double transfers on this device, it appears one might be the preview, and the second is the real image. Is there a way to determine which MemoryData is the preview and which is not? They both come at the same time one after the other.

Thanks.

[soukoku]

I actually don't know how to piece together the memory transfer tiles myself so I'm no help on that front.

As for the double image, I imagine the preview image could be a smaller size or resolution? See if you can tell with ImageInfo on the event. Or maybe it has a fixed order like preview-real-preview-real so you could guess it that way?

[mikebm]

Hey, thanks for the response. I now suspect it is one image actually coming over in two buffers via DataTransfer events. The reason I suspect this is that the first is exactly 4096kb. I've tried each individually and both combined and neither seem to be an actual image. It says it's a TIFF in the MemoryInfo, but doesn't seem to be. I tried passing the buffer to the native stream function and it couldn't decipher it as a bitmap or tiff either. I'll keep digging on this but yeah the data seems really weird. I need to find a simulator that can do memory buffer transfers. I am having to do a remote session with someone to test this and it's not been going well, hah.

Library has been working great with some minor modifications. Really appreciate what you've put into it.

I'll share my findings when I figure it out.

[mikebm]

I got it working... partially at least.

I modified DataTransferredEventArgs to take a MemoryStream for the MemoryData. I also remove the MemoryInfo, because I changed it to buffer all the image chunks into a single MemoryStream buffer, then fire the DataTransferred event.

The reason for this change, is the consumer has no idea when the transfer starts/ends, so its easier to deal with when it has the whole buffer in place.

To do this, I modified DoImageMemoryXFer to look like this: (Notice the memory stream and re-location of DoImageXferredEventRoutine

  static ReturnCode DoImageMemoryXfer(ITwainSessionInternal session)
  {
      TWSetupMemXfer memInfo;
      ReturnCode xrc = session.DGControl.SetupMemXfer.Get(out memInfo);
      if (xrc == ReturnCode.Success)
      {
          TWImageMemXfer xferInfo = new TWImageMemXfer();
          try
          {
              // how to tell if going to xfer in strip vs tile?
              // if tile don't allocate memory in app?

              xferInfo.Memory = new TWMemory
              {
                  Flags = MemoryFlags.AppOwns | MemoryFlags.Pointer,
                  Length = memInfo.Preferred,
                  TheMem = PlatformInfo.Current.MemoryManager.Allocate(memInfo.Preferred)
              };

              using MemoryStream outputStream = new MemoryStream();
              do
              {
                  xrc = session.DGImage.ImageMemXfer.Get(xferInfo);

                  if (xrc == ReturnCode.Success || xrc == ReturnCode.XferDone)
                  {
                      if (session.State != 7)
                      {
                          session.ChangeState(7, true);
                      }

                      // optimize and allocate buffer only once instead of inside the loop?
                      byte[] buffer = new byte[(int)xferInfo.BytesWritten];

                      IntPtr lockPtr = IntPtr.Zero;
                      try
                      {
                          lockPtr = PlatformInfo.Current.MemoryManager.Lock(
                              xferInfo.Memory.TheMem
                          );
                          Marshal.Copy(lockPtr, buffer, 0, buffer.Length);
                          outputStream.Write(buffer, 0, buffer.Length);
                      }
                      finally
                      {
                          if (lockPtr != IntPtr.Zero)
                          {
                              PlatformInfo.Current.MemoryManager.Unlock(xferInfo.Memory.TheMem);
                          }
                      }
                  }
              } while (xrc == ReturnCode.Success);
              outputStream.Position = 0;

              DoImageXferredEventRoutine(session, IntPtr.Zero, outputStream, null, (FileFormat)0);

              HandleReturnCode(session, xrc);
          }
          catch (Exception ex)
          {
              session.SafeSyncableRaiseEvent(new TransferErrorEventArgs(ex));
          }
          finally
          {
              session.ChangeState(6, true);
              if (xferInfo.Memory.TheMem != IntPtr.Zero)
              {
                  PlatformInfo.Current.MemoryManager.Free(xferInfo.Memory.TheMem);
              }
          }
      }
      return xrc;
  }

Now for the fun part... I haven't found a simulator that can do any CompressionType's other than None, so I was able to implement that. I also use SkiaSharp to handle my images.

The main issue is that SkiaSharp doesn't support 24bit images. It supports 32bit though, so I had to convert the pixel data over to 32bit. That looked like this in the DataTransferred event handler.

if (e.MemoryStream != null)
        {
            if (e.ImageInfo.Compression == CompressionType.None)
            {
                var info = new SKImageInfo(e.ImageInfo.ImageWidth, e.ImageInfo.ImageLength, SKColorType.Rgba8888, SKAlphaType.Premul);

                if (e.ImageInfo.BitsPerPixel == 24)
                {
                    // Convert 24bit pixel data to 32bit.
                    try
                    {
                        IntPtr dataPtr = Convert24BitTo32Bit(e.MemoryStream);
                        img = new SKBitmap(info);
                        img.InstallPixels(info, dataPtr, e.ImageInfo.ImageWidth * 4, delegate { Marshal.FreeHGlobal(dataPtr); }, null);
                    }
                    finally
                    {
                        e.MemoryStream.Dispose();
                    }
                }
            }

            if (img?.IsNull == true)
            {
                img.Dispose();
                img = null;
            }
        }

I still need to implement and test other bit formats, plus other compression types, but its a good start.

The function I wrote to convert the bits over is as follows. Yes, it could be optimized to use SIMD instructions, but this works well for now.

    private unsafe IntPtr Convert24BitTo32Bit(MemoryStream ms)
    {
        // Read the bytes from the MemoryStream
        byte[] rgbData = ms.ToArray();

        int bufferSize = rgbData.Length * 4 / 3;

        // Allocate a buffer for the converted data
        IntPtr buffer = Marshal.AllocHGlobal(bufferSize);

        Span<byte> span = new Span<byte>(buffer.ToPointer(), bufferSize);

        for (int i = 0; i < rgbData.Length; i += 3)
        {
            int pixelIndex = i / 3; // Calculate the pixel index from the byte index
            byte r = rgbData[i];
            byte g = rgbData[i + 1];
            byte b = rgbData[i + 2];
            byte a = 255;

            int spanIndex = pixelIndex * 4;
            span[spanIndex] = r;
            span[spanIndex + 1] = g;
            span[spanIndex + 2] = b;
            span[spanIndex + 3] = a;
        }
        return buffer;
    }