AsImage / AsTexture DX11

[elliotwoods]

Hey @mrvux , @smakhtin

Vux wrote both AsTexture and AsImage during Node13. I can't find copies of these files at the moment, do you have them?

Vadim has also imlpemented AsTexture, and i've merged his version in at https://github.com/elliotwoods/VVVV.Packs.Image/blob/master/src/nodes/plugins/Image/OpenCV/src/Destinations/AsTextureDX11Instance.cs

If the files are lost then i'll have a go at implementing AsImage myself later today.

[smakhtin]

@elliotwoods My AsTexture is based in Vux's one, i just fixed color space bug.

This is all 3 files, that he sent to me https://gist.github.com/smakhtin/08c8eaae8e5db470685e. I think my AsDX11Texture is ok, VideoIn(DX11) already in core repo, so you need to check only FromDX11TextureInstance.cs.

[elliotwoods]

i'm curious if there's any issue at all for users i've got a feeling that .net will just throw an exception to vvvv when it tries to inspect the classes, so it'll just ignore those nodes from the NodeBrowser, and will fail to instantiate them (if the nodes are already saved in a patch for example).

for developers, i think it makes most sense for the time being to reference the dx11 binaries (as these are more common to have available, and are in a static relative location).

[elliotwoods]

@smakhtin - this can't be right, 'FromTexture' has a function which basically tries to fill an image with random values. There is some reference to DX11 in the node, but not the instance.

[smakhtin]

Yep, maybe we can just replace it by camera data. Not tested yet.

[elliotwoods]

i'm talking about AsImage (which is called FromTexture in that link). currently i have no example of how to perform DX11.Texture2D -> CVImage

[smakhtin]

Yep, you just need to send the correct data to FOutput, not the random one ). If you want, just assign this issue to me, I will fix it in next commits.

[elliotwoods]

Yep, you just need to send the correct data to FOutput, not the random one ).

the code isn't there though.

[sebllll]

i managed to compile the whole pack today. what was not that easy... perhaps there should be some more info in the readme). but the AsTexture (DX11) does not work together with VideoIn (CLeye), while it does work with the directshow videoIn for example.

do you have an idea about the reason for this behaviour?

[elliotwoods]

hey! if you remember any of the troubles that you had setting up, please dump them into a github issue that'd be super valuable (generally there's a lot of things that happen whilst setting up the environment which only happen at that time, and you forget about them for the rest of the time).

not sure what could be causing your issue since the image data should be anonymous as it gets passed around. perhaps CLEye's videoinput is not calling .Send() on it's output?

On 6 August 2013 17:14, sebl notifications@github.com wrote:

i managed to compile the whole pack today. what was not that easy... perhaps there should be some more info in the readme). but the AsTexture (DX11) does not work together with VideoIn (CLeye), while it does work with the directshow videoIn for example.

do you have an idea about the reason for this behaviour?

— Reply to this email directly or view it on GitHubhttps://github.com/elliotwoods/VVVV.Packs.Image/issues/10#issuecomment-22191084 .

Elliot Woods elliot elliot@kimchiandchips.com@KimchiAndChips.comhttp://www.kimchiandchips.com/

[sebllll]

just to make this clear: cleyes videoinput works perfectly with asTexture (dx9).

see the other issue for my "instructions" :)

[elliotwoods]

ah. this could be something really stupid like not supporting monochrome or color images with dx11. can you try a mono/color image?

[sebllll]

isn't working either. i modified your helpfile so that we can have the exact same test: https://github.com/sebllll/VVVV.Packs.Image/blob/d6feeb0e56af426c7919561a59a9c4184f750bb3/lib/nodes/plugins/VideoIn%20(CLEye)%20help.v4p

perhaps it isn't working only on my machine/setup...

[elliotwoods]

can you try an Info (OpenCV) node and an Info (DX11.Texture2D) and screenshot the results?

On 6 August 2013 18:57, sebl notifications@github.com wrote:

isn't working either. i modified your helpfile so that we can have the exact same test: https://github.com/sebllll/VVVV.Packs.Image/blob/d6feeb0e56af426c7919561a59a9c4184f750bb3/lib/nodes/plugins/VideoIn%20(CLEye)%20help.v4p

perhaps it isn't working only on my machine/setup...

— Reply to this email directly or view it on GitHubhttps://github.com/elliotwoods/VVVV.Packs.Image/issues/10#issuecomment-22198102 .

Elliot Woods elliot elliot@kimchiandchips.com@KimchiAndChips.comhttp://www.kimchiandchips.com/

[sebllll]

et voilà

[sebllll]

ah, and i got an error in the TTY when connecting the AsTexture (DX11) node upwards:

00:00:45 ERR : System.NullReferenceException in VVVV.Nodes.OpenCV: Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.
Stacktrace:
   bei VVVV.Nodes.OpenCV.AsTextureDX11Instance.CreateTexture(DX11RenderContext context) in c:\vvvv\vvvv-sdk\vvvv45\VVVV.Packs.Image\src\nodes\plugins\Image\OpenCV\src\Destinations\AsTextureDX11Instance.cs:Zeile 145.
   bei VVVV.Nodes.OpenCV.AsTextureDX11Node.Update(IPluginIO pin, DX11RenderContext context) in c:\vvvv\vvvv-sdk\vvvv45\VVVV.Packs.Image\src\nodes\plugins\Image\OpenCV\src\Destinations\AsTextureDX11Node.cs:Zeile 48.
   bei VVVV.DX11.Lib.RenderGraph.DX11DeviceRenderer.ProcessNode(DX11Node node)

but the corresponding lines don't tell me enough... https://github.com/elliotwoods/VVVV.Packs.Image/blob/master/src/nodes/plugins/Image/OpenCV/src/Destinations/AsTextureDX11Instance.cs#L145 https://github.com/elliotwoods/VVVV.Packs.Image/blob/master/src/nodes/plugins/Image/OpenCV/src/Destinations/AsTextureDX11Node.cs#L48

[sebllll]

ok, next one:

perhaps we can cut it down to the case if the cvImageLink transports a spread of textures?

fun fact: the color difference in dx11 (is that because of the traditional bgr usage in opencv)

[elliotwoods]

haha. interesting. i'm going to have a vvvv dev environment setup again either friday this week or tuesday next week. can it wait until then?

chances are it's a spread thing. also that's BGR vs RGB happening. i'll check that out too.

[sebllll]

ai, nice. get in touch with me if i can help.

[smakhtin]

Yep, look like BGR problem.

@sebllll You can write a simple shader, to fix the color space. Just revert the order of colors. @elliotwoods Should we make a convention about color output for future plugins?

[sebllll]

Ok, i narrowed this down to 2 problems:

• [ ] AsTextureDX11Instance.cs can't deal with RGBA8-Textures (or L8 or anything different from RGB8)
• [ ] AsTextureDX11Node.cs only takes the first input slice > there can't be spreads of images converted :)

while the second one might be a bit simpler to solve, i can't figure out the troublemaker that causes the first problem. though i vaguely think, that CreateTexture() might be the problem

Still there is this secondary rgb/bgr thingy that maybe should be discussed in a broader way, talking about conventions. this is easily fixable as both of you mentioned

[mrvux]

When you copy data to texture, be careful of pitch (specially with L8). When you call texture.WriteData it does a fast copy and assumes image pitch = texture pitch, wich is not always the case. you also have texture.WriteDataPitch which will take care of this (but is a little bit slower, not that bad still).

Here is my vlc code by the way to deal with that (for R8 remove the *4)

            if (vlc.Width * 4 == texture.GetRowPitch())
            {
                texture.WriteData(ptr, vlc.Width * vlc.Height * 4);
            }
            else
            {
                texture.WriteDataPitch(ptr, vlc.Width * vlc.Height * 4);
            }

Also you can set a texture to SlimDX.DXGI.Format.B8G8R8A8_UNorm , dx device has bgr support enabled (please note that you're not allowed to use this as render target, but it's fine as dynamic texture).

[sebllll]

puh, that one helped me just a tiny bit, but increased my understanding.

i managed to manually fix the CLeye ImageLink (L8) > AsTexture (DX11), but i don't understand where the error is exactly.

seems that this piece doesn't work properly:

CVImageAttributes attr = FBufferConverted.ImageAttributes.Clone() as CVImageAttributes;

https://github.com/sebllll/VVVV.Packs.Image/commit/bc2b4d55450e224a432947cf8e330febe7d8a97b

[elliotwoods]

hmm.. i see the error (i didn't write this function btw, so sorry for not realising this error earlier).

there should be an else block after: https://github.com/elliotwoods/VVVV.Packs.Image/blob/master/src/nodes/plugins/Image/OpenCV/src/Destinations/AsTextureDX11Instance.cs#L45 which allocates for when no conversion is necessary.

On 16 August 2013 12:30, sebl notifications@github.com wrote:

puh, that one helped me just a tiny bit, but increased my understanding.

i managed to manually fix the CLeye ImageLink (L8) > AsTexture (DX11), but i don't understand where the error is exactly.

seems that this piece doesn't work properly:

CVImageAttributes attr = FBufferConverted.ImageAttributes.Clone() as CVImageAttributes;

— Reply to this email directly or view it on GitHubhttps://github.com/elliotwoods/VVVV.Packs.Image/issues/10#issuecomment-22760968 .

Elliot Woods elliot elliot@kimchiandchips.com@KimchiAndChips.comhttp://www.kimchiandchips.com/

[sebllll]

whoop, got it almost done.

i added the else block as you mentioned before:

else
{
    FBufferConverted = new CVImageDoubleBuffer();
    FBufferConverted.Initialise(new CVImageAttributes(FInput.ImageAttributes.Size, FInput.ImageAttributes.ColourFormat));
}

the only issue now is at the time of connecting a second AsTexture (DX11) there's an error:

00:00:38 ERR : System.ArgumentException in mscorlib: Ein Element mit dem gleichen Schlüssel wurde bereits hinzugefügt.

Stacktrace:
   bei System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
   bei System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   bei SlimDX.ObjectTable.Add(ComObject comObject, ComObject owner)
   bei SlimDX.ComObject.Construct(IUnknown* pointer)
   bei SlimDX.Direct3D11.ShaderResourceView..ctor(Device device, Resource resource)
   bei FeralTic.DX11.Resources.DX11DynamicTexture2D..ctor(DX11RenderContext context, Int32 width, Int32 height, Format format)
   bei VVVV.Nodes.OpenCV.AsTextureDX11Instance.CreateTexture(DX11RenderContext context) in c:\vvvv\vvvv-sdk\vvvv45\VVVV.Packs.Image\src\nodes\plugins\Image\OpenCV\src\Destinations\AsTextureDX11Instance.cs:Zeile 178.
   bei VVVV.Nodes.OpenCV.AsTextureDX11Node.Update(IPluginIO pin, DX11RenderContext context) in c:\vvvv\vvvv-sdk\vvvv45\VVVV.Packs.Image\src\nodes\plugins\Image\OpenCV\src\Destinations\AsTextureDX11Node.cs:Zeile 49.
   bei VVVV.DX11.Lib.RenderGraph.DX11DeviceRenderer.ProcessNode(DX11Node node)

but i think, that must be fixed in the AsTextureDX11Node.cs.

thank you all so far!

[smakhtin]

@sebllll Can you make a pull request with this fix? I will check the new error, it's hard to understand non english stack traces.

I also can fix the spreads issue, looks like i just forgot to add for loop, after debugging.

@elliotwoods I think we definitely should make a color format convention, because without it we will have too many bugs like this. I think RGB8 is optimal, why we need alpha channel in camera streams?

[sebllll]

hi @smakhtin i can surely make a PR tomorrow. for now, the stacktrace says something like: An element with the same key has been already added.

i already kind of fixed the rgb/bgr thingy here https://github.com/sebllll/VVVV.Packs.Image/blob/master/src/nodes/plugins/Image/OpenCV/src/Destinations/AsTextureDX11Instance.cs#L126

[elliotwoods]

hey. ok i'll pull your copy and do some fixes myself then pull into master looking at this today

[elliotwoods]

@sebllll - i notice you've got some other changes in your master branch at present. anything else you want pulling in?

[sebllll]

hi @elliotwoods , sorry i'm not on my dev-machine today.

but i try to explain the changes:

https://github.com/sebllll/VVVV.Packs.Image/commit/bc2b4d55450e224a432947cf8e330febe7d8a97b#L1R112 just added L8-Mode (when CLeye is set to a Mono-Mode, it outputs L8)

https://github.com/sebllll/VVVV.Packs.Image/commit/b379955a4ee9bd3242eafc9e2a88e95b63505e50#L1R126 that fixed rgb/bgr conversion (should be done for the other modes as well...)

https://github.com/sebllll/VVVV.Packs.Image/commit/b379955a4ee9bd3242eafc9e2a88e95b63505e50#L1L49 that one was your tip with the else statement

https://github.com/sebllll/VVVV.Packs.Image/commit/b379955a4ee9bd3242eafc9e2a88e95b63505e50#L1R158 here i "fixed" the error itself

https://github.com/sebllll/VVVV.Packs.Image/commit/b379955a4ee9bd3242eafc9e2a88e95b63505e50#L2R42 that was just a test, because i didn't understand it. it accidentially commited

sorry, the whole commit is a bit messy, because i did an "amend commit" though it was pushed already...

[elliotwoods]

sure, no worries. i've pulled in your master branch for the 2 relevant files and am going from there i'm recoding them quite a bit. will upload dll's to my server for testing when you can

[elliotwoods]

ok. i've done a pretty big rewrite and it's not working!

@mrvux - is there any kind of single slice restriction with Pin<DX11Resource<DX11DynamicTexture2D>> that i'm not aware of?

i've pushed my current changes to https://github.com/elliotwoods/VVVV.Packs.Image/blob/master/src/nodes/plugins/Image/OpenCV/src/Destinations/AsTextureDX11Node.cs https://github.com/elliotwoods/VVVV.Packs.Image/blob/master/src/nodes/plugins/Image/OpenCV/src/Destinations/AsTextureDX11Instance.cs

any chance for someone to give a second looksie?

it works with single slices, and as far as i can see, it should work with many slices also.

[sebllll]

hm, it doesn't work here for rgba8 textures. only for rgb8 ones (if i hit the refresh button on imageload).

the error for the rgba8 looks like a flaw in the FNeedsRefresh dictionary:

00:00:57  *  : [graph event] unexpected error: Access violation at address FF000204. Read of address FF000204
00:00:57  -  : To help us track this error down, enable the ExceptionDialog via the menu or starting vvvv with /showexceptions.
00:00:57 ERR : System.Collections.Generic.KeyNotFoundException in mscorlib: Der angegebene Schlüssel war nicht im Wörterbuch angegeben. (= The given key was not specified in the dictionary.)

Stacktrace:
   bei System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   bei VVVV.Nodes.OpenCV.AsTextureDX11Instance.UpdateTexture(DX11RenderContext context) in c:\vvvv\vvvv-sdk\vvvv45\VVVV.Packs.Image\src\nodes\plugins\Image\OpenCV\src\Destinations\AsTextureDX11Instance.cs:Zeile 121.
   bei VVVV.Nodes.OpenCV.AsTextureDX11Node.Update(IPluginIO pin, DX11RenderContext context) in c:\vvvv\vvvv-sdk\vvvv45\VVVV.Packs.Image\src\nodes\plugins\Image\OpenCV\src\Destinations\AsTextureDX11Node.cs:Zeile 49.
   bei VVVV.DX11.Lib.RenderGraph.DX11DeviceRenderer.ProcessNode(DX11Node node)

but, the good point is: spreads are working (if they are not rgba8)

[smakhtin]

@elliotwoods What is the current state of this? I currently started to code another CV project, so can check this bug.

[elliotwoods]

@smakhtin - i recoded AsTextureDX11 (instance+node) i'm encountering some bugs especially with spreads. (perhaps because i'm using ISpread instead of Pin?) if you could try this out that'd be great! check sebl's notes in his last post esp re rgba8

[smakhtin]

Ok, i will check it, thx. Currently checking my old version of AsDX11 texture - buggy as hell ).

[smakhtin]

I currently testing AsTexture(DX11) for 1 image. Looks like L8 images not working, trying to figure out why.

[elliotwoods]

i think i've got this sorted now i've had a proper look. there was a bunch of really glaring errors and some quite subtle ones also

please test @sebllll if you have time.

[elliotwoods]

p.s. i'll reopen if this still doesn't fix your issue.

[elliotwoods]

eeek. wait a minute. more fixes :)

[elliotwoods]

ah actually, seems fixed. give it a go!

[sebllll]

ai elliot.

AsTexture (DX11) works like a charm now even when spreaded.

AsImage seems to be not spreadable. if i use AsTexture > asImage > AsTexture, there's only one slice (formerly were two). i tried to attach a screenshot (but due to a crappy internet connection i'm not sure that worked)

Though, TTY states some errors:

On initializing AsTexture (DX11) (e.g. ALT+RClick):

00:01:53 ERR : System.ArgumentException in mscorlib: Ein Element mit dem gleichen Schlüssel wurde bereits hinzugefügt. (= An element with the same key has been already added.)

Stacktrace:
   bei System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
   bei System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   bei SlimDX.ObjectTable.Add(ComObject comObject, ComObject owner)
   bei SlimDX.ComObject.Construct(IUnknown* pointer)
   bei SlimDX.Direct3D11.ShaderResourceView..ctor(Device device, Resource resource)
   bei FeralTic.DX11.Resources.DX11DynamicTexture2D..ctor(DX11RenderContext context, Int32 width, Int32 height, Format format)
   bei VVVV.Nodes.OpenCV.AsTextureDX11Instance.CheckTextureAllocation(DX11RenderContext context) in c:\vvvv\vvvv-sdk\vvvv45\VVVV.Packs.Image\src\nodes\plugins\Image\OpenCV\src\Destinations\AsTextureDX11Instance.cs:Zeile 147.
   bei VVVV.Nodes.OpenCV.AsTextureDX11Instance.UpdateTexture(DX11RenderContext context) in c:\vvvv\vvvv-sdk\vvvv45\VVVV.Packs.Image\src\nodes\plugins\Image\OpenCV\src\Destinations\AsTextureDX11Instance.cs:Zeile 119.
   bei VVVV.Nodes.OpenCV.AsTextureDX11Node.Update(IPluginIO pin, DX11RenderContext context) in c:\vvvv\vvvv-sdk\vvvv45\VVVV.Packs.Image\src\nodes\plugins\Image\OpenCV\src\Destinations\AsTextureDX11Node.cs:Zeile 49.
   bei VVVV.DX11.Lib.RenderGraph.DX11DeviceRenderer.ProcessNode(DX11Node node)

And, when initializing AsImage:

00:05:37 ERR : System.NullReferenceException in VVVV.Nodes.OpenCV: Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt. (=Object reference not set to an object instance.)

Stacktrace:
   bei VVVV.Nodes.OpenCV.FromDX11TextureNode.Evaluate(Int32 SpreadMax) in c:\vvvv\vvvv-sdk\vvvv45\VVVV.Packs.Image\src\nodes\plugins\Image\OpenCV\src\Sources\AsImageDX11.cs:Zeile 118.

[elliotwoods]

Thanks for testing. Can you confirm that your image type issues are gone?

I'll look into the issues that you've described On Sep 4, 2013 7:24 PM, "sebl" notifications@github.com wrote:

ai elliot.

AsTexture (DX11) works like a charm now even when spreaded.

AsImage seems to be not spreadable. if i use AsTexture > asImage > AsTexture, there's only one slice (formerly were two). i tried to attach a screenshot (but due to a crappy internet connection i'm not sure that worked)

Though, TTY states some errors:

On initializing AsTexture (DX11) (e.g. ALT+RClick):

00:01:53 ERR : System.ArgumentException in mscorlib: Ein Element mit dem gleichen Schlüssel wurde bereits hinzugefügt. (= An element with the same key has been already added.)

Stacktrace: bei System.ThrowHelper.ThrowArgumentException(ExceptionResource resource) bei System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add) bei SlimDX.ObjectTable.Add(ComObject comObject, ComObject owner) bei SlimDX.ComObject.Construct(IUnknown* pointer) bei SlimDX.Direct3D11.ShaderResourceView..ctor(Device device, Resource resource) bei FeralTic.DX11.Resources.DX11DynamicTexture2D..ctor(DX11RenderContext context, Int32 width, Int32 height, Format format) bei VVVV.Nodes.OpenCV.AsTextureDX11Instance.CheckTextureAllocation(DX11RenderContext context) in c:\vvvv\vvvv-sdk\vvvv45\VVVV.Packs.Image\src\nodes\plugins\Image\OpenCV\src\Destinations\AsTextureDX11Instance.cs:Zeile 147. bei VVVV.Nodes.OpenCV.AsTextureDX11Instance.UpdateTexture(DX11RenderContext context) in c:\vvvv\vvvv-sdk\vvvv45\VVVV.Packs.Image\src\nodes\plugins\Image\OpenCV\src\Destinations\AsTextureDX11Instance.cs:Zeile 119. bei VVVV.Nodes.OpenCV.AsTextureDX11Node.Update(IPluginIO pin, DX11RenderContext context) in c:\vvvv\vvvv-sdk\vvvv45\VVVV.Packs.Image\src\nodes\plugins\Image\OpenCV\src\Destinations\AsTextureDX11Node.cs:Zeile 49. bei VVVV.DX11.Lib.RenderGraph.DX11DeviceRenderer.ProcessNode(DX11Node node)

And, when initializing AsImage:

00:05:37 ERR : System.NullReferenceException in VVVV.Nodes.OpenCV: Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt. (=Object reference not set to an object instance.)

Stacktrace: bei VVVV.Nodes.OpenCV.FromDX11TextureNode.Evaluate(Int32 SpreadMax) in c:\vvvv\vvvv-sdk\vvvv45\VVVV.Packs.Image\src\nodes\plugins\Image\OpenCV\src\Sources\AsImageDX11.cs:Zeile 118.

[image: asimage]https://f.cloud.github.com/assets/646501/1079588/dffe99b0-154b-11e3-8b81-58dd94a39eaa.PNG

— Reply to this email directly or view it on GitHubhttps://github.com/elliotwoods/VVVV.Packs.Image/issues/10#issuecomment-23779412 .

[sebllll]

yes. works with cleye mono and colored. and imageload

[elliotwoods]

so to confirm that the current issues are that there's a spreading issue, and TTY complaints will look into this for alpha

[elliotwoods]

i can recreate:

and can see the issue in the code:

                    DX11Texture2D t = this.FInput[0][this.AssignedContext];
                    var height = t.Height;
                    var width = t.Width;

                    var imageLink = FOutput[0];
                    var imageAttributes = imageLink.ImageAttributes;
                    var desiredImageFormat = ToOpenCVFormat(t.Format);

looking at this now