Slimming down SharpDX (The main assembly, not the project)

[PathogenDavid]

First, I apologize if this has been discussed before or there is already an ongoing effort to do what I am describing. I could not find anything related to it.

Second, I am mostly posting this because I am making these changes to a personal version of SharpDX anyway. I thought I should start a discussion to determine if I should make my changes in a manner that allows them to be integrated into the main SharpDX repository at some point. I realize that what I am proposing is somewhat of a breaking change, but I thought I should say something in case Alexandre is already working on something similar at Silicon Studio or there is a general interest in this being done.

---

SharpDX advertises itself as "an open-source project delivering the full DirectX API under the .Net platform". While SharpDX certainly accomplishes this goal, it also comes with a ton of utility code to facilitate easier development of graphical applications. Most of this is the SharpDX Toolkit, but there is still much of it in the main SharpDX assemblies.

This unnecessary code seems to come from three primary locations:

1. Extra math stuff that was brought in when the SlimDX math library was added. These are types that may be useful for 3D math, but are never used by any of the binding assemblies. (Example: Angle)
2. Utility code that serves narrow internal uses for Toolkit. (Example: TaskUtil)
3. General utility code to make certain things easier (Example: RandomUtil)

As an example, the following files are unnecessary for building and using the SharpDX bindings:

Angle.cs
BoundingBox.cs
BoundingFrustum.cs
BoundingSphere.cs
Collections/ObservableCollection.cs
Collections/ObservableCollectionEventArgs.cs
Collections/ObservableDictionary.cs
Collections/ObservableDictionaryEventArgs.cs
Collision.cs
ComBaseStreamNative.cs
ComponentCollection.cs
DataBuffer.cs
Diagnostics/CollectionDebugView.cs
Direct3D/PixHelper.cs
Enums.cs
FrustumCameraParams.cs
Half.cs
Half2.cs
Half3.cs
Half4.cs
HalfUtils.cs
IO/NamespaceDoc.cs
IO/PathUtility.cs
IServiceRegistry.cs
IdentityEqualityComparer.cs
MathUtil.cs
Matrix3x3.cs
Multimedia/RiffChunk.cs
Multimedia/RiffParser.cs
Multimedia/SoundStream.cs
Multimedia/SpeakersExtensions.cs
Multimedia/WavWriter.cs
OrientedBoundingBox.cs
Plane.cs
RandomUtil.cs
Ray.cs
Serialization/ArrayLengthType.cs
Serialization/BinarySerializer.cs
Serialization/DynamicSerializerAttribute.cs
Serialization/IDataSerializable.cs
Serialization/InvalidChunkException.cs
Serialization/NamespaceDoc.cs
Serialization/SerializeFlags.cs
Serialization/SerializerMode.cs
ServiceEventArgs.cs
SingletonString.cs
Threading/TaskUtil.cs
TimerTick.cs
Windows/NamespaceDoc.cs
Windows/RenderControl.cs
Windows/RenderForm.cs
Windows/RenderLoop.cs

There may be more, but these are what I found while slimming down SharpDX for my own project. I more or less determined this through a combination of Visual Studio's "Find all references" tool, and removing+re-adding things while building for every target platform. I also removed much of the inter-dependencies between math library bits and anything using the serialization simply for the purpose of making things bare bones.

---

My proposal would be to:

• Move all math-related structs/classes into a SharpDX.Math assembly. (Potentially continuing to use the SharpDX namesapce to prevent breaking applications father than a missing library reference.)
• Move serialization into separate assembly. (This might not be super realistic, but it'd be nice to include a DirectX binding library without also getting a serialization library.)
• Move Direct3D utilities into separate assembly. (Keep graphics stuff separate.)
• Move Multimedia classes into separate assembly. (Keep audio stuff separate.)
• Move all general utility stuff into SharpDX.Toolkit.Utilities
• Move things more appropriate for toolkit (SharpDX.Windows) to new Toolkit library.
• Separate Toolkit into entirely separate repository (Similar to how the samples are completely separate.)

My personal primary motivations for this are:

• To further the consistency of the SharpDX/Toolkit division
• Remove unecessary baggage from the core assembly. (So, for example, you can use DirectX11 without an assembly that also includes a bunch of audio stuff.)
• Enhance the separation of interests in the codebase.
• Make it so a project can integrate SharpDX more easily without using the SharpDX math library. (This is my biggest motivation, the primary benefit of this is for cross-platform game engines that don't want to rely on another math implementation. Unfortunately, C# doesn't really let you fudge this by using hacky casts like SharpDX::Color* c = (SharpDX::Color)(void)&myColorThatHasSameMemoryLayout; so no "free" casts.)

I guess the overall goal is to make SharpDX more of a pure DirectX managed binding, rather than the "DirectX managed binding (And more!)" that it is right now.

---

Once again, I am not posting this issue because I am trying to tell you how to maintain your library. I am posting it because if it is something the primary maintainers desire, I am willing to put more effort into "doing it right" rather than making my own custom fork of SharpDX.

TL;DR: The main SharpDX assembly nets you a lot of unnecessary cruft, I am reducing it for my own purposes and want to know if there is interest in this being something done in the main repo.

[xoofx]

Hi @PathogenDavid,

Thanks for your feedback!

Indeed almost everything you are referring have been around for some times (some parts were in skype/email discussions with @ArtiomCiumac, other part are well known long standing issues I have in my head) and were actually partly scheduled for this year, just that we hadn't time to start this large refactoring or even share them here, so if you are willing to help with this refactoring, I would be glad to bootstrap the process. Your proposal is welcome!

Also we had plan this year to make SharpDX to be PCL compatible, so it is a lot related to this large refactoring.

For the background, I agree that SharpDX has grown a bit outside its initial boundaries. Lots of the things that are in the library also came from the design of SlimDX, mostly because I wanted to have some backward compatibility with this library, and It helped also to bootstrap the project (like the SlimMath port) Though SlimDX was a big fat assembly, I wanted to split and decouple it. That being said, I hate to manage an additional assemblies just to decouple things to store just 3-4 classes in this new assembly, so if we could avoid such a case, I would be happy. The Toolkit also has brought some unnecessary code to the core assembly (serialization and some utils maths). The Multimedia is not ideal. The Direct3D is not a huge issue here, as it is only 2-3 classes, but could indeed be moved into a common Direct3D assembly...etc. I could discuss of this for hours, so I'm glad that you have an opportunity to help here.

At Silicon Studio, we have copied back the math classes from SharpDX into our own assemblies (a SliconStudio.Mathematics assembly), so I fully see your concern about duplication.

Move all math-related structs/classes into a SharpDX.Math assembly. (Potentially continuing to use the SharpDX namesapce to prevent breaking applications father than a missing library reference.)

Yes. I had great projects for Math (mainly to bring some native math builtin interops to speedup things) but lack of time has not helped me to push things here. There is some consideration to later integrate the next .NET SIMD into the Math assembly, so this is something that is going to require some refactoring when .NET SIMD will be really ready. So overall, I agree that if we can make the separation, that would be better. I don't mind even to change the namespace. There are a couple of things that could be done to avoid even using SharpDX.Math in some situations. For example, if a method is taking a parameter like a Color4 or Vector4, we can provide only a method that will take a generic and just check that this generic is 16 bytes (and explain in the doc that the type is expecting to have 4 floats inside). It has been done in some part of SharpDX code. It is more problematic for some assemblies that are using these types as field struct but we could try to thing for a solution there. If we could completely avoid the SharpDX.Math assembly for DXGI/D3DCompiler/D3D11, that would be already great. For other legacy assemblies (Direct3D9, Direct3D10...etc.), we shouldn't bother.

Move serialization into separate assembly. (This might not be super realistic, but it'd be nice to include a DirectX binding library without also getting a serialization library.)

I agree. The serialization was a quick hack to be able to serialize data in the Toolkit, so we could move classes in the SharpDX.Toolkit assembly. Unfortunately, the design was a bit dirty from the beginning, so there is some strong coupling between the type and the serializer (so all Maths for example are using it), but this should be manageable to have serialziers outside the types, and register them to a factory instead.

Move Direct3D utilities into separate assembly. (Keep graphics stuff separate.)

I'm not completely happy about this, because it would require to create a SharpDX.Direct3D assembly for just 2-3 classes... but well, why not...

Move Multimedia classes into separate assembly. (Keep audio stuff separate.)

Yes. There are some dependencies that we would have to remove (For example, D3DCompiler is using just a little decoder FourCC but we could remove this dependency).

Move all general utility stuff into SharpDX.Toolkit.Utilities

Yes, though most of the methods are used not by the Toolkit but by the interop layer in SharpDX, but no problem to split them.

Move things more appropriate for toolkit (SharpDX.Windows) to new Toolkit library.

Hm, it is not strictly related to the Toolkit, so It should stay in its own assembly SharpDX.Windows. Lots of samples in SharpDX would still require to use a Window, without using the Toolkit.

Separate Toolkit into entirely separate repository (Similar to how the samples are completely separate.)

Yes. The Toolkit is accessing some internals from core assemblies, so It might require to expose these internals (but it is fine).

One thing is to check that all platforms are building fine. As long as you do this for all major refactoring, that will be fine.

Also, everything should keep the entire history of git (using git subtree...etc.), so I would probably prefer to setup a branch with an initial rough cleanup of assemblies so if you want to get into the details, you will be able to help there. What do you think?

[ArtiomCiumac]

All exposed ideas are great and I agree with them, however this is a huge breaking change, so I propose to implement it in SharpDX v3.0, while v2.x should stay as is.

[xoofx]

@ArtiomCiumac, sure, I was not expecting to put it as a minor version! ;) I have just started to setup the split of assemblies first and will push this on a branch in the following days. It will not necessarily compile, but from this, we will be able to focus on the different parts.

[PathogenDavid]

That being said, I hate to manage an additional assemblies just to decouple things to store just 3-4 classes in this new assembly, [...]

True, true. After experiencing your Mixed Platforms issue first hand, I can't say I blame you for not wanting to add more assemblies than necessary. Since the Direct3D bits are fairly widely used, they can probably live there.

when .NET SIMD will be really ready.

Yup, I am pretty excited for that! No more hacks that break when Microsoft updates (I dunno why I am linking this considering you made it...)

I don't mind even to change the namespace.

OK, cool. I wasn't sure how much of a concern backwards compability was.

we can provide only a method that will take a generic and just check that this generic is 16 bytes (and explain in the doc that the type is expecting to have 4 floats inside).

Yup, this was the type of solution I was leaning towards for those. My only concern was it might make the API more confusing to use. The other route I was thinking is make the code that is currently pre-written generated as well, but that might make it a pain to edit since T4 templates don't get syntax highlighted, etc in Visual Studio.

It is more problematic for some assemblies that are using these types as field struct but we could try to thing for a solution there.

Its not super ideal, but maybe dumb container structs of Float3/Float4/etc and forcing the consumer to implement implicit operators would be the easiest. There's probably something else though, I've not given this one much thought.

If we could completely avoid the SharpDX.Math assembly for DXGI/D3DCompiler/D3D11, that would be already great. For other legacy assemblies (Direct3D9, Direct3D10...etc.), we shouldn't bother.

That sounds good to me, I did a little (manual) dependency analysis and found that Direct3D9 is one of the worst offenders in math usage:

https://skydrive.live.com/redir?page=view&resid=798F537493B1975E!4031&authkey=!ANuki2UKIKiVb_M

Unfortunately, Direct2D1 is pretty bad too.

and register them to a factory instead.

Yeah, this is something I wanted to do as well.

Yes, though most of the methods are used not by the Toolkit but by the interop layer in SharpDX, but no problem to split them.

Sorry, my wording was bad there. I wasn't referring to the "Utilities" class, I was referring to stuff like TimerTick and SingletonString.

Hm, it is not strictly related to the Toolkit, so It should stay in its own assembly SharpDX.Windows. Lots of samples in SharpDX would still require to use a Window, without using the Toolkit.

Yeah, that'd make more sense.

One thing is to check that all platforms are building fine. As long as you do this for all major refactoring, that will be fine.[...]so I would probably prefer to setup a branch with an initial rough cleanup of assemblies so if you want to get into the details

I have a branch going on over in my personal fork: https://github.com/PathogenDavid/SharpDX/commits/sharpdxdiet

Although it is pretty rough since I was mostly focusing on seeing "What can I remove and have this still work?", so I'd be OK with re-doing some of the stuff from there on a different branch to keep a clean history. (In particular, I completely mangled the poor math code.)

@ArtiomCiumac Yup, I would not expect something like this to go main line until the next major version.

[PathogenDavid]

I started modifying SharpGen to produce output that uses generics instead of hard-coded SharpDX Math types. Its not 100% rock solid/tested and should probably get a little cleanup, but it works for removing the dependency on ViewportF and Rectangle by Direct3D11. It also needs to emit a typeparam documentation element still.

Here is a sample of its output:

        internal void GetViewports<TViewportF>(ref int numViewportsRef, TViewportF[] viewportsRef) 
            where TViewportF : struct 
        {
            if (Utilities.SizeOf<TViewportF>() != 16)
                throw new ArgumentException("The size of TViewportF must be 16 bytes.", "TViewportF");

            unsafe {
                fixed (void* numViewportsRef_ = &numViewportsRef)
                    SharpDX.Direct3D11.LocalInterop.Callivoid(_nativePointer, numViewportsRef_, Interop.Fixed(ref viewportsRef),((void**)(*(void**)_nativePointer))[95]);       
            }
        }

The change to the mapping XML looks like this:

    <bind from="D3D11_VIEWPORT" to="SharpDX.ViewportF" use-generic="TViewportF" />

EDIT: I forgot to mention. The size that TViewportF will be checked against is grabbed from the define element for SharpDX.ViewportF. There's also a sizeof attribute you can use instead.

My changes are live in my sharpdxdiet branch, but do be warned that I've only tested with Debug|Net40 and all only the Direct3D11 project enabled. It might break if other bindings try to build right now since they haven't had non-generated code updated.

Let me know if either of you have any questions or concerns with how I've gone about this so far.

[xoofx]

Great! I will have a look as soon as I have setup the new branch

[xoofx]

I have tested some split of the Matthematics, Direct3D and Multimedia, and move some types to the Toolkit (serialization, components, collections...etc.). Size of assemblies are:

• SharpDX: 227Ko
• SharpDX.Mathematics: 242Ko
• SharpDX.Driect3D: 12Ko
• SharpDX.Multimedia: 32ko

Before this, SharpDX was alone around 592Ko, so considering that Direct3D and Multimedia are very small, I will probably let them inside SharpDX, as it seems not really worth to bother a decoupling.

[xoofx]

Hm, as I feared, splitting Mathematics is not easy. For example, in DXGI, some simple structs are using SharpDX.Rectangle, or SharpDX.Color4.The problem is these two types have some properties and methods that would pull more structs from Mathematics. We could define some generic types in SharpDX, like kind of Int4, Float4 that would not have any methods. But then in SharpDX.Mathematics we would like to provide automatic casting between these raw types and Rectangle/Vector4/Color4 for example... I would prefer SharpDX.Mathematics to not have a dependency to SharpDX, but it seems difficult...

Ok, so what could we easily move to SharpDX.Mathematics? All Ray, Collision, boundingbox, Plane, Angle...etc. stuff. But It seems that moving types that are used for some interop scenarios (Vector4/Color4/Matrix...etc.) would cause more pain when using SharpDX...

EDIT: we could still use generic for method parameters but for struct members, this is really not possible as explained above...

So, what do you think @ArtiomCiumac and @PathogenDavid?

[ArtiomCiumac]

What methods are required from Rectangle and Color4 structures? Maybe we could add some interfaces and add them as restriction to generic method calls? For example:

public void SetSomeRectangle<TRectangle>(TRectangle value)
    where TRectangle: struct, IWidthProperty
{
    var width = value.Width;
    ...
}

This will allow decoupling from Math implementation, while still providing some more behavior than just data transfer.

[xoofx]

@ArtiomCiumac the problem is less for parameter marshalling in methods (that can be solved with generics) but mainly with struct members. There are a couple of structs around that are using SharpDX.Rectangle or Color4 for example.

[ArtiomCiumac]

Ok, I see the problem. I think we should leave these structures in SharpDX assembly, as they are mandatory to support DX interop (same as SharpDX.Bool structure). We can strip any methods - and just create them as extension methods in SharpDX.Mathematics assembly.

As regarding to another structures - I am not sure if added benefit outweights the increased complexity (I mean usage of generics).

[xoofx]

I will probably keep standard types in SharpDX (VectorX, ColorX, MatrixX...etc.) and move all collision/angle/bounding box stuff to a Mathematics assembly. It seems to be the only practical split without duplicating types and will avoid too much confusion when using SharpDX.

[xoofx]

I have pushed a new branch diet with an initial commit (18d89eb) that contains mainly:

• Move some math classes from SharpDX to SharpDX.Mathematics assembly
• Move several classes used only by the Toolkit to SharpDX.Toolkit
• Remove dependency on IDataSerializable for all math types.
• The toolkit is not compiling due to the changes to the serializer. More work will be needed to fix and rewrite this part.
• I haven't fixed the namespace for the basic vectors types but I have moved them to a sub folder Mathematics in SharpDX. I'm still unsure I want to break this much.
• I haven't created a SharpDX.Windows assembly (not yet sure about the name), that would contains only Windows Desktop specific types (RenderLoop...etc in namespace SharpDX.Windows)
• I have moved some methods that were using SharpDX.Plane (Matrix.Reflection) to the Plane class to avoid having to pull Plane to SharpDX assembly
• I haven't moved Direct3D or Multimedia from SharpDX assembly to a dedicated assembly, as It doesn't bring enough value for the breaking change...

Results so far are not huge as SharpDX assembly has lost only 140Ko. Anyway, the most important goal here is to be able to deliver a SharpDX PCL (at least for core assemblies) so if we can get there, that will be already great.

[ArtiomCiumac]

I haven't created a SharpDX.Windows assembly (not yet sure about the name) ...

Call it SharpDX.Win32 as its contents are specific to the Desktop platform only.

I haven't moved Direct3D or Multimedia ...

PixHelper is used only in Toolkit and is applicable only to Desktop platform - so it can be safely moved to the SharpDX.Win32 assembly, mentioned above. ShaderMacro is a structure used in generated code - so it should be in the root namespace same as VectorX, etc.

The SharpDX.Multimedia namespace should be analysed in the same way as some files are needed only in one assembly (AudioEndpointRole is needed in SharpDX.MediaFoundation) while anothers - are core structures (FourCC).

[PathogenDavid]

Sorry for the lack of comments on this yesterday, had a particularly long work day.

so considering that Direct3D and Multimedia are very small, I will probably let them inside SharpDX, as it seems not really worth to bother a decoupling.

Sounds OK to me. I mostly mentioned them as something I saw that could be separated out, but if they are impacting things so little then its probably not worth having an extra assembly floating around.

EDIT: we could still use generic for method parameters but for struct members, this is really not possible as explained above...

Yeah, this is certainly troublesome. I had not noticed / considered the depencencies from the generated structures. It seems that there's only a handful of these, maybe it'd be OK to require the minimal versions for when they are needed but allow generics for everything that can? We could also allow the math types to be genericed out too for the structs, but that might make it obnoxious to use those structs. IE: OutputDescription<SharpDx.Rectangle> every time you use OutputDescription, or GammaControl<ARES.Math.ColorRGBA> for GammaControl with a custom color type. Too bad C# doesn't support default type params.

As regarding to another structures - I am not sure if added benefit outweights the increased complexity (I mean usage of generics).

Are you referring to using generics for the interface classes? I still think it'd be valuable since it would lessen the blur between binding and binding++ that there is right now.

I haven't created a SharpDX.Windows assembly (not yet sure about the name)

SharpDX.Windows.Forms or SharpDX.WinForms, maybe?

PixHelper is used only in Toolkit and is applicable only to Desktop platform - so it can be safely moved to the SharpDX.Win32 assembly, mentioned above.

PixHelper is also only relevant for DX9, right? Or does Pix still work for DX10/DX11? I was under the impression it was replaced with the Visual Studio graphics debugging tools for modern DirectX.

The SharpDX.Multimedia namespace should be analysed in the same way as some files are needed only in one assembly

I definitely agree with this. It would also prevent confusing when reading documentation / browsing around in Intellisense. I know I've been bitten several times when exploring a new API and finding that exposed types / methods / whatever sound like what I want but are only relevant for a part of the API I'm not using or are only useful internally.

@xoofx Also, did you have any comments on my changes other than the accidental heap allocation?

[xoofx]

Call it SharpDX.Win32 as its contents are specific to the Desktop platform only.

Win32 could be confusing as it is not only for 32bit machine. SharpDX.Windows is a bit related to the WinForm system, not the OS. We could name it WinForm... not yet a big fan but could grow...

PixHelper is used only in Toolkit and is applicable only to Desktop platform - so it can be safely moved to the SharpDX.Win32 assembly, mentioned above.

It doesn't mean that it is not used outside the Toolkit! ;) We are using it at work without using the Toolkit. As it can be used on Direct3D9 as well as Direct3D10 and Direct3D11, hence the reason it is in SharpDX assembly.

ShaderMacro is only related to shader compiler in Direct3D/D3DCompiler/D3D9, so I would prefer to keep it inside Direct3D namespace.

[PathogenDavid]

[Pix...]hence the reason it is in SharpDX assembly.

Would it make more sense for PixHelper to be in SharpDX.Diagnostics?

[xoofx]

It seems that there's only a handful of these, maybe it'd be OK to require the minimal versions for when they are needed but allow generics for everything that can?

We can do it for method parameters, that's fine (And only in recent core assemblies, like Direct3D11).

Also, did you have any comments on my changes other than the accidental heap allocation?

If you meant the SharpGen changes, yeah sorry, I just had a quick look. Still not sure it is worth the additional complexity. Some of the methods which can take generics are mapped to IntPtr already, so they can be extended by some handwritten methods. There are only few methods that would require this, so not sure about the benefits...

[ArtiomCiumac]

Quite crazy idea, but... What do you think about making a mechanism that would allow linking SharpDX to a math library by the end-user? The C++ headers are really needed just initially, after that - the intermediary data (output of GccXml) can be stored somewhere and the user can build the needed assemblies himself by defining the mapping dictionary (something like Vector2 -> MyAssembly.MyNamespace.MyVector2).

This could be a "special" build of SharpDX which will not require the native headers as everything will be already prepared for generation and compilation. This will not require any generic "magic", while allowing anyone to link to his own math library and preserving the "wrapper-only" state. Quite crazy though...

[PathogenDavid]

@ArtiomCiumac This is actually mostly the way I was planning to do things initially, but I thought the generics route was nice since it allowed using other math libraries to be used without special versions. Plus it means the manually written code would either need to start being generated (not hard to do, but it makes it harder to maintain) or have it use generics while the interfaces are all generated with the custom math types. (Not too far out there, actually.)

[ArtiomCiumac]

With the "generics" route we hit the situation when some structures need to be used during code generation phase as members of other structures - and we can't use generics there. My opinion is that using generics in some areas, while in others - our own structures will lead to confusion.

[PathogenDavid]

Hmm, that is definitely true. I was going to say we could allow generics to be toggled for those that need it (a fairly simply change with the way I have it implemented), but I guess that isn't much different than allowing people to use their own math library like you suggested.

[xoofx]

Quite crazy idea, but... What do you think about making a mechanism that would allow linking SharpDX to a math library by the end-user?

Hm, if someone want this kind of customization, It can already do it by modifying mapping.xml files. But I really don't want to introduce such a customization. We have been using SharpDX in Paradox without any problem even using our own math library. There were only 2/3 calls to cast from our math lib to SharpDX lib (ClearRenderTargetView...etc.). So it is really not a huge issue...

But, after reconsidering it, I'm going to look how much exactly pure basic types we need for interop. Probably just a few, like RawRectangle, RawFloat4 (without any methods in them, just simple structs) so if we are making SharpDX.Mathematics dependent on SharpDX, with implicit operators conversion that would be almost transparent for users and unnoticeable in terms of perf (as there are no critical methods/structs for these cases in SharpDX). This would slim down SharpDX assembly and allow a real SharpDX.Mathematics to evolve on its own (I would accept the dependency to SharpDX) The SharpDX.Mathematics would be used by the Toolkit and it could even have its own repository. I will try this in the following days.

[PathogenDavid]

Sounds like a plan. Any way I can help?

[xoofx]

Sounds like a plan. Any way I can help?

Of course! Though during this initial split it is difficult to work concurrently on it, but right after, there will be lots of opportunities to help:

• Create a new SharpDX.WinForm or WindowsDesktop assembly (as it could be specific to desktop, still not sure)
• Remove support for net20 platform
• Add support core assemblies PCL for Windows App Store / Windows Phone Store and Windows Deskop. We will still have pure desktop assemblies (for example to support Direct3D9 on desktop). But for people wanting to target Desktop/AppStore/PhoneStore, we will provide a common PCL assemblies.
• Make the Toolkit work again - mainly fixing serialization.
• Migrate the Toolkit to another repo (with preserve commit history)
• Prepare nuget dev packages
• Branch all samples to use the new nuget and update them to reference SharpDX.Mathematics....etc.)

I will be busy for the next 2 months and probably won't be able to work on all these issues after the initial split. So any help with these is welcome

[xoofx]

Ok, I have pushed the changes and added new raw types in SharpDX.Native namespace (if you find a better name?) and added implicit converters to SharpDX.Mathematics. The toolkit is not compiling. I have also only tested compilation on net40 so it might not work in other config. This is not tested. Let me know what you think about it.

[PathogenDavid]

Awesome, I will take a look over it when I get home today. I'll also look at your earlier list and start working on some of those things.

Ok, I have pushed the changes and added new raw types in SharpDX.Native namespace (if you find a better name?)

I would think SharpDX.Mathematics.Native might be better. Or SharpDX.Mathematics.Interop. If I saw SharpDX.Native, I'd assume it was an assembly full of P/Invoke stuff.

[xoofx]

@PathogenDavid I have renamed to SharpDX.Mathematics.Interop. You can go ahead on this branch and for example add method with generics instead of using interop types. If the SharpGen approach is worth it, why not, but if it is just a couple of methods to rewrite, we can keep manual implems instead. Just focus on core recent API, DXGI, Direct3D11, and possibly Direct2D1/DWrite/WIC. Other APIs are less important.

As I said earlier, I'm going to be busy for the next weeks but feel free to open pull-request on this branch and I will have a look at them.

[PathogenDavid]

If the SharpGen approach is worth it, why not,

I'm thinking it may be. I've spent this morning working on fixing the toolkit and ran into the issue that anything that takes arrays. For example, EG: SharpDX.Direct3D11.RasterizerStage.GetViewports doesn't work with an array of ViewportF even though ViewportF casts to RawViewportF.

So either SharpGen needs to expose a way to pass the array as a void pointer, or it needs to allow generics so that the manually implemented code in DeviceContext.RasterizerStage.cs is able to take in different types of arrays.

[xoofx]

This issue for arrays should be fixed by previous commit. Modifying SharpGen is fine if there are lots of method candidates for generic parameters, otherwise manual changes are fine. Moreover when we need to provide methods supporting single argument and array varargs, it is easier to do this manually.

[ArtiomCiumac]

In the commits above I have fixed compilation on all platforms. This wasn't been tested yet it it runs correctly, but, at least, now all platforms are compiling fine and ready for further experiments.

[PathogenDavid]

Hey all! Sorry for the radio silence, I've been unable to have time for this project for various reasons the past month, but I'm back to help again!

Before I got lost in the jungle, I had started working on splitting the toolkit away from the main repo.

My interpretation of "Migrate the Toolkit to another repo (with preserve commit history)" was that you wanted a repository with only toolkit stuff in its history. I tried several solutions involving various forms of git filter-branch, but I never really got something satisfactory. The main issue is that it is always fairly aggressive, and don't take into account renames, which caused some major rifts in history to appear.

I also looked into git sub-tree, but it didn't seem like an ideal solution since the Toolkit stuff isn't completely isolated into one folder. I think this could work though if we are OK with maintaining the history for things within Source/Toolkit, and loosing it for everything else.

The other alternative is that the history of everything else non-toolkit gets lugged around with the repository. Its not ideal, but it is certainly the easiest solution.

Here is the SharpDX repository with only things I thought were necessary for the Toolkit: https://gist.github.com/PathogenDavid/0a0a8443c6179cc33a04 (This may be out of date to any changes made since this commit.)

So these are the options as I see them:

1. We leave the history of the non-toolkit stuff in for the new toolkit repo. (Bigger repo, but easiest to do.)
2. Use git subtree and only lose the history for stuff that isn't toolkit source code.
3. Use some magical git command I didn't think of / find.
4. Write a tool that uses something like GitSharp to do it. (I looked into their briefly and it seemed pretty overkill.)

Any thoughts or preferences? I don't like lugging around the extra unnecessary history, but #1 might be the ideal solution.

[xoofx]

Thanks @PathogenDavid I won't be able to follow this closely as I'm on vacation for a few weeks. I would probably prefer the git subtree way, not a big issue if non toolkit source code doesn't appear in toolkit source code, as they will still appear in main repo (we will not rewrite/filter-branch the main repo). The main repo will contain at some point a commit that will delete all Toolkit stuff.

Also, not entirely sure if I would like to duplicate some build files (or even SharpCLI) between the repos. Ideally, It would be better, but as we have to maintain/recompile/distribute them, I would prefer to ease our task on this. I'm thinking that It would not be a huge issue if the Toolkit repo had a dependency on the main repo (sub-module), so that we are able to reuse build files and integrate SharpCLI directly into the toolkit sln. May be there wil be some relative paths problems with the build files, but this should be fixable. There are some downsides with this solution (direct dependency with internal builds, maintain submodule...etc.) and discrepancy (ideally would be easier to have a dependency on nuget packages instead)

Lastly, there are some issues to consider on how the Toolkit will be distributed (dependency to main assemblies) and how it is going to affect nuget distrib. Though not a huge problem, so we will have a look later at this.

[ArtiomCiumac]

Hello there!

I have pushed a few more commits into diet branch. A new SharpDX.Desktop assembly was added for everything that is related to Desktop platform.

Also I have moved the LogFont class to SharpDX.DirectWrite namespace as it is used only there. If this is fine - there are a few more structs that are used only in one assembly and I can move them as well. Please let me know what you think about it.

[xoofx]

Closing this issue, as the latest master is integrating the diet branch for SharpDX. I will fix the Toolkit repo later.