Swap out 3D math classes

[kaveh808]

Replace the existing point.lisp and matrix.lisp with Shinmera's classes (vectors, matrices, quaternions, and transforms).

Let's wait on this until the SBCL ports are done.

https://github.com/Shinmera/3d-vectors https://github.com/Shinmera/3d-matrices https://github.com/Shinmera/3d-quaternions

[kaveh808]

I also came across this, which is simpler (vec3 and mat4 only) and optimized for SIMD on SBCL, who are the developers.

https://github.com/nikodemus/sb-cga http://nikodemus.github.io/sb-cga/

[kaveh808]

Also found this:

https://git.mfiano.net/mfiano/origin https://quickdocs.org/origin

[kaveh808]

@foretspaisibles Do you have any strong preference between the above (or below) libraries? I'm thinking of sb-cga.

[JMC-design]

Does sb-cga even work now that sb-simd has been added to sbcl after many changes while no changes have been made to sb-cga?

I am planning of adding origin to my point protocol so if you're interested in performance differences I can check that. Though not sure how important that is, I think checking bounds with 3d-vectors was about 15x faster than clos based points, only about 8x using generics. As always, seems to depend on how much flexibility you want.

[kaveh808]

Two more options:

https://git.mfiano.net/mfiano/gfxmath

https://github.com/cbaggers/rtg-math

[kaveh808]

Don't know.

Checking on performance differences would be useful, thanks.

Some of my concerns are simplicity, the ability to dispatch on vector and matrix types in methods, and the ability to pass vectors efficiently to OpenGL or other graphics engine.

[awolven]

3d-vectors are structs. 3d-matrices are backed by common lisp arrays. But since 3d-vectors is row major and OpenGL is column major, causing you to shuffle elements, the simplest thing to do is to unpack the array into a foreign array when you need to pass it into GL, via a GL 1.1 command, or in other graphics toolkits, to shaders.

I have been doing this for 4 years now and have experienced no performance problems.

BTW, 3d-vectors and 3d-matrices can be built with an element type of single-float or double-float. For CAD purposes one needs to use double-floats. When I pass vectors and matrices off to the graphics language, I (coerce number 'single-float). I haven't experienced performance problems with that either.

On Mon, Aug 22, 2022 at 3:48 PM Kaveh Kardan @.***> wrote:

Don't know.

Checking on performance differences would be useful, thanks.

Some of my concerns are simplicity, the ability to dispatch on vector and matrix types in methods, and the ability to pass vectors efficiently to OpenGL or other graphics engine.

— Reply to this email directly, view it on GitHub https://github.com/kaveh808/kons-9/issues/23#issuecomment-1223000587, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGMMOWQIFI4OVGNDGX7Y3V2PRSFANCNFSM563LG5AQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[kaveh808]

The gfxmath docs mention doing computations in double precision and then passing single precision to the gpu.

3d-vectors and 3d-matrices seem to be the most elaborate and complete version. I just wonder if we need all the complexity involved, or if I'm over thinking things. :)

On a side note, if we were to use one of these libraries, what is the best way of doing so? Should we just do a Quicklisp load of the library, or should we make local copies in our project of the sources, possibly modifying them to suit our needs better?

[awolven]

Many of these libraries I have written are on github, and if they are not yet, i.e. Hemlock and Regen, they will be soon. I could just take commit requests and/or have separate branches of experimental stuff which could be pushed to by users. Quicklisp registration takes forever.

On Mon, Aug 22, 2022 at 4:15 PM Kaveh Kardan @.***> wrote:

The gfxmath docs mention doing computations in double precision and then passing single precision to the gpu.

3d-vectors and 3d-matrices seem to be the most elaborate and complete version. I just wonder if we need all the complexity involved, or if I'm over thinking things. :)

On a side note, if we were to use one of these libraries, what is the best way of doing so? Should we just do a Quicklisp load of the library, or should we make local copies in our project of the sources, possibly modifying them to suit our needs better?

— Reply to this email directly, view it on GitHub https://github.com/kaveh808/kons-9/issues/23#issuecomment-1223065503, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGMMMOVEI466B7UPKUMUTV2PUYPANCNFSM563LG5AQ . You are receiving this because you commented.Message ID: @.***>

[kaveh808]

Cool, thanks.

I was thinking specifically of the vector and matrix libraries. Should we just copy the code over to our project and then mess with it?

[awolven]

You can "git submodule" anybody's libraries you like in kons-9.

On Mon, Aug 22, 2022 at 4:41 PM Andrew Wolven @.***> wrote:

Many of these libraries I have written are on github, and if they are not yet, i.e. Hemlock and Regen, they will be soon. I could just take commit requests and/or have separate branches of experimental stuff which could be pushed to by users. Quicklisp registration takes forever.

On Mon, Aug 22, 2022 at 4:15 PM Kaveh Kardan @.***> wrote:

The gfxmath docs mention doing computations in double precision and then passing single precision to the gpu.

3d-vectors and 3d-matrices seem to be the most elaborate and complete version. I just wonder if we need all the complexity involved, or if I'm over thinking things. :)

On a side note, if we were to use one of these libraries, what is the best way of doing so? Should we just do a Quicklisp load of the library, or should we make local copies in our project of the sources, possibly modifying them to suit our needs better?

— Reply to this email directly, view it on GitHub https://github.com/kaveh808/kons-9/issues/23#issuecomment-1223065503, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGMMMOVEI466B7UPKUMUTV2PUYPANCNFSM563LG5AQ . You are receiving this because you commented.Message ID: @.***>

[awolven]

no, use the git submodule facility, and we can submit commit requests to shinmera/mfiano/|3b|, etc.

On Mon, Aug 22, 2022 at 4:45 PM Kaveh Kardan @.***> wrote:

Cool, thanks.

I was thinking specifically of the vector and matrix libraries. Should we just copy the code over to our project and then mess with it?

— Reply to this email directly, view it on GitHub https://github.com/kaveh808/kons-9/issues/23#issuecomment-1223135832, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGMMN53F55BYXFOVPT5KLV2PYGZANCNFSM563LG5AQ . You are receiving this because you commented.Message ID: @.***>

[awolven]

PS Right now the quicklisp version of 3d-vectors and 3d-matrices have some issues compiling them in SBCL, those issues are easy to workaround though, and I could at least inform shinmera of the problems or possibly submit a patch.

On Mon, Aug 22, 2022 at 4:46 PM Andrew Wolven @.***> wrote:

no, use the git submodule facility, and we can submit commit requests to shinmera/mfiano/|3b|, etc.

On Mon, Aug 22, 2022 at 4:45 PM Kaveh Kardan @.***> wrote:

Cool, thanks.

I was thinking specifically of the vector and matrix libraries. Should we just copy the code over to our project and then mess with it?

— Reply to this email directly, view it on GitHub https://github.com/kaveh808/kons-9/issues/23#issuecomment-1223135832, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGMMN53F55BYXFOVPT5KLV2PYGZANCNFSM563LG5AQ . You are receiving this because you commented.Message ID: @.***>

[kaveh808]

Got it. Thanks.

[JMC-design]

Which version of SBCL? I had no problems with 3d-vectors. I'm not sure if any of mfianos future plans for his libraries will be realized. He seemed quite happy to jump over to Julia. The gotcha with 3d-vectors is you can only have single or doubles, not both at the same time.

[awolven]

The gotcha with 3d-vectors is you can only have single or doubles, not both at the same time.

This isn't really a bad thing, Lisp developers should not have to constantly decide whether to use single-floats or double-floats, and having functions like (vec3f ...) and (vec3d ...) to cause bugs with. It's not a performance issue on 64 bit machines, it is more of a space issue.

On Mon, Aug 22, 2022 at 5:18 PM Johannes Martinez Calzada < @.***> wrote:

Which version of SBCL? I had no problems with 3d-vectors. I'm not sure if any of mfianos future plans for his libraries will be realized. He seemed quite happy to jump over to Julia. The gotcha with 3d-vectors is you can only have single or doubles, not both at the same time.

— Reply to this email directly, view it on GitHub https://github.com/kaveh808/kons-9/issues/23#issuecomment-1223220196, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGMMNRA7CIEB6Q2C6BDQ3V2P4CZANCNFSM563LG5AQ . You are receiving this because you commented.Message ID: @.***>

[JMC-design]

It's a limitation that forces you to choose performance or precision. Which might not be bad if you use singles for real time and then can switch to doubles for rendering, but then you might not get what you think you will. I'm also not thinking of developers, I'm thinking of content creators. I like to be able to view my stuff from different points of view. I also like not losing information in a chain of transformations. Kinda why my colour stuff is all 16bit ints by default.

[JMC-design]

So after making some preliminary backends of 3d-vectors and origin for my point protocol I've learned a few things. It should have been obvious that generic dispatch on arrays is a bit of a mess. Origin has float and double versions of vec2 3 and 4. So in my 'fast' internal package every function is one big typecase. Which doesn't seem so bad considering 3d-vectors is pretty much typecases everywhere as well. Since 3d-vectors are all structs you can dispatch on any of those types.

I only tested two things so far. One was calculating the bounds of 1 000 000 random points, and the other was (reduce #'point:* list-of-1000000-points). Origin was about 3x as fast as arrays without types, about double that dropping the protocol and using the internal package. 3d-vectors was almost 2x as fast as origin, and double that again with the internal package. On reduce using the generic protocol took 0.32secs for both origin and 3dv, internal packages were .024 for origin and .028 for 3d. Not such a big difference now even with generics.

I'm guessing each will probably have advantages over the other in different areas. Nice to know for some stuff the generic protocol isn't extremely slow, so at least I didn't waste my time.

as for gfxmath, that seems to have a weird take where everything is a matrix. Have not experimented with it.

I just wish Shinmera named his stuff to be used with prefixes, such a big namespace pollution.

[kaveh808]

Nice.

Did you try sb-cga? I'm curious about that one.

[JMC-design]

Not yet. I think that might only show advantage, if it does, over matrix multiplication. It doesn't even have a multiply vector by vector so couldn't try that. It was virtually the same as Origin when reducing by cross-product. I think the bounds example is affected by access, and I'm a little surprised the struct access is faster than vectors, but Shinmera's got some interesting optimizations in there.

and if access will be the bottleneck I'm not sure if vectors will show much advantage even when transferring to the gpu. And then all the math on the gpu will be the same. It might be better to test that once it's updated to newer opengl and start packing buffer objects.

[kaveh808]

Yes, we have to figure out how best to pack our data for GPU compute and rendering. For OpenL and/or Vulkan.

And then there's sb-simd... :)

[kaveh808]

Is someone willing to investigate how we can have vector and matrix operations with SIMD support, now that SBCL has them?

Maybe start here?

https://github.com/nikodemus/sb-cga http://nikodemus.github.io/sb-cga/

[foretspaisibles]

@kaveh808 I do not have worked with these libs but I can review them.

A usual tidy point to performant computing when programming is memory allocation: it is necessary to have facilities to write something like “please write the sum of A and B into C” and it is a bit in tension with our habit to write code in a more functional style. Our point primitives for instance.

[JMC-design]

If you want points as arrays, I'd suggest going with origin then. Both origin and 3d-vectors are ~6x faster for the simple benchmark you added for your change, at least on my machine. though I still think the bottleneck will be access, and it seems like if they're the right size you can just use origin vecs as is with sb-simd. Whether that makes any difference in packing simds...

[kaveh808]

@JMC-design OK, let's use Origin, with an eye to adding SIMD to it later.

What's the best system way to add it to our project? I do want to make some changes and extensions to it.

@foretspaisibles Origin has those kinds of functions for reducing memory allocation. Would be interesting to use them in the internal mat/vec code.

[JMC-design]

Note I'm still not sure whether the vectors in origin will be faster when packing points for the gpu or simd. I think we need a non trivial test case to determine that. I was a little surprised that the structs used by 3d-vectors where faster at bounds checking than origin, and that was just min max comparisons and setfing into the result. Then again, I don't understand everything Shinmera is doing in there.

[kaveh808]

I'm happy to punt the SIMD stuff until later.

Origin looks nice and clean, and also has some raycasting code which we may be able to use.

Let's get it into our project and I'll make the changes throughout the codebase. :)

[JMC-design]

As for a non trivial test case, I've been fooling around with cl-opengl and doing anything modern pretty much requires you using the lower level %gl package and then becomes an issue of how do you want to abstract over all the ffi stuff? If you don't then you're left with a bunch of ffi that should really be compile time not runtime.
Which is something both Cepl and Trial do. As to which does it better, not sure how to measure that. The only thing I know is I've never seen anything built with Cepl besides people playing around. While there are quite a few small games written with Trial but not sure how that translates to 3d creation.