Support mesh reuse

[bramcohen]

In SolidPython currently if you reference an object twice it gets inlined twice and the mesh gets generated twice, which is a waste of CPU. OpenSCAD supports mesh reuse via the slightly strange mechanism of the children() function but it does work.

Prescad generates code which uses this technique: https://github.com/bramcohen/prescad

Prescad's output is a bit ugly because it always uses children() and never inlines even for things which are only used once. Ideally SolidPython should make a new module and invocation using children() whenever an object gets used more than once but inline it in the more common case of being used exactly once. Prescad also gives a warning when something is declared but never used which might or might not be a useful thing for SolidPython to warn about.

[jeff-dh]

I think we can all agree that the generated SolidPython code is not ideal, because it does not (re)use any modules / functions / methods (what ever you want to call them) as we all know it from other languages.

But, in contrast to prescad -- which is a preprocessor --, SolidPython is a real language or to put it more precise it's a library for a real language, Python.

This means that in contrast to prescad which -- I assume -- basically does a search & replace on the prescad-code to "generate" OpenScad code, SolidPython actually generates OpenScad code and as such you are able to do a lot more with it.

The bottom line is, that a python function -- which I assume would be the entity you would transform to a OpenScad module -- is a lot more expressive than a OpenScad module and there for you can't translate an arbitrary (Solid)Python function to an OpenScad module which -- as far as I understand it -- would be necessary to achieve what you're asking for.

Furthermore there would arise a problem with parameters (and passed children) because of the different points in time of evaluation. This is a crucial issues which runs through a lot of SolidPython "issues". Python parameters get evaluated while the OpenScad code is generated and generates static OpenScad code. Or from a different persprective, if you use a parameter in a python function -- or even pass it further down to another function -- you actually don't know that it's a parameter, it's just a r-value and there for it would be pretty tricky to translate it to OpenScad as parameter. (I'm not able to explain this point more precise at the moment, but maybe this gives you an idea about the issue).

There for I think it's more or less impossible to do what you would like to do with the fundamental idea of SolidPython. I think you would hit a lot of problems and the fundamental issues are on the one hand that python is a lot more expressive and on the other hand that there are different execution -- and there for evaluation -- times in SolidPython -- which would cause quite some issues.

The only idea what you could do which pops into my head is, that you could generate individual OpenScad models and import them into a composite model.

Create a wheel.py -> wheel.scad and import the wheel.scad into car.py -> car.scad.

I hope this makes some sense for you.

PS: I only had a brief look at prescad, so my assessments about it might be wrong or incomplete!

[jeff-dh]

After thinking about it for a couple of days I came up with something, but I don't know whether this is useful for you. I just played around with SolidPython to see what could be possible. This works with the exp_solid fork but should somehow be back portable.

It is basically using the scad_inline hack again to get around the python-scad-barrier.

from solid import *
from solid.extensions.greedy_scad_interface import *
import inspect

def compileToOpenScad(func):
    #collect, prepare and wrap parameters
    parameters = []
    headerParameters = ''

    for p in inspect.getfullargspec(func).args:
        parameters.append(ScadValue(p))
        headerParameters += p + ","

    if headerParameters[-1] == ",":
        headerParameters = headerParameters[:-1]

    #build a ObjectBase that represents the OpenScad module in the SolidPython
    #object tree
    scadFunc = ObjectBase()
    scadFunc.add(ScadValue(f"module {func.__name__}({headerParameters}){{\n"))
    scadFunc.add(func(*parameters))
    scadFunc.add(ScadValue("}\n"))
    return scadFunc

def blub(a):
    #this does not "compile", because you can't compare / evaluate a ScadValue
    #because it's value is unknown at python runtime
    #if a == 1:
    #    assert(False)

    return circle(a)

#compile the blub function
scad = compileToOpenScad(blub)

#add a call to blub to the "main scad object"
scad.add(ScadValue("blub(1);"));

#render scad code
print(scad_render(scad))

"""
    This prints:

    module blub(a){
    circle(r = a);
    }
    blub(1)

"""

What is the use case you would like to be able to use this feature? With this it should be possible to create openscad libraries with SolidPython. But I would not suggest to use it if you use SolidPython simply to model 3D objects. That would be a pain and you don't gain anything.

Furthermore this creates some traps:

def blub():
    return text(get_date_time())

compileToScad(blub)

The resulting openscad function would output constant text --> the date_time of the python execution / openscad generation.

[occivink]

@bramcohen can you elaborate on how the mesh reuse mechanism works (and/or point to some existing documentation of it, I cannot really find anything)? You say that it is possible by using children, does this mean that in this example:

module test() {
    children();
    translate([10,0,0]) children();
} 
test() { cube([5,5,5]); }

the cube mesh is reused? And that in this other example below, the mesh is recomputed instead?

module test() { cube([5,5,5]); }
test();
translate([10,0,0]) test();

[bramcohen]

Here's a very simple example of how Prescad works. Let's take a very simple solidpython bit of code:

from solid import *
s = sphere(10)
s2 = union()(s, translate((5, 0, 0))(s))
print(scad_render(s2))

Which generates a simple output:

union() {
    sphere(r = 10);
    translate(v = [5, 0, 0]) {
        sphere(r = 10);
    }
}

Obviously the sphere is rendered twice. The equivalent prescad code looks very similar:

!PRESCAD!
s = sphere(10);
s2 = union(){s, translate([5, 0, 0]) {s}};
s2

But it generates very different output:

module prescadfunc0() prescadfunc1() {sphere(10);}
module prescadfunc1() prescadfunc2() {children(0);union(){children(0), translate([5, 0, 0]) {children(0)}};}
module prescadfunc2() children(1)
prescadfunc0();

Let's ignore that it's calling into prescadfunc2 to do nothing whatsoever, that could be optimized out. The point is that the sphere is generated exactly once in prescadfunc0 and gets passed in to prescadfunc1 which reuses the same mesh twice using child(0).

The whole approach is very similar to solidpython but different in the output which it (currently) generates. Prescad is good at reuse and terrible at inlining while solidpython is good at inlining and terrible at reuse. It should in principle be possible to make something which does both well, but prescad is a nasty little knot of code so I haven't added in that feature yet (or gotten that final unnecessary call out, harumph) and in solidpython I suspect you'd have to make the whole thing multiple passses and change how a bunch of stuff works to keep it from always inlining everything everywhere. They both suffer from the same problem of calculating scalars in one pass and then meshes in another so things can change out from under you but even the underlying openscad has that problem.

[jeff-dh]

What would you like to achieve with this?

And how do you want to handle the point, that python is a lot more expressive than OpenSCAD?

From my point of view OpenSCAD is only the "backend" for SolidPython. An easy way to generate .stl files from the -- kind of -- csg tree SolidPython is building up. I would be more interested in completely getting rid of OpenSCAD and generating a CSG tree / .stl-file directly from SolidPython.

[bramcohen]

What would you like to achieve with this?

At the moment I'd like for solidpython to output code which renders as fast in openscad as the prescad code does. I have no particular attachment to prescad as a toolchain, I wrote it to autogenerate the spaghetti openscad code I was writing by hand. My usual language is Python and I'd rather do everything in that.

And how do you want to handle the point, that python is a lot more expressive than OpenSCAD? From my point of view OpenSCAD is only the "backend" for SolidPython. An easy way to generate .stl files from the -- kind of -- csg tree SolidPython is building up. I would be more interested in completely getting rid of OpenSCAD and generating a CSG tree / .stl-file directly from SolidPython.

Having Python directly generate the STL would be far preferable. Then it would be easy to memoize the meshes generated in each part of the tree. Openscad is mostly acting as an awkward wrapper around CGAL. But in addition to reusing meshes properly there are a few bits of support I'd need before ditching openscad completely, like minkowski sums.

[jeff-dh]

Having Python directly generate the STL would be far preferable. Then it would be easy to memoize the meshes generated in each part of the tree. Openscad is mostly acting as an awkward wrapper around CGAL. But in addition to reusing meshes properly there are a few bits of support I'd need before ditching openscad completely, like minkowski sums.

Ok, I see.

You're right, we would lose the ability to use any OpenSCAD library (my idea of SolidPython includes a strong usage of the BOSL library, cf. exp_solid).

The only think I can think of -- at least at the moment -- is what I proposed above. Obviously it's a rough sketch. If anybody would spend some time on it (adding child-support, maybe wrapping it into a decorator, adding some convenience,...) I guess we could get to a point where we could explicitly encapsulate meshes.

[nicolaslussier]

Isn't the goal to have an exact geometry description?

I should get involved. Having the possibility to use or not openSCAD sounds like a good idea an other would be to improve openSCAD.

Thanks for your investment in this cool project

On Tue, May 31, 2022, 8:50 AM Bram Cohen @.***> wrote:

What would you like to achieve with this?

At the moment I'd like for solidpython to output code which renders as fast in openscad as the prescad code does. I have no particular attachment to prescad as a toolchain, I wrote it to autogenerate the spaghetti openscad code I was writing by hand. My usual language is Python and I'd rather do everything in that.

And how do you want to handle the point, that python is a lot more expressive than OpenSCAD? From my point of view OpenSCAD is only the "backend" for SolidPython. An easy way to generate .stl files from the -- kind of -- csg tree SolidPython is building up. I would be more interested in completely getting rid of OpenSCAD and generating a CSG tree / .stl-file directly from SolidPython.

Having Python directly generate the STL would be far preferable. Then it would be easy to memoize the meshes generated in each part of the tree. Openscad is mostly acting as an awkward wrapper around CGAL. But in addition to reusing meshes properly there are a few bits of support I'd need before ditching openscad completely, like minkowski sums.

— Reply to this email directly, view it on GitHub https://github.com/SolidCode/SolidPython/issues/197#issuecomment-1142093690, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFT436675NNMMMAFTJ7OHNTVMYDHVANCNFSM5XCWA54A . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[etjones]

This is a great conversation, guys! As always, @jeff-dh, thanks for your deep thinking on the capabilities and possibilities of the system.

I'm hearing a couple different issues here from your original issue, @bramcohen: 1) SP-generated code results in code duplication that can be slower than reusing OpenSCAD modules with children()

2) SP-generated OpenSCAD code is less readable than natively-written OpenSCAD because there's essentially no module reuse.

There are circumstances where issue 1 creates a significant performance penalty. If you wanted 10,000 copies of some complex fractal shape, OpenSCAD code using children() could probably calculate final geometry much much faster than SP-generated OpenSCAD, which would have to do 10K times as much work.

The generated OpenSCAD code would also be 10K times longer and thus not as clearly readable.

Years and years ago, I did some tests comparing creating many (let's say 10K) cubes in a SP-loop (so, 10k+ lines of resultant OpenSCAD code) and compared rendering time versus native OpenSCAD loop creating the same 10K cubes (3 lines of code). Despite the extra parsing work required, both files rendered in essentially equal amounts of time. However, I didn't try using complex geometry, instancing with children(), or superimposing shapes so the renderer had to do extra calculations all throughout. I imagine that we would see significant time differences between the two in those cases.

My take is that issue 1 is basically a corner case: nice if we could resolve it easily, but if this is a big problem someone is working against often, they should probably just work in stock OpenSCAD.

I have more sympathy for issue 2. I'd love it if SolidPython could produce compact, idiomatic reuse-friendly OpenSCAD code. I don't really know how to go about doing that, though. @jeff-dh has written some great stuff in the expSolid fork that allows interleaving Python and OpenSCAD code in the same file-- and I think his proposed solution would resolve both issue 1 and issue 2. I also think there's a significant conceptual burden in specifying "this part of the code should be translated carefully to OpenSCAD modules, but it only works for some kinds of code actions", and I don't think I want to demand quite that much awareness from users. I'm very hesitant to add more wrinkles to the language for issues that aren't fundamental to using the system.

My feeling right now is to leave this issue as a known wart on the SolidPython -> OpenSCAD system, because the cures seem worse than the disease. I'll leave this issue open for the moment in case there's more y'all want to say about it.

[bramcohen]

This is a great conversation, guys! As always, @jeff-dh, thanks for your deep thinking on the capabilities and possibilities of the system.

I'm hearing a couple different issues here from your original issue, @bramcohen:

1. SP-generated code results in code duplication that can be slower than reusing OpenSCAD modules with children()

That's certainly my problem.

2. SP-generated OpenSCAD code is less readable than natively-written OpenSCAD because there's essentially no module reuse.

This isn't something I've thought about and prescad output isn't particularly readable or maintainable. But the amount of work to make prescad output look a lot more like solidpython output is probably on the order of a weekend project, where the other direction looks like a lot more work, so I'll probably take a stab at that in the hopes that the updated prescad internals could then be used as the basis for something in solidpython. It won't support hole() though, because that has some more difficult to support semantics which I don't understand. As seems to always happen with 3d projects none of them is exactly a superset of the others and there's tradeoffs with every tool.

There are circumstances where issue 1 creates a significant performance penalty. If you wanted 10,000 copies of some complex fractal shape, OpenSCAD code using children() could probably calculate final geometry much much faster than SP-generated OpenSCAD, which would have to do 10K times as much work.

The generated OpenSCAD code would also be 10K times longer and thus not as clearly readable.

It happens a bit differently than you expect. A typical case is where there are lots of spherical cutouts and bulges with the same radius so you want to make a high rendering sphere exactly once and reuse it many times. There are also many models where the same whole big thing is reused a handful of times. Incidentally, the built-in openscad sphere module is a disaster. There are some better ones written but what you really want is octahedron interpolation which has been on my list of stuff to do for a while.

My take is that issue 1 is basically a corner case: nice if we could resolve it easily, but if this is a big problem someone is working against often, they should probably just work in stock OpenSCAD.

Having lived that nightmare I'd argue they should work in Prescad 😊. One thing both prescad and solidpython fix is that you can write your code forwards rather than backwards, writing a series of alterations to do sequentially rather than saying to do the last step on the next to last step on the second to last step etc.

I have more sympathy for issue 2. I'd love it if SolidPython could produce compact, idiomatic reuse-friendly OpenSCAD code. I don't really know how to go about doing that, though.

I have no idea what idiomatic openscad code looks like! People seem to mostly write simple stuff in it, and the underlying language has some deep brain damage about how assignments work which is why prescad is clear on using openscad as a compilation target and not a possible bunch of extensions which openscad itself could be made to support. Maybe someone who knows parsers better than me could make that happen, and maybe I should file some requests on the openscad project, since if I'm making ridiculous requests from one project I might as well be making ridiculous requests from two.

@jeff-dh has written some great stuff in the expSolid fork that allows interleaving Python and OpenSCAD code in the same file-- and I think his proposed solution would resolve both issue 1 and issue 2. I also think there's a significant conceptual burden in specifying "this part of the code should be translated carefully to OpenSCAD modules, but it only works for some kinds of code actions", and I don't think I want to demand quite that much awareness from users. I'm very hesitant to add more wrinkles to the language for issues that aren't fundamental to using the system.

To be clear minkowski sum is missing completely from solidpython (unless I missed something). It could be added easily enough, but that's orthogonal to these other issues. I seem to use a lot of weird/advanced stuff. Prescad is very a limited tool. It does not (yet) allow for defining new modules in the 'forwards' style, mostly because it has the same technical headaches with getting that to work as solidpython does. The benefit of this approach is that it is a clear incremental win over native openscad coding and is a super tiny codebase. The downside of this approach is that it's only an incremental win over native openscad coding.

My feeling right now is to leave this issue as a known wart on the SolidPython -> OpenSCAD system, because the cures seem worse than the disease. I'll leave this issue open for the moment in case there's more y'all want to say about it.

In principle it should be possible to make solidpython give the exact output it does now except in cases of mesh reuse but I agree that it would be a lot of work and deep code surgery. It's fine if you close this issue out, but I'd like a chance to update prescad to make solidpython-style output and let y'all know by posting about it here first.

[etjones]

In principle it should be possible to make solidpython give the exact output it does now except in cases of mesh reuse but I agree that it would be a lot of work and deep code surgery. It's fine if you close this issue out, but I'd like a chance to update prescad to make solidpython-style output and let y'all know by posting about it here first.

Absolutely. Let us know what you come up with!

[jeff-dh]

On Tue, 31 May 2022 13:30:02 -0700 Bram Cohen @.***> wrote:

Btw / Offtopic:

... [prescad] won't support hole() though...

Take a look at the BOSL(2) library! It has a hole feature and if you can call native openscad functions / library functions you could use the hole feature in prescad.

Similar for minkowski. I don't know whether bosl supports minkowski, but I'm pretty sure you'll find something similar in the BOSL(2) library and exp_solid supports BOSL(2).

[jeff-dh]

Soooo, I got one step further.

This

from solid import *
from exportReturnValueAsModuleExtension import exportReturnValueAsModule

@exportReturnValueAsModule
def blub(a):
    #this does not "compile", because you can't compare / evaluate a ScadValue
    #because it's value is unknown at python runtime
    #if a == 1:
    #    assert(False)

    return circle(a)

import time
myModel = union()
myModel += translate(10, 0, 0)(blub(3))
myModel += translate(20, 0, 0)(blub(4))
myModel += translate(30, 0, 0)(blub(time.time()%10))

print(scad_render(myModel))

generates this

module blub(a){
        circle(r = a);
}

union() {
        translate(v = [10, 0, 0]) {
                blub(3);}
        translate(v = [20, 0, 0]) {
                blub(4);}
        translate(v = [30, 0, 0]) {
                blub(4.569560766220093);}
}

with this exp_solid user space extension

from solid.extensions.greedy_scad_interface import *
from solid.core.utils import indent
import inspect

registeredModules = {}

def getRegisteredModulesStr():
    s = ""
    for f in registeredModules:
        s += registeredModules[f]

    return s

from solid.core.extension_manager import default_extension_manager
default_extension_manager.register_pre_render(lambda root : getRegisteredModulesStr())

def exportReturnValueAsModule(func):
    def parametersToStr(args):
        s = ""
        for a in args:
            s += str(a) + ","
        if len(s):
            #cut of trailing ","
            s = s[:-1]
        return s

    if not func in registeredModules:
        argSpecs = inspect.getfullargspec(func).args 
        parameters = [ScadValue(p) for p in argSpecs]

        moduleCode = f"module {func.__name__}({parametersToStr(argSpecs)}){{\n"
        moduleCode += indent(func(*parameters)._render())
        moduleCode += "}\n"
        registeredModules[func] = moduleCode

    return lambda *args : ScadValue(f"{func.__name__}({parametersToStr(args)});")

and since it's pretty explicit I would think about whether we might "want to demand quite that much awareness from users"....

It does not support this child stuff yet, since I don't really really understand it yet :D I would need to take a look at it again and think about how that concept fits into the SolidPython way of things.

but it creates openscad modules from the return value(!!!) of a python function. The return values (or what gets exported as module) is actually a with OpenScadValues(!) parameterized tree of SolidPython.OpenScadObjects.

I think this could be half the way we need to go, right?

Btw: that's an exp_solid extension! If you grab and use exp_solid you can just plug that code in and it should work.

[jeff-dh]

There's one bug in the extension. It needs a minimal change in the greedy_scad_interface extension. For now I post the diff here. I can set up a branch if anybody is interested.

diff --git a/solid/extensions/greedy_scad_interface/scad_variable.py b/solid/extensions/greedy_scad_interface/scad_variable.py
index ef19b6b..c4fa85f 100644
--- a/solid/extensions/greedy_scad_interface/scad_variable.py
+++ b/solid/extensions/greedy_scad_interface/scad_variable.py
@@ -17,6 +17,9 @@ class ScadValue:
     def __repr__(self):
         return f'{self.value}'

+    def _render(self):
+        return self.__repr__()
+
     def __operator_base__(self, op, other):
         return ScadValue(f'({self} {op} {other})')

[etjones]

Yeah, put it in a branch and let’s take a look!

[jeff-dh]

https://github.com/jeff-dh/SolidPython/tree/exportReturnValueAsModule

there are two file in the root directory:

exportResultAsModule_example.py and exportReturnValueAsModuleExtension.py

you should be able to run exportResultAsModule_example.py

[jeff-dh]

(I did not work with OpenSCAD itself for quite some time, so I'm not really into the children stuff....)

I don't really get my head around the children concept and how to integrate it into SolidPython.

The way I would use SolidPython is to use hierarchical functions to compose models (the way we usually use iterative programming languages).

like this:

module blub(a){
    circle(r = a);
}

module bla()
{
    union() {
        translate(v = [10, 0, 0]) {
            blub(3);}
        translate(v = [20, 0, 0]) {
            blub(4);}
        translate(v = [30, 0, 0]) {
            blub(6.131652593612671);}
    }
}

bla();

Why do we need children at all?

From my point of understanding at the moment I think children are something similar like an OpenScadObject that get's passed in as parameter into a function. And it does support "var_args" (~> *args). But at the sametime it does not really seem to be that easy. It somehow seems to me like a OpenScad hack (to improve the performance??), but I don't really understand what's going on under the hood and don't really get the complete concept behind it.

(is there really somekind of "pre-rendering" going on??? Why don't they pass the children into modules as regular parameters? At the end -- I assume -- they actually just link copies(!!) of a subtree (children) into another one, right....?!?!?)

So I don't really get it and would be interested in ideas / suggestions / collective brainstorming about that topic.

Can anyone come up with an example / idea / syntax how we could integrate it into SolidPython?

[bramcohen]

From my point of understanding at the moment I think children are something similar like an OpenScadObject that get's passed in as parameter into a function. And it does support "var_args" (~> *args). But at the sametime it does not really seem to be that easy. It somehow seems to me like a OpenScad hack (to improve the performance??), but I don't really understand what's going on under the hood and don't really get the complete concept behind it.

In OpenScad the passed parameters fall into two completely different namespaces, the scalars and the meshes. When a function/module is called it's done as mymodule(scalar_parameters){mesh_parameters}. Usually you don't run into this brain damage because most user generated things only take scalars and return a single mesh, but bulit-ins get passed meshes. In addition to the bizarre asymmetry in how parameters are passed there's a bizarre asymmetry in how they're received. Scalars are fairly normal, but meshes passed in to you can only be accessed by using children() which can only access them by index. They can't be given names. I think there's another function to get the number of children but I don't remember its name off the top of my head.

The way Prescad handles making these things 'persistent' is to stack a bunch of modules each of which generates one new thing and then passes everything it was passed plus the one new thing to the next function, so each module can access all the things which were already created, like this:

module prescadfunc0() prescadfunc1() {expression1}
module prescadfunc1() prescadfunc2() {children(0);expression2}
module prescadfunc2() prescadfunc3() {children(0);children(1);expression3}
module prescadfunc3() prescadfunc4() {children(0);children(1);children(2);expression4}

Etc.

It then optimizes out the children past the lass place they're used to get the bloat under control.

I spent a while writing this sort of garbage by hand before working out the pattern and wrote Prescad to do it for me.

[jeff-dh]

Hmmmm could it be, that we are looking at it from two different angles?

If I understand it correctly -- you are trying to get to something which implicitly converts "arbitrary" PreScad / OpenScad / ... expressions into nested module calls using the children stuff, right? In other words an optimizer that runs on the """resulting syntax-tree""" and optimizes it, right? I don't really like that idea........ one reason is because SolidPython does not contain anything similar yet and another is that I'm afraid the resulting code would again be pretty unreadable / maintainable /...... (I assume a gcc optimized syntax-tree is pretty unreadable for humans while it's "optimal" in terms of cpu usage) and I think it would be relatively hard to write a balanced optmizer.

I would prefer some new SolidPython concept / syntax to give the user the ability to explicitly use the children mechanism of OpenSCAD if he wants to.

I would like to start from the point where we're at the moment:

@exportReturnValueAsModule
def myModule(scalar):
    #do something

myModule(1)

How do we integrate children into it?

#could be translated to a call using children stuff
@exportReturnValueAsModule
def myModule(scalar, children):
    #do something using children[...]

someMesh = someMeshCreatingFunction()
myModule(1, children=[someMesh])

We could also just check whether a parameter is an OpenScadObject / Mesh and if it is, we pass it into the resulting module using the children mechanism so that would be somthing like this:

def myModule(scalar, mesh1):
    #do something / mesh1 would correspond to children(0) in OpenScad

myModule(1, circle(2))

This could translate to:

module myModule(scalar) { #do something using children(0)};

myModule(1) { circle(2); };

BUUUT, the big question for me is, is this really any better than:

@export...
def myModule(scalar):
    someMesh = someMeshCreatingFunction() #this function could be an exported module itself, right?

Because that would be what I would do and what is intuitive to me and consistent to the rest of SolidPython from my opinion. The only reason to integrate something else would be if you would get a serious performance boost.

Do you think or know whether using children instead of a call to a submodule to receive a mesh is more performant?

[jeff-dh]

I have the feeling, that one major difference between prescad and SolidPython (and our points of view) seems to be that, prescad is following a "compiler-kind" approach, while SolidPython does not (from my perspective ;).

SolidPython has no knowledge about any code structure! SolidPython has no concept of functions, variables, calls, return values,..... the library just provides the tools to create a "geometric tree" and generate OpenSCAD code from it. The functions, variables, calls, return values are just there by chance (because we use python) and can be used to structure the creation process, but that's pure python stuff, SolidPython has no clue about it! It does not care (and know!) whether the model was created using these structural elements or whether it was generated by a script which consist of spaghetti code. The result is identical.

[bramcohen]

Do you think or know whether using children instead of a call to a submodule to receive a mesh is more performant?

I think we're talking past each other a bit here. The problem is that you can't assign variables, either local or global, to meshes (unless I'm a few generations of openscad behind and they've fixed a giant heap of brain damage there. That would be nice.) It's a very bizarre restriction which seems optimized to maximize pain. You can't hack around it by making a module which returns some kind of persistent global. That thing will have to recalculate the mesh every time it's called.

Making there be solidpython access to children() might be doable but it involves some terrifying introspection and then you're stuck writing the same horrendous spaghetti code to get around its restrictions but now you're doing it in Python (although at least you could use names instead of indices).

The big problem is that the information you need is technically available in solidpython but it involves digging around the syntax directed acyclic graph (the fact that it isn't just a tree is important here) and looking into the internals to figure out what everything is and generating openscad code based on it which looks very unlike the nice clean inlining which solidpython is set up to do right now.

[jeff-dh]

I think we're talking past each other a bit here. The problem is that you can't assign variables, either local or global, to meshes (unless I'm a few generations of openscad behind and they've fixed a giant heap of brain damage there. That would be nice.) It's a very bizarre restriction which seems optimized to maximize pain. You can't hack around it by making a module which returns some kind of persistent global. That thing will have to recalculate the mesh every time it's called.

Yeah, there was something in the back of my head and I was waiting for that ;)

Making there be solidpython access to children() might be doable but it involves some terrifying introspection and then you're stuck writing the same horrendous spaghetti code to get around its restrictions but now you're doing it in Python (although at least you could use names instead of indices).

Exactly..... but.... I like the "design" of SolidPython as it is and in general I guess that's not a big deal. This would give people who "go to extremes" the opportunity to use children and increase the rendering speed. Btw: realizing the approach above (accessing children) I think would be 10 more lines in the extension.

The big problem is that the information you need is technically available in solidpython but it involves digging around the syntax directed acyclic graph (the fact that it isn't just a tree is important here) and looking into the internals to figure out what everything is and generating openscad code based on it which looks very unlike the nice clean inlining which solidpython is set up to do right now.

Yeap....... let me think about it.....

a few quick ideas: I think we could create one big module which wrapps everything and then extract big edges (and I guess it would make sense to determine the size of the subgraph below it to avoid the prescad issue) with multiple references, extract them into a module of it's own and pass them as children into the big "main" module, right?

Btw: SolidPython is not inlining anything. It is just traversing the whole tree and dumps it out.

[jeff-dh]

I added child support to the extension. I know that's not what you're looking for, but I guess it's still interesting.....

The extension is not really clean yet and probably still has some edges, I just hacked it down bit by bit. I would spent some more time on it before I would call it production code. But it seems(!) to work.

from solid import *
from exportReturnValueAsModuleExtension import exportReturnValueAsModule

@exportReturnValueAsModule
def blub(a, children):
    #this does not "compile", because you can't compare / evaluate a ScadValue
    #because it's value is unknown at python runtime
    #if a == 1:
    #    assert(False)

    return circle(a) + \
           translate(0, 5, 0)(children(0)) + \
           translate(0, -5, 0)(children(1))

import time
myChildModel = square(2)

myModel = union()
myModel += translate(10, 0, 0)(blub(3, myChildModel, myChildModel))
myModel += translate(20, 0, 0)(blub(4))
myModel += translate(30, 0, 0)(blub(time.time()%10, myChildModel))

#render scad code
print(scad_render(myModel))

generates

module blub(a){
    union() {
        circle(r = a);
        translate(v = [0, 5, 0]) {
            children(0);
        }
        translate(v = [0, -5, 0]) {
            children(1);
        }
    }
}

union() {
    translate(v = [10, 0, 0]) {
        blub(3){square(size = 2);square(size = 2);};
    }
    translate(v = [20, 0, 0]) {
        blub(4);
    }
    translate(v = [30, 0, 0]) {
        blub(9.465762615203857){square(size = 2);};
    }
}

So with this I guess you should be able to express what you like even though it's not the nicest way and it's not what you really want.

I got the other idea (optimizing the tree) in the back of my head, maybe I'll give it a try in the next couple of days.

PS: this demonstrates pretty good the extension "mechanism" of exp_solid and I think this is something really cool. With it we could set up arbitrary extension repos and could add all kinds of stuff without touching -- a hopefully stable and clean -- SolidPython library itself.

[jeff-dh]

I added an optimizer extension to the branch mentioned above. Again it's a rough sketch of an optimizer, that uses the algorithm mentioned above. It has some issues (the algorithm used to optimize the graph) but does "something". At least it shows how an optimizer could be integrated into exp_solid (and in general into the SolidPython approach).

If anyone has ideas for an optimizer algorithm, you're very welcome to contribute or overtake it!

The optimizer for now optimizes this

union() {
        difference() {
                union() {
                        cube(size = 1);
                        sphere(r = 2);
                }
                union() {
                        circle(r = 5);
                        cube(size = 2);
                        cube(size = 2);
                }
        }
        translate(v = [10, 0, 0]) {
                union() {
                        cube(size = 1);
                        sphere(r = 2);
                }
        }
}

to this

module mainModule() {
union() {
        difference() {
                children(0);
                union() {
                        circle(r = 5);
                        children(1);
                        children(1);
                }
        }
        translate(v = [10, 0, 0]) {
                children(0);
        }
}
}
mainModule(){union() {
        cube(size = 1);
        sphere(r = 2);
}
cube(size = 2);
}

[jeff-dh]

I change the optimizer algorithm and it now extracts "recursively" all the children. I think that's pretty much what you were asking for, right?

It's not tested a lot and not really clean yet.

@bramcohen could provide the SolidPython model that caused the performance issues? I'd like to have a (couple?) of complex test files.

c = cube(2)
m1 = cube(1) + sphere(2)
m2 = circle(5) + c + c + m1
model1 = m1 - m2 + m1.translateX(10)

renders to

module wrapperModule0() {
union() {
        difference() {
                children(3);
                union() {
                        circle(r = 5);
                        children(0);
                        children(0);
                        children(2);
                        children(1);
                }
        }
        translate(v = [10, 0, 0]) {
                children(3);
        }
}
}
module wrapperModule1() {
wrapperModule0(){children(0);children(1);children(2);union() {
        children(2);
        children(1);
}
}}
module wrapperModule2() {
wrapperModule1(){children(0);children(1);cube(size = 1);
}}
module wrapperModule3() {
wrapperModule2(){children(0);sphere(r = 2);
}}
module wrapperModule4() {
wrapperModule3(){cube(size = 2);
}}
wrapperModule4(){};

[bramcohen]

Here's a good test case which should do a good job of showing the performance difference with reuse:

from solid import *
s1 = sphere(10, segments = 50)
s2 = union()(s1, translate((5, 0, 0))(s1))
s3 = union()(s2, translate((0, 5, 0))(s2))
s4 = union()(s3, translate((0, 0, 5))(s3))
s5 = union()(s4, translate((10, 0, 0))(s4))
s6 = union()(s5, translate((0, 10, 0))(s5))
s7 = union()(s6, translate((0, 0, 10))(s6))
print(scad_render(s7))

That's going to make 64 high quality sphere and union them one at a time when done wrong, but it only renders the sphere once and unions it 6 times when doing it right.

[jeff-dh]

So, I think it's doing exactly what it's supposed to do. This is the optimized output:

module wrapperModule0() {
union() {
    children(5);
    translate(v = [0, 0, 10]) {
        children(5);
    }
}
}
module wrapperModule1() {
wrapperModule0(){children(0);children(1);children(2);children(3);children(4);union()
{
    children(4);
    translate(v = [0, 10, 0]) {
        children(4);
    }
}
}}
module wrapperModule2() {
wrapperModule1(){children(0);children(1);children(2);children(3);union()
{
    children(3);
    translate(v = [10, 0, 0]) {
        children(3);
    }
}
}}
module wrapperModule3() {
wrapperModule2(){children(0);children(1);children(2);union() {
    children(2);
    translate(v = [0, 0, 5]) {
        children(2);
    }
}
}}
module wrapperModule4() {
wrapperModule3(){children(0);children(1);union() {
    children(1);
    translate(v = [0, 5, 0]) {
        children(1);
    }
}
}}
module wrapperModule5() {
wrapperModule4(){children(0);union() {
    children(0);
    translate(v = [5, 0, 0]) {
        children(0);
    }
}
}}
module wrapperModule6() {
wrapperModule5(){sphere($fn = 50, r = 10);
}}
wrapperModule6(){};

BUUUUUUT!: It does not increase the rendering speed! It actually takes exactly the same amount of time to render the optimized and unoptimized openscad code! Is it possible, that OpenSCAD is doing some optimization / caching on it own? The console output also says something about cached geometries and cached cgal poyhedrons.....

[jeff-dh]

I spent a lot of time analysing the source code for OpenSCAD in the hopes of improving its cache, using interpolation of models during rendering and I even tried to port some of its math functions to Cuda libraries so my GTX1080 could help out. This was a mammoth fail as the developers of OpenSCAD have already done a far better job than I could have. The way the code is rendered from top to bottom and the way the objects are arranged in the rendering process is very good.

https://medium.com/@mr_koz/faster-3d-modelling-with-openscad-d6443f3eea79

.........for me this raises the question whether what we're talking about makes any sense (at least concerning performance) or whether the openscad optmizer does the job for us anyway no matter what we throw at it.....

Do you have a model that renders faster with prescad than with SolidPython? If so, could you provide it including the prescad code and / or prescad output?

[occivink]

I notice that you're always passing all children down to the lower wrapper, even if they're not being used. I don't know whether that has an impact (edit: it does not, I tested), just something I've noticed. I've written raw .scad corresponding to the test case of @bramcohen naive.scad which just inlines all translations for the 64 spheres reuse.scad which uses the children() mechanism, similar to what you posted but slightly cleaned up. I can clearly see a difference in rendering speed, with reuse.scad taking 49s and naive.scad 3m41s. The generated meshes look identical. So it does seem like taking advantage of children() can lead to substantial performance improvements.

[jeff-dh]

I get the following results on my machine:

regular.scad -> 1.08m (the unoptimized output of SolidPython / exp_solid without using the optimizer) opti.scad -> 1.07m (the optimized output of exp_solid using the extension, this is what I posted above) reuse.scad -> 1.1m (your reuse.scad file)

That's all the same + measuring errors.

I notice that you're always passing all children down to the lower wrapper, even if they're not being used. I don't know whether that has an impact, just something I've noticed.

I think this is necessary to be generic. You can build models that use children on all kinds of "layers". At the same time passing an unused parameter should not slow down rendering. Getting rid of it would make the optimizer more complex without any advantage.

[occivink]

Indeed, you're right that the regular output of SolidPython with Bram's test case produces something that is as fast as when using the children() optimization, even though it looks like everything is always recomputed (closer to the naive approach). I guess OpenScad does some automatic optimization when some of the branches of the graph are identical. It does make one wonder whether adding this mesh reuse optimization is worth it for SolidPython.

[jeff-dh]

Btw / Offtopic: with the "expressiveness" of (Solid)Python we can express the -- "not working" -- example from above a little bit nicer and shorter:

from solid import *

s = sphere(r = 10, _fn=50)
for i in [5, 10]:
    for t in [translateX, translateY, translateZ]:
            s = s + t(i)(s)

yeah, that's what makes the difference -- at least for me ;)

PS: translate[X,Y,Z] is only available in the exp_solid dev branch.

[bramcohen]

I've written raw .scad corresponding to the test case of @bramcohen naive.scad which just inlines all translations for the 64 spheres reuse.scad which uses the children() mechanism, similar to what you posted but slightly cleaned up. I can clearly see a difference in rendering speed, with reuse.scad taking 49s and naive.scad 3m41s. The generated meshes look identical. So it does seem like taking advantage of children() can lead to substantial performance improvements.

On my machine these run in 8:02 and 2:05 which is a similar meaningful difference. There's some weird stuff in openscad where you have to explicitly say 'render' because it has much faster operations which usually produce something which looks okay (like in this case) but sometimes fail horribly and aren't ready to export to STL. Maybe that's the difference? I'm on a mac using openscad 2021.1 Also using the solidpython which is in homebrew, no idea why it's on the dev branch

[jeff-dh]

On my machine these run in 8:02 and 2:05 which is a similar meaningful difference.

Yeah, right, same over here, but that has no meaning. You're comparing an arbitrary way to express the model (which for some reason is horribly slow) versus an optimized output version of SolidPython. What does is say? The optimized output is better than the worst(?) way to express that model in OpenSCAD, so what?

If you compare the optimized version vs. the unoptimized output of the head of SolidPython master the result is that there's no difference in speed. E.g.: whether we use children in the SolidPython output or not has no effect on the rendering speed.

Again the same question: Do you have a model that renders faster in prescad than in SolidPython? If so could you please provide it.

I / we need a SolidPython output that is significantly slower than an alternative OpoenSCAD code of the same model. And that's not the case with the example above, the regular SolidPython output is as fast as any other (known) OpenSCAD code that expresses the model.

Check whether the example you posted above renders faster when you "compile" it with prescad vs. "compiling" it with SolidPython master head.

[jeff-dh]

There's some weird stuff in openscad where you have to explicitly say 'render' because it has much faster operations which usually produce something which looks okay (like in this case) but sometimes fail horribly and aren't ready to export to STL.

PS: I use openscad -o blub.stl blub.scad to stopwatch the rendering which should pretty sure do what we want, right?

[jeff-dh]

In other words, it turns out that this:

union()
{
    sphere(10, $fn=50);
    sphere(10, $fn=50);
}

takes the same time to render as this

module bla()
{
    union()
    {
        children(0);
        children(0);
    }
}

bla(){sphere(10, $fn=50);}

And that contradicts the main assumption of this whole issue, right?

PS: If you use the gui to render OpenSCAD code, make sure to flush the caches before hitting "render", otherwise the results are meaningless (Design->Flush Caches)

[bramcohen]

If you compare the optimized version vs. the unoptimized output of the head of SolidPython master the result is that there's no difference in speed. E.g.: whether we use children in the SolidPython output or not has no effect on the rendering speed.

Those both render in about 2 and a half minutes on my machine, so apparently that cacheing really does happen. Seems like a huge hack job, but hey it works. I guess I can go ahead and use solidpython as is and not take a performance hit.

[bramcohen]

So since this seems unnecessary maybe I should close this issue out? Sorry for not figuring out that it wasn't needed in advance, hope people at least found investigating it worthwile.

[jeff-dh]

yep

[jeff-dh]

(I did not work with OpenSCAD itself for quite some time, so I'm not really into the children stuff....)

I don't really get my head around the children concept and how to integrate it into SolidPython.

The way I would use SolidPython is to use hierarchical functions to compose models (the way we usually use iterative programming languages).

like this:

module blub(a){
    circle(r = a);
}

module bla()
{
    union() {
        translate(v = [10, 0, 0]) {
            blub(3);}
        translate(v = [20, 0, 0]) {
            blub(4);}
        translate(v = [30, 0, 0]) {
            blub(6.131652593612671);}
    }
}

bla();

Why do we need children at all?

From my point of understanding at the moment I think children are something similar like an OpenScadObject that get's passed in as parameter into a function. And it does support "var_args" (~> *args). But at the sametime it does not really seem to be that easy. It somehow seems to me like a OpenScad hack (to improve the performance??), but I don't really understand what's going on under the hood and don't really get the complete concept behind it.

(is there really somekind of "pre-rendering" going on??? Why don't they pass the children into modules as regular parameters? At the end -- I assume -- they actually just link copies(!!) of a subtree (children) into another one, right....?!?!?)

So I don't really get it and would be interested in ideas / suggestions / collective brainstorming about that topic.

Can anyone come up with an example / idea / syntax how we could integrate it into SolidPython?