TorqueScript has suboptimal performance

[jamesu]

Compared to other embeddable scripting languages, TorqueScript on average is quite slow. Anyone who has used TorqueScript heavily in an iTorque project has probably realized this. Mobile devices are of course slow, but it seems to me even doing simple things in TorqueScript is costly.

After actually benchmarking a rather simple function call ( https://gist.github.com/3737516 ), I was shocked after seeing just how bad it can get.

I'm not quite sure what the best way of tackling this is, so I'll provide as much information as I can in this Issue.

My tests seem to indicate that function calls in TorqueScript appear to be quite expensive, whereas in other embeddable scripting languages this is far from the case. Variable lookups also appear to be costly though this is true of any scripting language.

For reference, I tested 6 methods of adding two numbers together:

• scriptTest1 returns p1 + p2
• scriptTest2 sets global variable $_R to $_0 + $_1
• scriptTest3 is the same as scriptTest2 except it copies the global to local variables (similar performance to test 1)
• scriptTest4 is the same as scriptTest1, except the addition is done in C (ends up being faster)
• scriptTest5 is the same as scriptTest2, except the addition is done in C (and is actually slower)
• scriptTest6 adds two literals with no function calls, and is the fastest

I've placed the specific code I used for benchmarking TorqueScript, Ruby and Lua in a gist: https://gist.github.com/3760645 . All results are in milliseconds, the last number being the average time taken.

Torquescript results (Macbook Pro 2010 2.4ghz):

Torque3D RELEASE
scriptTest1,2220,2196,2224,2248,2632,2304
scriptTest2,692,688,696,692,692,692
scriptTest3,2224,2208,2236,2220,2208,2219.2
scriptTest4,448,436,436,432,432,436.8
scriptTest5,964,956,960,960,976,963.2
scriptTest6,280,288,284,280,288,284

The worst case here is 2.3s in the pure script function call. It seems making a local variable is costly as test 3 is the second worst case. Test 5 is oddly slower than test 2, and would be even slower if I used the actual Con:: getIntVariable function which does an int->string->int conversion.

The best case is 0.29s, which beats mruby but not lua.

mruby results for tests 1,2,3,6 (latest version, Macbook Pro 2010 2.4ghz):

TEST,1,2,3,4,5,AVG
scriptTest1,642.965,616.527,607.443,604.453,607.673,615.8122
scriptTest2,717.744,717.211,716.863,726.13,714.424,718.4744
scriptTest3,729.728,726.318,722.327,728.388,722.21,725.7942
scriptTest6,490.863,481.995,495.214,484.136,481.721,486.7858

Note that the worst case here is 0.73s.

Lua results for tests 1,2,3,6 (v5.2.1, Macbook Pro 2010 2.4ghz):

TEST,1,2,3,4,5,AVG
scriptTest1,177.313000,173.866000,173.777000,173.635000,172.777000,174.273600
scriptTest2,365.673000,366.093000,366.950000,367.679000,373.810000,368.041000
scriptTest3,495.440000,493.399000,490.451000,492.778000,491.064000,492.626400
scriptTest6,151.421000,150.907000,150.767000,149.743000,151.732000,150.914000

Note that the worst case here is 0.49s.

[EricPreisz-GG]

2 options. 1) Optimize the compiler with V-tune, fix things, and test the heck out of the engine. 2) Ditch Torque Script.

[dottools]

I would prefer that we try to fix features before tossing them out as garbage, especially such a critical part that is the glue of the game engine that TorqueScript is. So, lets go with option 1 for the time being. ;)

[Xenovore]

I would like to see TorqueScript completely replaced with Lua, personally. As far as scripting languages go, it's doesn't get much faster than Lua.

[kripken]

I would recommend using JavaScript, for the following reasons:

1. Should someday someone want to add the option to run Torque3D games as HTML5, compiled to JavaScript, this would make it much much more feasible. (I actually work on that area myself, have ported another 3D game engine to HTML5 - https://developer.mozilla.org/demos/detail/bananabread - and might be curious to try the same with Torque3D now that it is open source). The web is becoming a very important compilation target since everyone has a browser, and browsers are just now becoming powerful enough to run full 3D games.
2. Modern JS engines are very fast. Comparisons to Lua(JIT) are difficult because it's hard to find apples-to-apples comparisons, but from my knowledge of the internals of their implementations, they should be on par with each other.
3. There are 3 modern and fast open source implementations of JavaScript: Google's V8, Firefox's SpiderMonkey and WebKit's JSC.
4. JavaScript is likely the most familiar programming language because it is used on the web (and now also Node.js etc.).

[EricPreisz-GG]

We are implementing V8 on a service project that we are working on right now. Very nice.

[Demolishun]

Or turn TS into a JIT compiler... Eric, I was thinking about modifying the macros used to create TS commands and variables. Would it be feasible to modify those to automagically create bindings into another VM? I was thinking of looking into this with Python. Have the macros generate the Python function calls and globals in parallel to TS commands.

Also, I know .NET code can be compiled from Python so I am guessing Mono does this too. That could be another avenue is to switch to .NET for the VM. You may even be able to target the .NET VM via Torque Script. Ooh, that is a good option. Target a VM that has already got a large code base like .NET. Then you can program in many languages. I wonder if there is a Lua translator for .NET?

[losinggeneration]

I for some reason always cringe when I hear of ECMAScript in game engines. It's kind of a strange thing because I actually like the language (or at least a subset of the language commonly called "The Good Parts" thanks to Douglas Crockford.) Some people don't realize that ECMAScript and Lua actually have a fair number of things in common if you disregard syntax. I think the reason I tend to like Lua in engines better is because it's actually quite similar to "The Good Parts" of ECMAScript without the rest that makes ECMAScript so treacherous for new devs. For instance, many people are surprised by the following:

if ( "0" == false ) { console.log("Are they equal?"); }

Now, that said. I'd personally like to see more than one scripting language implemented in addition to TS, perhaps in a sort of plugin nature. Getting rid of TS would be a mistake IMO since it will break lots of existing code (and in this case I think that's unacceptable since it's an unneeded break.) I'd personally prefer it not to be rebuilt on something like the CLR (.NET for those who don't know what the VM for it is) or the JVM. Again, though, if this ends up being the way it goes, I hope it's as an optional plugin. I'm going to say this just to stress the point: TS should stay around to keep legacy code working

[setlec]

i don't like the idea of replacing a scripting language by another, firstly it will take a long time to replace it and i'm pretty sure that there are many ppl that have learned to use TS. must improve first and then maybe someday we might re-discuss this.

[Demolishun]

Can someone explain to me what TS really is? I know there is a compiler, but the actual language calls are just a set of console callbacks. The callbacks are called from an interface called the console. This interface is where most of the slow down comes from because it is a string lookup. So if the calls were exported to say Python calls then we would gain the advantage of the Python interpreter being more efficient. It should effectively allow more than one language to exist in the same engine. The same thing could be done for Lua or any other language. That would certainly make it modular.

I will have to rethink my interface with Python. If I can tell the engine to create Python function calls when the T3D console calls are created by modifying the macro then I can gain automatic export of all console calls. Which would still allow someone to write standard console functions as those would be exported as well. Right now I can call console functions, but having the calls directly tied to Python calls would make it faster and not have the lookup issue. Currently the console can call Python as well and it is slightly faster due to the design.

After that it should be trivial to convert the TS code to Python code or leave as is. It won't matter. It will just be faster to be in Python. Since I am using SWIG it may be a portable concept to use Lua or any other SWIG supported language.

@Setlec, It took me 3 months to write my Python interface for T3D part time. It is in the resources at GG. I will be updating it soon to reflect a licensing change to MIT as well as adding support for T2D. It did not take a long time. What I am talking about here is making it closer to the metal than it is now. Python is also an easier language to grasp than TS and has thousands of professionally designed packages already ported to use with it.

[jamesu]

@Demolishun TorqueScript itself is the bulk of what is in console combined with the core stuff like as StringTable. The interpreter itself is compiledEval & StringStack.

There appears to be a half-completed rewrite of the Console binding system in engineApi.h which everything using "DefineEngine..." uses, but as it's incomplete an a lot of the code still uses the "Console*" macros it's mostly just a fancy newer way of exposing console functions and objects. There is also a Javascript binding API in the web plugin code which uses Console binding system to bind methods from Javascript. Come to think of it several years ago there was even a resource for a python binding system back in the TGE 1.x days which I recall was used in Minions of Mirth.

On a general note...

If anyone is going the "Ditch torque script route", keep in mind the need to work with existing code unless you fancy rewriting all the script code (core, editor, ...). Also consider the possible need to run on mobile devices which are more of a prevalent platform nowadays.

It seems that everyones favourite language is being put forward as a replacement, maybe it is wiser to finish the engineAPI stuff and stick on a pluggable scripting system?

As for "Optimizing the TorqueScript compiler", I think that's a good step in any case. For a start there's clearly something wrong with function calls currently which I think can be solved with a little refactoring of how function parameters are passed.

[crabmusket]

I agree with James and Demolishun ('zat you, Frank?). Ditching TS would break lots of things, but at the same time, I'd really like to see a more modern and non-Torque-specific scripting language available. Making scripting languages a plugin sort of deal would be fantastic.

[Demolishun]

@eightyeight, Yep, zat me. http://www.garagegames.com/community/resources/view/21706 This is a new Python binding to T3D using SWIG. It is working now. Now, I want to know more about how this scripting plugin is supposed to work. If that is possible and it would be fast enough I would update my resource to use that. I already have a working framework even if it is not perfect. It is based upon SWIG so theoretically it could support any of the SWIG supported scripting languages. I have kept the code separated so Python stuff is in one file and generic SWIG stuff is in another.

[brenttaylor]

While I agree ditching Torque Script would alienate anyone with a sizable preexisting code base, in it's current form it's just not a viable language for any sort of demanding work.

Plugging a different language in (Lua or JS, I'd avoid python) is certainly a solution and would broaden the user base significantly. As it stands, Torque Script is sort of the bastard child of Javascript and Perl. It's not particularly easy to learn and there are a number of gotchas and inconsistencies with the language syntax. It's easily my most despised feature of Torque.

As much as I dislike Torque Script, I propose a middle ground. Plug in the Lua or V8 VM and compile down Torque Script to the chosen VM's byte code. If done properly, you even have the option of using a mix of languages during development Torque Script, Lua/Moonscript or Javascript/Coffeescript.

As for the reason I'd steer clear of Python. I'll be honest, I love Python. However the standard library is huge and there are, as mentioned, thousands of preexisting professional quality libs. This is great for the developers but can be terrifying when looked at from a security perspective. Sandboxing the V8 or Lua VM's is easy. Sandboxing Python, not so much. This is particularly scary when downloading levels/missions from an untrusted source when joining a server.

Python is not a bad option, but if chosen, we would need to go over it's implementation very carefully.

[Dzonatas]

I've looked at this project at-a-glance, so far. I've compared it to Second Life. When the VM bit was mentioned, that got me interested. I thought about the Torque3D version of SL. One thing I would do differently than SL is separate the VM and 3D into separate processes. That works good with high frame rates (tested SL with ReSTful queues).

I wouldn't bother with any 3D SDK if it doesn't aim to work in VMX mode instead as an SDK.

[Demolishun]

I am pretty sure Lua could coexist with Python at the same time. I agree on the sandboxing issue with Python. That has always been an issue due to the introspection of the language. Lua could handle sandboxed code, TS could handle legacy code, and Python (or insert your favored language here) could handle everything else. Also, levels and the like should not be downloaded in script form. They should be XML and so they can be parsed. Levels as script is not a smart thing to begin with.

What I am really concerned about here is there are lots of "I would want this", but no action. I am willing to put forth action, but I will be supporting Python and other requested SWIG languages. The reason I am using SWIG is because it is something I understand and has multiple language bindings, including .NET. I also know how to write libraries for Python. From what I have seen of Lua it would be even simpler to support. If someone wants to work on a complete replacement of the VM I will help, but unless I see action I am going to take the route I prefer. If we end up with competing choices that would not hurt my feelings one bit. It is actually going to make the engine that much better by having choices. I fully expect the .NET version already in the works to be used by a lot of people as well.

To be honest about TS. I don't like it. It tries to be C/C++, but it is not. I have started a rewrite of the VM to use functions for each op code rather than a switch statement. That was my first step to optimizing for eventual JIT implementation. The function code takes advantage of tail call optimization to reduce return calls for each opcode. When I compared startup times it was about the same. So at least it has not hurt performance. I was hoping to lay the ground work for a faster version. So if any VM experts would like to get a hold of my work I can make it available. Oh and we should point out that missions as script are certainly NOT sandboxed 'as is'.

"Python is not a bad option, but if chosen, we would need to go over it's implementation very carefully." From what I have read is "Don't use Python for untrusted code." So supporting a sandboxable language in addition to anything else would be a good idea. So I like your Lua idea for that type of application. Also, we can design the interface to make it so you can comment out any language a person does not want. If someone only wants Lua, we can oblige.

[brenttaylor]

@Demolishun My hesitation stems from Garage Games apparently already implementing V8 for some of their service work. I'm fine with V8, I'm just waiting word if it's going to make it back down the pipeline to the public repo here. If it's not, then yes, I'd very much be interested in working on integrating Lua. It's small, it's fast, it works on every platform I can think of, it's easy to learn and use, and I have a decent amount of experience integrating it into projects.

However, I'd like to make it clear that my preference would be to rip Torque Script out entirely. It's a horrendous language and adding further abstraction to support both TS and Lua will only slow both implementations down. I understand that this would alienate anyone with prior code bases. Considering Torque has just moved to be open source under the MIT, it might be best to treat this beast as a completely new engine.

[Dzonatas]

Virtual machine extensions allows us to skip the sandbox analog. Anything graphic intensive should be written with VMX. On machine that don't support VMX, you need the sandbox analogy of VMs, which is access control on the host. In guest mode you don't need that overhead, as the guest OS can deny all device access except to cache and display.

Past 3D engines redo their VM and become web browsers (with V8), as that is common development phase. I rather see you use VMX, and move forward.

[brenttaylor]

@Dzonatas Maybe I'm getting my terms mixed up, but I think we're talking about a very different type of virtual machine. We're talking about VM's that interpret bytecode (CLR, JVM, etc.), not hardware sandboxing and segmentation.. The two are not analogous in this context.

In addition, VMX tech is not supported on any console or embedded device that I'm aware of (iOS, Android, Xbox 360, PS3, Wii, etc.)

[lukaspj]

@Demolishun I think it is a great idea to implement the SWIG interface! Just looking at the languages on the compatibility page it would give support for java, ruby, python, perl and seems like it can even handle c#! But as you already have it running with Python, do you have any idea of how much faster it is? I'm with Demolishun on this one. On a sidenote how much work is it to make it work with new languages?

Btw SWIG compatibility page: http://www.swig.org/compat.html

[Demolishun]

@Dzonatas, Like BrentTaylor pointed out. VMX is not applicable here. Nobody wants to work below the OS. Unless I am missing something is there VMs for scripting languages using the VMX interfaces?

"adding further abstraction to support both TS and Lua will only slow both implementations down" I don't think it will. When you create a TS console command you are basically writing a C++ function that automagically creates a TS command. This same macro could create any number of bindings to other scripting languages. The call from the scripting language to the actual function would be direct with nothing in between. So it would be just as fast.

Throwing out TS is not a good idea. There is too much existing code showing how to use the engine. Perhaps a couple years down the road as people start using other interfaces. Or even rewriting the TS compiler to target another VM would be an option.

[Demolishun]

There is another problem about using another scripting language. The C++ in many places calls TS functions. That would have to be reconciled if the scripting language gets replaces. Even SimObject creation is called using a script function from a C++ call.

[Demolishun]

@lukaspj, Those are not automatic. It still requires work to add each language. I don't know if the interface is faster than TS. There is a Python class abstraction in between. I tried to get rid of that and it would not compile. The goal was not to replace TS at the time. I have done some novel things that do speed up this interface. For instance you can literally create callbacks in either direction. You can also import Python objects into TS. That way TS can use those objects as if they are TS objects.

For speed I would need to re-evaluate the calling mechanism, I think. That is what I was referring to with the console command macros. You should be able to create a Python command that calls the C++ code created when you create a console function. Object lookup would have to be re-evaluated as well.

There are ways to speed up the interface and SWIG has support for getting rid of the class abstraction layer. I tried to have it produce an interface with the abstraction class turned off and I got all sorts of errors. So I decided I wanted to finish the interface as is so I could continue working on my game. I mostly wanted all the libs Python offers. It is currently not an ideal substitute for TS, but it works.

When the Python code calls TS commands it does so through Con::exec. I believe this is pretty close to the metal, but there might be faster ways. I am certain there are better ways to interface the engine for speed.

[brenttaylor]

@Demolishun Ah, this is where we have a methodology disagreement. I greatly dislike using Macro's for things like this. They tend to be overused and make debugging painful. Using Macro's, I agree, would not slow the system down one bit. Doing it properly, certainly would. However, let's hold off on the discussion of how the "proper" way to do this is. Let's get some of the bigger issues settled. What course of action do we take? Fix TS in it's current state? Rewrite it to compile down to the bytecode of a proper and well maintained VM? Rip it out and replace it with another language? Or do we write a plugin system for it?

I fully understand the arguments for wanting to keep TS as an option, but as I've mentioned, I really do believe that we should be treating the new release of Torque as a completely new engine. I see little reason to hinder it's direction at this early stage.

As for TS being so fully ingrained in the C++ source code, I'm very much of the opinion that this is part of the problem with TS as a whole. Regardless of whether we keep TS or replace it, this needs to be cleaned up dramatically.

As for a plugin system, here are my thoughts on the subject: On the surface, I love the idea. However I do feel the cons are rather significant. Primarily that you'll be segmenting the community. Resources will be in a number of languages, languages that not everyone is likely to know. Unity suffers from this as do frameworks like Leadwerks. A certain amount of uniformity is a Good Thing(TM).

EDIT: @'d the wrong person. My bad.

[Dzonatas]

@brenttaylor I read the new XBox has VMX, and last I read Atom processors do not. I am aware of game loops and console-only modes.

@Demolishun Anything that wants full screen mode will want to be friendly with the hypervisor, which runs with VMX. One simple setup of the system would be Torque3D, ASP.NET/XSP, and linux with no other packages. Absolute minimal before you rid of the kernel itself. As said, you don't need to go that far, but along the way any system under VMX doesn't have to deal with all the BS from the other device drivers and window managers. We can deal with it from the HTTP server direct to device driver.

I have an agent-region in mind that detaches from the user. Games usually deal with inventory, and we need regions, not sandboxes, where that inventory can be as active as the frame rate. Skew is the common reason for buffer bloat, and we could get mesh optimization into the kernel or its module. The user mode programmer sees this connection between graphics and regions usually different than what is optimal for meshes, which obviously need the physics processor. In other words, after minimal, we need the 3D server to interop or co-op with the physics server.

There may be many, and at that point the co-op methods scales. I like avatar to have its own region if it wants to compete. I want each avatar with its own specialized VM. The server needs to be agnostic, and thus the choice in high-level language is irrelevant to the VM.

[brenttaylor]

@Dzonatas Again, we're talking about two completely different things here. Even so, this is a very bad idea. On consoles and handheld devices we're resource constrained as is. And again, most of these devices don't support VMX. If the new XBox in development supports it, I'd love a citation about it. None of the other consoles or handhelds support it.

[Dzonatas]

@brenttaylor The ideal script language you want is COLLADA. Realize that when any process serves those files that other devices only need to request resources that relates to that process or more optimally embeds it for other processes, devices, and cores. The search engine has your citation: "new xbox vmx", came up top.

It was not any bad idea when it was hard work to get it this far towards open standards.

[Demolishun]

@Dzonatas, You are speaking completely different language. I don't understand any part of what you are saying. I did a search and found 0 libraries or implementations referring to VMX. The onus will be on you to educate us and create/find a reference application to show how this should be done. Your text makes me think you are either using a translator or we are talking to a bot. For ex: "Skew is the common reason for buffer bloat, and we could get mesh optimization into the kernel or its module." This makes no sense in any context.

COLLADA eh. Maybe this should be embedded in the Neko VM usign ASP tech. Perhaps we can use SSE3 and VW instructions to optimize the bit rate flow for pixel spark reduction? Should we be using a Wankel Rotary Microprocessor like the Propeller too?

[brenttaylor]

@Demolishun I'm glad I'm not the only one who thought he sounded like a markov bot. It could just be a very bad translator, but that's not the vibe I'm getting.

[Dzonatas]

@Demolishun I don't judge, and maybe that is why it didn't make sense if you expected to be judgmental. I wanted to meet the people that are involved this project, and this discussion seemed like it was one good start. In games, it is always easy to find disagreement and difference, or follow any other trend.

VMX already exists in many stationary computers. It makes sense in that context, so no complaints. I think you can answer your own questions.

VMX is hardware instructions, so it is not some library. It sounds like it is new tech to you.

@brenttaylor That attitude is why projects fail. Good luck.

[Demolishun]

@Dzonatas, If you want to pursue helping with the VMX just show us by writing some code. I will do the same with my approach. I saw on your page you have worked on .NET/Mono. Maybe you can bring that to the table? I just don't think it is necessary to work at the hardware level. It is hard to take someone serious when they mention Collada as a scripting language. Collada is a graphics interchange format. It has not flow control constructs useful to writing game logic.

[Demolishun]

@brenttaylor, You have some good ideas that I will think about. It would be very nice to have a language like Lua for sandboxing. I will be modifying/redoing the ScriptT3D project in the near future for T2D. At that time I will evaluate the effort to include Lua. It won't be an ideal interface unless I see a simple way to use the macros as I mentioned before. I would like to understand how the modular code is supposed to work as well. I may have a look at that too. There are still issues with the macro approach as it does not completely address dynamic function and variable creation in TS. I may have to find a mechanism to make that transparent to any scripting languages attached. Right now I have solved that through console variable lookup, but that is a string lookup and just as slow as TS variable lookup. It is a similar story with SimObjects. Although with those I can import them as objects into Python, but methods still have to be looked up.

[Dzonatas]

@Demolishun Anything in XML is like scripts to me, as it is simply not code. It works with canoncalization, however.

As an interchange format, COLLADA needs its asset server, but I wrote HTTP server, above. The format does have sections for shader languages and object definitions, so it doesn't forbid it. Plus, it is more dynamic than that. People tried to go direct from COLLADA format to render, and they missed the whole point of the exchange features.

There is no need to be the most abstract while being agnostic about any new code except that there are truly genuine asset that are quite unique and available under open standards. What's impossible is for anybody to replicate that unique data, and COLLADA does not define how to access unique asset data... only the most common in the format mentioned.

Scripts and shapes, the two things that categorize 3D worlds. You know how to transform XML into XHTML with ECMA. JPEG2000 allows us to embed XML, so already every face on every object can have its own script when extracted from the JP2 format based on modern standards.

[jamesu]

I'm sure we can talk all day about whose scripting language or upcoming technology is the best replacement, but my intention with this Issue is to suggest a realistic solution to the problem which I and many other Torque developers have faced: at the moment TorqueScript is fine for small-scale coding glue, but once you go beyond that you start to run into issues with performance.

I've experimented with a few more extreme methods myself without much success, including even changing the function parameter passing so it passes native types instead of just strings (sadly this introduces too many other problems without much speed benefit).

Looking in the profiler, besides the interpreter i'm seeing a lot of time being spent in:

• Dictionary::lookup
• dSprintf (called from Con::getData, StringStack::setFloatValue, Con::setIntVariable, Con::setFloatVariable)
• dAtof called from Dictionary::Entry::setStringValue and StringStack::getFloatValue
• dAtoi (similar to dAtof)
• Dictionary::reset

Looking at this I'm pretty sure at least variable conversion can be optimized as a start.

[Demolishun]

@James, You mentioned the engineAPI.h. What do you see needing to be completed? I am kind of clueless as to how that actually works. I just don't understand why that is better than the old console command binding methods. I do like the idea of a pluggable scripting system, but I don't understand how it works. Is there a whitepaper somewhere in the game industry about that type of system?

After doing the work to convert the VM to function calls versus a switch statement I lost interest in spending time writing a VM. There are way more talented people than I with already fast and proven VMs. So I would like to avoid trying to bring TS up to speed. I am more in favor of grafting a VM in some way onto T3D. It may not be pretty, it may not be good code, but I can use it now rather than 1 to 2 years from now. So some sort of TS and cyborg is better than just TS now.

[jamesu]

@Demolishun engineAPI.h itself isn't a pluggable scripting system, it just provides an abstraction so you can define engine objects and methods using the DefineEngine* macros, which allows you to define functions using native types (as opposed to an argv[] string array). From what I can tell when you call these methods from TS, it uses the normal Console API but automatically converts all the string parameters to native types. If you look in engineDoc.cpp and engineXMLExport.cpp, there are a few examples of how to enumerate this information. Looks a bit incomplete but maybe with a little work it could be good, which is why I mentioned it.

[Demolishun]

Oh, okay. I had no idea that worked. I saw the consoleXMLExport was all defined out and when I tried to enable that it did not work. The only issue would be commands that take variable amount of arguments, but that could be handled with the standard console export. This could indeed make things much simpler. With SWIG I can predefine data type conversion as well. This would make my current interface better if it works.

[Demolishun]

I just tried using exportEngineAPIToXML() and it creates an object with no data. So this is not working at the moment. If it can be made to work then this will make creating interfaces to any language simpler. Now I see why you were so interested in this. You help my mind do that "mental click". Thanks!

I put in a counter and it never finds any engine exports. So to make this work I need to understand what it missing then.

[losinggeneration]

@Demolishun I was going to try to take this message off issue but couldn't find an email or other method to contact you directly (since I hate adding to an already long issue).

Some examples of C++ plugin systems within game engines would be Ogre3D (albeit not for scripting, but it would be the same general idea), PixelLight IIRC actually has scripting as plugins (though, it makes heavy use of the RTTI throughout the framework and I think uses this to help creating some of the bindings.) Careful not to use any code from PixelLight though, it's LGPL, Ogre3D is fine if you want to use their plugin loading mechanism. The actually "plugin" doesn't have to be dynamically loaded (which will actually require platform specific code like dlopen), but could be done within the compiled engine and called/instantiated based on a defined interface. (I kind of think this discussion should be taken off issue onto the forum.)

[brenttaylor]

@losinggeneration I'd have to disagree. This is exactly where this kind of thing should be discussed. It doesn't become an actual ticket until someone tags it and this is a more open place for that discussion. I'd think it's best to keep it all in one place.

[losinggeneration]

@brenttaylor Except the majority of the discussion would be better suited in a different issue at the very least (since replacing TorqueScript isn't really relevant to this issue.) So I stand by that this isn't the correct place to be talking about possible plugin systems, if TorqueScript should be removed, adding other scripting, etc.

[Demolishun]

This was Eric Preisz's suggestion: "options. 1) Optimize the compiler with V-tune, fix things, and test the heck out of the engine. 2) Ditch Torque Script." I think we have been talking about option 2.

[losinggeneration]

@Demolishun Heh, I had actually forgot about that specific comment. I stand corrected.

[ChrisCalef]

So, while we're on the subject, coming from a position of relative ignorance (I've never installed a scripting language in an engine, always just used whatever was there)... can someone lay out some of the actual steps necessary for anyone who was going to either replace Torque Script entirely, or else (preferably) add another scripting language as an additional option to work side by side with Torque Script? How and where do you have to start cutting to actually do that, and how does the job vary depending on which language you choose?

Also, is there a reason when people are casting around for a good choice, that they don't just go straight to Mono/C#? Given the power, dependability, similarity in syntax to C-family languages, cross platform compatibility (for Mono anyway) and widespread industry use (eg Unity), it would seem like the obvious starting place to me... unless hooking it up is an order of magnitude more difficult than some of the other options.

Feedback? Very curious on this front... Just had to fix a set of totally random script errors that popped up while porting to the MIT T3D build. The code in question looked completely perfect, but the common thread turned out to be the presence of preceding comments that had three slashes at the beginning of the line, rather than two, ie:

/// /// comments... /// function myFunc(%arg) {...}

If I changed those to look like this: // // comments // function myFunc(%arg) {..}

.. then they compiled just fine. I for one am more than ready to start looking at well supported third party scripting languages.

[ChrisCalef]

And since we're on the subject, shall we start compiling a list of competitive GUI options as well? Qt, anyone?

[brenttaylor]

@ChrisCalef While there's nothing wrong with Mono/.NET, it's a pain to embed compared to options such as Python, Lua or any number of others.

Unfortunately ripping out Torque Script and replacing it is quite the undertaking. It's embedded throughout the engine itself. Adding another language alongside it isn't terribly difficult, it just doesn't really solve the problem which is speed.

As for GUI options, this isn't really the discussion for that.

[ChrisCalef]

Any specifics as to exactly what makes it more of a pain than the other ones? (Just curious, because I'm very tempted to go that way, but don't want to walk blindly into another WOOOORRRMMMMHHOOOOOLLLE if you know what I mean... )

Re: running alongside, it would solve the speed problem at least for any new code that gets added, and it might pay off in the short term to convert only the existing Torque Scripts that are creating the biggest lags, while avoiding the nastiness of converting everything and ripping out all the hooks in the engine.

[brenttaylor]

@ChrisCalef So much of the engine actually calls TS functions and does a lot of it's work in TS that running another language alongside it isn't going to solve a lot of the problem. Any new code you write, will in theory be faster, certainly. The internals of the engine however are still going to chug along.

As for why Mono is more of a pain... Lua, for example, has a very simple API designed specifically for embedding itself in a code base. It's trivial to work with. It really is a few function calls and exposing functionality to Lua. Calling Lua is also very simple.

Mono on the other hand is actually very simple to initially set up and call an assembly to start working. Exposing functionality to C# is also fairly trivial. The hard part is calling into C# from C++. It's technically trivial to do, it's just not a small amount of code. You however do need to be very careful when keeping references and pointers around. You can get very subtle bugs. But again, I'm very much looking at it from the perspective of replacing Torque, not embedding a language to work alongside it. For information on embedding Mono, take a look here: http://www.mono-project.com/Embedding_Mono

[Demolishun]

There is already a .NET script language in the works and it will be a produce of WinterLeaf which you can purchase. They have already done the hard work of integrating and to me it would be worth the price.

@BrentTaylor, I don't think there will be a speed issue if the added scripting language calls the C++ call directly. The console actually calls the C++ directly, but it has to do string lookups and string parameter parsing and those are slow. I think James pointed out the issues above. With a language such as Python or Lua you don't have that slowdown. So if you can keep the scripting language from depending on the console and export functions and objects directly then it will actually be faster. The problem is exporting the information. As I pointed out above the function that is supposed to provide the map to this is not currently working.

Now, there is a compromise that could be made. That is have the scripting language lookup the function and objects then create a reference to those so the next time they are accessed it is faster as it would be direct access to the object. This is something I have partially implemented. This does not require the binding code to work.

I am starting to think that we should not abandon TS right now. However, if we did graft on a VM that worked fast by calling functions directly then it would be a little more effort to have the TS compiled for that VM. Then the TS would benefit from that speedup. So you don't lose the code and you get the speed. So people would not even need to program in a different language to still get the speed advantage.

One place to start is the code James pointed out. That is the code that supposed to export the binding points. It is literally a mapping of all the commands that are exposed to script in the compiled DLL. So if that were working a mapper function could be written for any scripting language. So that should be made to work to reduce the hacks needed to connect to the engine now. I know a lot of the hacks because I came up with a bunch!

Do we want to take a stab at this for a few weeks and see where we get?

[brenttaylor]

@Demolishun Having whatever language we tack on call C++ functions directly, as mentioned, only solves half the problem. That part I understand and it's relatively trivial to do. It's been done before, as far back as the old TGE days with Minions of Mirth and their Python resource. :) But again, that only solves half the problem.

As you mentioned before: "There is another problem about using another scripting language. The C++ in many places calls TS functions. That would have to be reconciled if the scripting language gets replaces. Even SimObject creation is called using a script function from a C++ call."

If your intention is only to speed up new code, then what you propose will work just fine. The engine itself should see a rather impressive speed boost however, if we solve the real problem. As I proposed before, we can either replace Torque Script with another language or we can keep the TS language syntax but compile it down to the Lua (or V8, or what have you) VM's byte code.

The second option solves both the speed issue and allows everyone to keep their old code bases. The downside is that the interface to TS internal to the engine is likely to change. So we'd have to do quite a bit of cleanup. I think it's worth the effort. I think it's the overall best solution to keep everyone happy and get the speed benefits which are the real issue.

However, I'd still prefer to just remove Torque Script all together. I just doubt the community, as it is today, will really want it. People looking to use the engine now that it's under MIT...that's another matter. I highly expect anyone new to the engine to be highly put off by TS.

[Demolishun]

You are right and those calls to the console would have to be intercepted. That should not be difficult.

I actually ported the original PyTorque to T3D and created a resource. That was my first attempt. However, it had problems. One thing is it did not release the GIL properly. This made it slow and did not thread properly. This was fixed in that first resource. I am guessing PrairieGames fixed this in their internal version for their game later. I also changed the callback mechanism when I rewrote my own resource using SWIG. The original PyTorque used a hack to get the namespace. I made a new console function type that included namespace information.

I don't really understand it, but I want to see if we can get the engineAPI stuff working. I hope there is someone hanging around that knows what that was supposed to do. That would help solve the issue with connecting to another VM.

I expect to have some competing implementations. When the dust settles we will have choices. What path were you thinking of taking?