Exporting functions for ReaScript

[Jeff0S]

From fingersmcgarnickle on December 14, 2011 13:17:09

I wrote a bit of throwaway code to test accessing ReaScript from reaper_sws.dll. Basically you export functions from the dll and then you can call them from ReaScript using the ctypes module.

This will enable users to edit MIDI events using ReaScript from the MIDI editor, without chunk editing. It also opens up new levels of customization capabilities, beyond this.

I am thinking I'll just export one function called ReaScriptImport which will return function pointers to functions from a table. The python import module could be generated from an action or shipped in the installer.

Does this sound okay to implement here or should I start a new project to do this. I think a fair few people would use this as Python is a pretty simple language to get your head around, given some examples.

Original issue: http://code.google.com/p/sws-extension/issues/detail?id=432

[Jeff0S]

From jeffos...@gmail.com on December 14, 2011 05:30:01

Excellent idea! Also, "delegating" chunk edition to a C++ extension will break the the 1 Mo limitation for ReaScript (show stopper with real projects..).

Personnaly, I would vote for the SWS solution rather than a distinct project: many, many things could be shared with ReaScript users!

[Jeff0S]

From fingersmcgarnickle on December 14, 2011 06:23:54

For sure. It is not only the 1Mb limit it is also the speed of chunk editing in C++. Plus most useful stuff has been coded already so writing Python wrappers is alot less work then reimplementing in Python. There is a lot of other stuff which could be done like:

• Session based storage of data for scripts
• Creating custom dialogs (would be a bit of work but possible)

I think the best way to do it is to have one function exported for Python to call, as described above, and one action to write a python file in the plugins or Reaper folder. This will be similar to how Reaper exports the reaper_plugin_functions.h file. This means nothing new is added to the installer and the python module is always up-to-date, or can be generated again if not.

[Jeff0S]

From swstim on December 14, 2011 07:49:50

Yeah for sure add this sort of thing to SWS. While I understand what you mean by exporting functions, have you considered using:

// register("myAPIfunc",funcaddress) then other plug-ins can use GetFunc("myAPIfunc") // (use "-myAPIfunc" to unregister)

(from reaper_plugin_functions.h)

Then maybe all you have to do is call RPR_SomeFunc() in Python and it automatically sets up the call? Schwa would know about this, I can ask him if you like.

As an added bonus then functions could be available to other C++ Reaper extensions.

[Jeff0S]

From fingersmcgarnickle on December 14, 2011 11:52:35

Hmmm... but you still need the function signature. Python needs to know the calling convention, parameters and return type. So there still needs to be some python ctypes code to call the function, which the inbuilt ReaScript code would not know.

[Jeff0S]

From swstim on December 14, 2011 12:07:25

Right, I wonder if there's a way that you could "communicate" the function sig to reaper and have it spit it out into the same place it puts all the RPR_ function signatures. (Which is where we need help from schwa) There's already something in place in Reaper for it, it would be nice if we could leverage it in this situation.

[Jeff0S]

From fingersmcgarnickle on December 14, 2011 12:44:38

That would be the file reaper_python.py in the new 4.14 pre-releases. It has all the ctypes code in there for each RPR function. It is now installed as part of Reaper (not generated). The idea would be to generate a new file which does something similar to reaper_python.py.

I guess we could ask Schwa how he generates the reaper_python.py file. We couldn't extend this module, just create another similar one.

I agree it would be nice to use existing Reaper functionality for this, but I can't really see a way to do this with the new way ReaScript is implemented.

Also, the functions exposed would just be the basis for helper python code to stop memory leaks and make it easy to use. For example a function like openMidiTake would alloc memory and a function closeMidiTake would free it. Expecting ReaScripters to handle this sort of stuff is not ideal. So a python class which opens a midi take on construction and closes it on deletion is needed.

[Jeff0S]

From schw...@gmail.com on December 14, 2011 14:23:06

We explored a lot of different ways to do the embedding and settled on what is in REAPER 4.14: a static .py module that wraps each function, and a dynamic table of function addresses that is generated at run-time. The extension could do the same thing: package a reaper_sws.py module and generate a run-time function address table. I think we could share the compile-time script that generates reaper_python.py from metadata. I'm happy to answer any questions about why/how we implemented this, either here or on the REAPER forums.

[Jeff0S]

From schw...@gmail.com on December 14, 2011 15:25:23

Thinking more, REAPER could put the sws functions into the single dynamic function address table. The extension could package reaper_sws.py, generated at compile time, similar to reaper_python.py. At run time, the extension could register reascript export functions with REAPER, so that REAPER can add those entries to the dynamic function address table. reaper_python.py currently keeps a file-local array with those addresses, all we would have to do is expose that array so that reaper_sws.py could access it.

[Jeff0S]

From fingersmcgarnickle on December 14, 2011 23:58:02

That sounds like a nicer way to do it. At the moment in reaper_python.py there is a dictionary _ft, keyed by the function name. So, maybe REAPER could add any registered API functions to that table (as Tim describes above). Then in reaper_sws.py we just import reaper_python, pull the function addresses from the table and write/generate the ctypes code for the functions.

[Jeff0S]

From schw...@gmail.com on December 15, 2011 05:06:10

Right. We'll make the change to the next REAPER release so that extension functions registered with "myAPIfunc" are added to the function address table, and we'll add an accessor function for _ft. What you guys will need to do is generate reaper_sws.py at compile time.

[Jeff0S]

From schw...@gmail.com on December 15, 2011 05:24:58

Correction: the comment is wrong in the REAPER API documentation. If you want to register/expose a function called myfunc() for reascript or other extensions to use, the syntax is register("API_myfunc"). Other extensions then get the function pointer with GetFunc("myfunc"). reascript users would get the function with RPR_myfunc.

Of course, other extensions will need your header in order to know the function signature, and reascript users will need your .py module, which contains the actual RPR_myfunc wrapper definition.

[Jeff0S]

From pyt...@yahoo.com on December 15, 2011 06:38:32

WOW !!! I hope this all works out.., it would open so many possibilities.

[Jeff0S]

From schw...@gmail.com on December 15, 2011 07:42:11

http://www.landoleet.org/dev/sws/ has a test build of REAPER:

Extension function addresses can be added with "API_myfunc" as before.

Extension function declarations and help text can be added with "APIdef_myfunc". Generate the API C++ header from REAPER to see more information about how to do this. After you add the declarations/help, it will be written to the API C++ header and the generated ReaScript documentation.

These functions will be added to the run-time function table. A python module that imports reaper_python.py can use rpr_getfp(myfunc) to get the function address.

Parameter wrapping/unwrapping functions in reaper_python.py are exposed to other python modules.

reascript_python.pl is what we run at compile-time to parse the exported REAPER functions and generate reaper_python.py. It is scanning a particular .cpp file for this struct:

APIdef g_apidefs[] = { { APIFUNC(functionname), "rettype", "parmtype1,parmtype2,..." }, };

Each line can contain additional information, such as function names and help (for your programming convenience), but only the name, return type, and parameter types will be parsed by the script.

Yes, that is a Perl script reading C++ to generate Python. Feel free to edit the script however you like.

Let me know if anything seems wrong or doesn't make sense!

[Jeff0S]

From fingersmcgarnickle on December 15, 2011 12:32:48

That was quick! Thanks Schwa, I'll give it a go.

[Jeff0S]

From jeffos...@gmail.com on December 15, 2011 14:07:02

Thank you schwa!

[Jeff0S]

From fingersmcgarnickle on December 22, 2011 02:26:40

I noticed this went in the latest pre-release. Sorry that I haven't had a chance to look at it yet. I was busy and now I am sick :-(

Soon though!

[Jeff0S]

From pyt...@yahoo.com on January 04, 2012 11:30:42

Hey fingers Happy New Year!!!

I hope your feeling better? I was just wondering how this project is coming along? I'm really excited about the possibilities this will introduce. Do you have a rough idea of when we might have something to work with or test out?

Thanks.

Anton

[Jeff0S]

From fingersmcgarnickle on January 05, 2012 02:34:07

Happy New Year to you too Anton!

I am okay now, just had the flu. I am currently working on pretty thorough testing of my MIDI chunk parsing and creation code to fix any bugs. The python part will be pretty easy and quick to do, but I want to be really confident the C++ code works properly. Hopefully a week or two.

[Jeff0S]

From pyt...@yahoo.com on January 05, 2012 05:27:58

Good to hear.., glad your being thorough too. It's going to be so nice to manipulate MIDI through ReaScript without having to deal with chunks. I think this is going to be that "missing link" that I've be waiting for ever since ReaScript was introduced.

[Jeff0S]

From pyt...@yahoo.com on February 02, 2012 20:03:03

Just thought I'd check in and see how close you're getting to a release?

[Jeff0S]

From mths.f...@googlemail.com on February 06, 2012 05:09:46

Hang in there Fingers! :)

[Jeff0S]

From fingersmcgarnickle on February 14, 2012 04:56:14

Sorry, haven't been able to do anything with this recently as I have been really busy with other stuff. So, I apologise for giving an incorrect ETA. It will hopefully be soon, I just need to find the time.

[Jeff0S]

From fingersmcgarnickle on March 08, 2012 15:15:00

Did something on this tonight but can't get it working (see http://forum.cockos.com/showthread.php?p=920823#post920823 )

[Jeff0S]

From fingersmcgarnickle on August 18, 2012 02:36:57

Sorry, still no progress on this. I still intend to do it but the last 6 months have been a bit crazy. New country, new job, new baby etc. No time for music or writing code.

[Jeff0S]

From mths.f...@googlemail.com on August 19, 2012 07:52:17

wow that sounds like heavy changes in your life, all the best! thanks for the status too, take your time.

[Jeff0S]

From jeffos...@gmail.com on September 06, 2012 10:36:53

Done on Windows OS, OSX: on progress.. See details and status in r837 . => feedback welcome!

__

@schwa (in case you still around..)

Many thanks again for all the details above, for the related REAPER updates and for sharing your scripts!

About those scripts:

• I guess they are easily subject to change.. Please, keep us updated (if you can!)
• Also, they seem to be Windows only.. I guess once generated you share those Windows' reaper_python.py files (32+64-bit) with a OSX building machine and just package those? (I am curious about that because OSX packaging is the last real issue I'm facing..)

Status: Started

[Jeff0S]

From jeffos...@gmail.com on September 07, 2012 00:53:37

^^ forget my remark about perl scripts in OSX: I got them working.

[Jeff0S]

From jeffos...@gmail.com on September 13, 2012 02:55:37

Done for OSX too ( r848 , r849 ). I let the issue opened a bit (need more feedback..)

__

Some details about my above remark "OSX packaging is the last real issue I'm facing..":

Unlike on Windows OS, we can't deliver different sws_python.py files in the "Plugins" folders (I mean the ones in REAPER.app, in REAPER64.app), i.e. REAPER install folders => so we have to deliver that in ~, i.e. REAPER resources folder => so we need 2 files (32 & 64-bit) since that resources folder can be shared by REAPER & REAPER64 (see bottom remark about that..)

Then, unfortunately, when delivering these files in "UserPlugins" user's script won't work (I mean without PATH tweaks, etc..): REAPER does not really manage this folder like the "Plugins" ones. => we have to deliver that in ~/Library/Application Support/REAPER/Scripts/

__

^^ ok, that why I did things like that on OSX.

That's ok but there's one thing I would like to improve on OSX so that things would be exactly like on Windows OS: ATM, on OSX, users have to import sws_python OR sys_python64 if they are running 32-bit or 64-bit. I'd like to add a 3rd "umbrella" file sws_python.py that would do something like (pseudo-code): if (x64) import sws_python64 else import sws_python32

I'll look if it is possible.. => users would not have to worry about that, just import sws_python like on windows OS

[Jeff0S]

From jeffos...@gmail.com on September 13, 2012 12:11:46

^^ done r858 now this is ready to be tested into the wild :)