Playpen

[gogins]

This is the old computer music playpen idea springing right back into life after my favorable experience with Python GTK used by python-abx.

The computer music playpen is an integrated development environment (IDE) for computer music that maximizes functionality and minimizes development time. The idea is to have the shortest, easiest "edit, render in real time and listen., edit..." cycle that is possible. Objectives:

• [x] Form builder for UI design.
• [x] HTML5 support for visual music, 3-D scores, etc.
• [x] Csound support via ctcsound.
• [x] CsoundAC support via Python.
• [x] Arbitrary core generator support via \<CsScore bin="generator_binary">, can be used with Common Lisp, etc.
• [x] Ideally, the IDE should be able to edit itself, i.e. each piece is its own IDE and can edit itself. Or, the IDE supports all of the above features which create a complete program that is the piece, and one launches the piece from the IDE.
• [x] Ideally, completely cross-platform. Haven't tested but should work on all GTK/WebKit/Gstreamer platforms.
• [x] Maintainable going forward! Very important!

Current approaches that implement some of this:

1. Code editor for HTML5 piece that runs in NW.js. Cons: no form builder, incomplete CsoundAC support, need for script to run pieces. Pros: Full-powered, easily customizable, lightweight.
2. Code editor for CsoundAC written in Python. Cons: No form builder (but possible with addon to editor), no HTML5 support (but possible with PyWebKit). Pros: easily customizable, lightweight. Doing this one.
3. QtCreator. Cons: poor HTML5 support in QtWebEngine, more compex coding. Pros: Good debugging.

[gogins]

Investigate:

• [x] Google Web Designer. Don't think it will do, designed to create ads, can't import existing HTML files into the form builder.
• [x] Glade integrated in Scite to create UI for PyGTK.
• [x] Wrap CsoundAC for Node.js using SWIG. Not doing as Python seems likely to work out.
• [x] WebKit or CEF integrated in Python pieces: pywebview is alive, cefpython is alive but may be behind, pywebkit is dead. How to combine this with regular Python GUI? WebKit2 is in gi.repository.

[gogins]

I think the playpen is quite doable using Python GTK from a decent code editor. The editor should have custom commands for:

• [ ] Running glade.
• [ ] Running the app.
• [ ] Running the app with realtime rendering already started.

This is all doable with a naming pattern (piece.py goes with piece.glade) and with standardized command line options for the piece, as I already have for Python pieces, but internalized in the piece itself.

Then there can be a template for the piece that loads the Glade file, interprets the command options, and has standard menus and windows in the piece.

It seems better to just write a playpen application using Python GTK.

[gogins]

Changed the design to be a Python GTK3 application, as that looks like it might well be cross-platform and everything I have tried has worked pretty well so far. I'm impressed by the cleanness of GTK. Outstanding tasks:

• [x] GUI builder for controls, using a Glade tree re-parented to the controls page.
• [x] Handle different method names and types for the values of all widgets likely to be used for controlling pieces.
• [x] Actually use the file naming convention.
• [x] Run Glade with a new Glade file for a new piece, and the existing XML for an existing piece, and rebuild the GUI with the XML saved by Glade when it exits.
• [x] User-specific settings go in a dictionary in the user settings directory.
• [x] Customize the theme just a bit, mainly darker background with green letters for the Csound console, and a better color for the text in the code editor.
• [x] Add the post-processing stuff.
• [x] The GTK tutorial text editor is not really suitable. I am using GtkSource.View. Seems to work just fine.
• [x] Create a Csound syntax file for the GtkSource.View? Python and C++ and HTML all work fine. I've looked into it, there are a lot of details and I am not going to do this unless it turns out I start relying on the playpen.
• [ ] Add a better csd example piece with a widget demo.
• [ ] Make the Csound file the basic container for Glade, Python, HTML, etc?
• [ ] Add Python example piece into the application program, so that user code is integrated into the playpen runtime. This does happen if the playpen simply calls exec(piece_python, globals(), locals()). In order to prevent the GUI from freezing, I have subclassed ctcsound.Csound as GtkCsound and overridden the perform method to use the Gtk event loop to run Csound with co-operative multiprocessing.
• [ ] Add HTML example piece, showing how to communicate between the playpen and the HTML and its JavaScript context. It looks like this can be done with WebKit Process Extensions. Here is a more extended example.
• [ ] Improve documentation, write installation stuff from Ubuntu standpoint with notes about other platforms.
• [ ] Create a Python package for this, see https://github.com/pypa/sampleproject. This militates against the JavaScript wrapper unless I build it with setuptools.

[gogins]

User-specific settings should include:

• Author.
• Publisher.
• Year.
• Notes.
• License.
• Audio output device.
• Audio input device.
• MIDI output device.
• MIDI input device.
• Soundfile editor.
• Gnome theme name.
• GtkSourceView style scheme name.

In reality, the use of a naming convention to derive the output soundfile name means that none of the Csound settings are actually required, although I will keep the audio output device as a convenience. The audio input and MIDI settings I will not implement.

[gogins]

Post-processing tasks, which if any can be done with GStreamer? A glance shows GStreamer harder to implement than adapting my existing script.

• [x] Normalize.
• [x] Tag.
• [x] Convert to MP3.
• [x] Convert to FLAC.
• [x] Convert to MP4 with spectrogram image.

[gogins]

For triggers, it may suffice to send 1 for the pressed signal and 0 for the released signal.

It may be cleaner to have different handlers for different signals, or for different widget classes.

[gogins]

GtkSourceView looks usable but also tricky. See https://stackoverflow.com/questions/10524196/load-gui-from-a-glade-with-gtksourceview-in-pygobject. I have to make Glade aware of SourceView like this https://cjenkins.wordpress.com/2012/05/08/use-gtksourceview-widget-in-glade/, but that didn't work. But adding the old directory for the catalog did work.

API is here: https://developer.gnome.org/gtksourceview/stable/.

Looks like I have to create the editor programmatically and swap it in for the current GtkTextView. No, I got Glade to use the GtkSource.View by adding its catalog directory.

Search and replace is now working but you always have to press enter in the search entry field to get stated, even if you are then going to replace that text. This is not quite standard.

[gogins]

Just found out about GJS. But it's not relevant to my use case, nor is Seed.

Wrong! GJS is very relevant -- it contains the API references that I absolutely need. But there is no user guide!!

[gogins]

My question is, considering that ctcsound.py comes with Csound, is there any way to get it into the WebKit JavaScript context without the C compiler?

[gogins]

My problem right now is that I don't really want to re-implement csound.node for WebKit2. My choices seem to be:

1. Drop HTML support from the playpen.
2. Re-implement csound.node for WebKit2. But I don't see evidence that a lot of people are doing this kind of thing. I will look harder. But there seems to be SWIG support. I am going to try this.
3. Implement my own rudimentary RPC protocol just in Python and JavaScript for a Csound object. This ends up re-doing exactly what the QtWebEngine does, and I hate that.

[gogins]

May need this: https://webkitgtk.org/reference/jsc-glib/stable/JSCContext.html.

[gogins]

OK, this is pretty bad. WebKit2Gtk+ deprecated webkit_frame_get_javascript_global_context because the browser now uses GJS instead of JSC. There are now two APIs for JavaScript in WebKit, one from Apple and one from Gtk.

It's only a problem for me because SWIG uses the wrong API, which is not the one used by Gtk. It would be fine if I were writing a C/C++ program to use WebKit, but it is not fine if I am writing such a program to use WebKit2Gtk+.

I may have to code the wrappers by hand.

[gogins]

As a proof of concept... Jesus that was hard! Finally got "hello" to say "Hello from Csound!"

1. You must use these APIs: https://webkitgtk.org/reference/jsc-glib/stable/.
2. But you must figure how to follow this pattern: https://karhm.com/javascriptcore_c_api/.
3. And then you must add the created function back into the JavaScript context:

       jsc_context_set_value (js_context,
                      "hello",
                      hello);    

[gogins]

I am now trying to see if this machinery is accessible from Python, which would save me from packaging a native module. See https://gjs.guide/.

I can do this if and only if the GJS wrappers for Python implement registering a Python function as a JavaScript function and/or method in the Web page's JavaScript context.

[gogins]

https://github.com/cztomczak/cefpython is an option. It can introspect Python objects and dynamically create JavaScript bindings to classes and methods. Unfortunately it does not support Python later than version 3.7 on Linux.

There is a mad plethora of GUI frameworks, browser frameworks, and form builders for Python, not all of which work and none of which constitute a version-compatible set of libraries with all features that I need (compatible with ctcsound, embeddable form builder, full-featured browser that exposes ctcsound).

[gogins]

There's a hack for supporting cefpython3 on Python3.8 that works locally, just add the version number in init.py, and a symbolic link from the Python3.8 site-packages.

Now message.html plays from cefpython. But the GUI responsiveness gets turned off. I think I can deal with this by performing bufferwise and yielding to the event loop.

No, I can't get that to work. There are too many sharp edges with cefpython.

[gogins]

Of course there is no user guide for using GJS... Perhaps people just write native code and register it for GObject Introspection, I could try doing that with Csound.

[gogins]

Back with cefpython. Everything is working in Python and in CefPython using my new SWIG wrapper for CsoundThreaded.

Not everything is working! Getting messages to print for the user by polling the internal Csound message queue works fine in Python, but is not working in JavaScript. Although everything else is working in javaScript.

Print statement debugging shows everything is fine in CsoundThread right up until the number of messages pending is returned. Evidently it is properly returned from C++ to Python but not from Python to JavaScript.

[gogins]

Plan:

• [x] Try installing all the Python/GTK+ stuff for Python 3.7. Done.
• [x] If that works, see if the polling starts to work and if the gtk3.py example works. See https://github.com/cztomczak/cefpython/issues/534, this is a show stopper for this approach to the playpen, cefpython will just not work with Gtk 3 without patching and rebuilding dependencies, and this I am not going to do.
• [ ] If those work, use cefpython3 to implement the browsers in the playpen.
• [ ] If they do not work, try to do classes in GObject introspection for the CsoundThread native class.

[gogins]

This installed pip for python3.7:

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
sudo python3.7 get-pip.py
sudo apt install libgirepository1.0-dev
sudo python3.7 -m pip install --ignore-installed PyGObject

Now I can run the Python stuff in the playpen with Python 3.7, but gtk3.py still segfaults.

[gogins]

My choices have now become:

• [x] Implement native CsoundThreaded for Gtk 3. I have got the constructor to return an empty object and if I can add a callable function to that object, I will proceed on this path.
• [x] Implement the form builder, code editor, and Python runner in CEF or cefpython. Not if any of the other options works...
• [x] Try another Python framework. PyQt5 works except for not printing the Csound runtime messages, it was easy to adapt the cefpython qt.py example. But this is a two step process, run qtcreator or qtbuilder then massage the ui file to make Python code that can then be used.

[gogins]

OK, the method implementation works and the pattern is now clear.

[gogins]

Can't seem to get callbacks set from Python and called from JavaScript working. But I can try a workaround, the callback is actually not a callable but just the name of the callback and it is called simply by executing JavaScript code from the CsoundThread inner callback. Can't get this to work either, leaving the message callback unimplemented.

This is not a show-stopper, as the messages are printed in the terminal.

Removed the messages pane from the UI, leaving all messages printing to stdout and stderr.

I may come back to this.

[gogins]

Remaining issues before cleaning up:

• [x] Restarting Csound in the browser. Put in Csound.Join after Csound.Stop and it seems to work now.
• [x] Using the playpen's global instance of Csound instead of a new instance; this is not a show-stopper. It may be due to the process separation in WebKit. After studying the documentation, it is easy to send the CSOUND address to the JavaScript engine, but as that is indeed running in a separate process, it is useless. So, not doing this.

[gogins]

Cleanup:

• [x] Remove unneeded diagnostic messages.
• [x] List all dependencies inREADME.md.
• [x] If the user clicks the Edit button, create and open a new Glade file if none exists for the piece.
• [x] See if values of widgets can be saved and restored after performing. Can't see it. Maybe I can do it myself as the Builder remains live. I only need to serialize channel values. I am going to try to use an XML DOM parser library. Didn't work. Actually just saving a dictionary of channel names and values, that seems to work fine, though in a separate file.
• [x] Clean up settings, add metadata as well as a style for the code editor.
• [x] If after editing/running a piece another piece is opened, things get messed up, and it is possible to wrongly overwrite a piece. This was caused by inconsistent use of a global variable for the piece filepath and the piece code, now fixed.
• [ ] After removing all widgets from the user controls layout, the controls panel is not empty as it should be.
• [x] Better examples (see next comment).
• [x] Add rudimentary User Guide to README.md.
• [x] Replace build scripts with a CMakeLists.txt.
• [x] Packaging -- keep it simple. Right now there is just a make install target, I may add a Debian package or other installers later.

[gogins]

Examples:

• [x] xanadu.csd, plain Csound piece with no extras, just to prove that things are working.
• [x] sierpinski-csound.py, Python and musx piece with a Glade-defined user interface; use a pluck that uses the controls! Using wgpluck2.
• [x] message.html, HTML piece with a JQuery-based user interface.

[gogins]

The XML parser will read the .glade file into memory and listen for value changes on control widgets, updating the XML in response. When saving, the .glade file will be written from the DOM and not from the .glade file previous read. This will work only if:

• [x] All control widget values can be saved in the .glade file. The Adjustment instances are indeed loaded from values in the .glade file. I assume other numeric control widgets would be as well.
• [x] The XML written from the DOM will read back into the Builder and restore the control widget values. Yes it does.
• [x] Save the current values of all control widgets to the UI file. _Not using the UI file, using a separate piece.ui.channels file.

[gogins]

The hard part is looking up elements from widgets. Some kind of extra metadata might simplify this. I can't arbitrarily add properties to the ui xml. But I could create a dictionary when I connect the signals.

[gogins]

The widget classes that I will save and restore, how their values can be accessed, and the xpath to the ui element that stores the value. This xpath begins at the element with the same id as the control channel name. "gk_channel" here stands for all channel names. For every get there is a corresponding set. Some of these are subclasses of others, but I need the specific class names in order to handle setting the values properly.

• [x] GtkButton, event data parameter: not saved, but sent to channel.
• [x] GtkCheckButton, get_active method, event data parameter.
• [x] GtkEditable, get_text method.
• [x] GtkRadioButton, get_active method, event data parameter.
• [x] GtkScale, get_value method.
• [x] GtkScaleButton, get_value method.
• [x] GtkSpinButton, get_value method.
• [x] GtkSwitch, event data parameter, get_state method.
• [x] GtkToggleButton, get_active method.
• [x] GtkVolumeButton, get_value method.

There are three categories: the get_value method, the get_active method, and the get_text method.

[gogins]

There is a fairly serious problem with the UI XML. Default widget values are not saved from ElementTree. I may be able to work around this, or I may use another method of storing the control channel values, e.g. with JSON for the map of channel names and values.

There are no Glade settings to control this, so ElementTree and GTK Builder are almost certainly not the problem.

I could work around this by constructing the elements for widget values in the XML, but I think I will go with a separate file, that should be less fragile.

[gogins]

I have a simple Python dict holding elementary type values for string channel names. This is saved and restored using JSON from piece.ui.channels. It is a separate file and I hate that, but the solution is simple and should be robust.

[gogins]

I'm closing this now. I'm going to start using it. No doubt there will be bugs. They will become separate issues.