MB Performance (speed) - Githubissues

pikurasa commented 7 years ago

Performance improvements need to be investigated for Music Blocks.

How do we improve MB's performance? How is GUI affecting performance? How does tone.js affect performance? Other?

I think a good goal would be to get MB to run at 1/3 of the processing power that it currently uses.

Devin

pikurasa commented 7 years ago

I noticed some things re: performance today:

the P-T Matrix runs much, much more smoothly than just running the Music Blocks code itself
the Rhythm Ruler runs alight (perhaps better than MB code), but not nearly as well as the P-T Matrix.

I think that we need to prioritize performance improvements for a while. The music is not performing the correct rhythms as a result, which is very problematic.

Hrishi1999 commented 7 years ago

For the Windows bundle, the performance is still the same (good). Still needs some work.

walterbender commented 7 years ago

Also see #788 And we need some decent performance tests so we can begin to measure the impact of any changes.

walterbender commented 6 years ago

I wrote a simple playback mechanism that in theory should be pretty similar in quality to the widget code: it simply accesses the synth without any block processing overhead. It is not exposed it in the UI except through a keyboard shortcut: Alt-P.

To test, run a project. After it completes, a playback queue will be available, Alt-P will replay the performance.

Caveats: (1) I haven't added graphics yet; (2) Multiple Alt-Ps in a row, while the performance is running will lead to overlapping performances.

I've done very little testing. Would be great if more people tried it:

https://github.com/walterbender/musicblocks/tree/playback

walterbender commented 6 years ago

I added the graphics internal to notes. Seems to work well. Still need to add graphics not processed in the notes, e.g., setxy, and also the set volume commands.

pikurasa commented 6 years ago

To test, run a project. After it completes, a playback queue will be available, Alt-P will replay the performance.

It would be nice if there were a way to set the queue without a full, realtime playback--something similar to lilypond save. Can we try to implement that?

What I have tested so far works well.

walterbender commented 6 years ago

I still have some issues with the graphics that occur outside of notes (their order of execution is messed up).

walterbender commented 6 years ago

Graphics issues are fixed (although there are a few commands I don't yet queue: fill, bezier). Still need some sort of lock. Please test when you get a chance. UI issues need to be discussed.

walterbender commented 6 years ago

fill and hollow line remain the only graphics blocks not yet set up to work with playback.... it is a bit tricky since they are clamps. I need to think about the set volume block as well.

walterbender commented 6 years ago

Added fill... works. I'll do the same for hollow line. Need to figure out a volume strategy (Related to #769 as well).

pikurasa commented 6 years ago

I am getting good results with this feature. Steve Reich's clapping piece sounds good in this mode.

It may make sense to have a play, pause, rewind, etc. as one has on an audio transport (separate from running the code) https://upload.wikimedia.org/wikipedia/commons/thumb/8/84/ADAT-XT.jpg/799px-ADAT-XT.jpg

walterbender commented 6 years ago

Thanks for testing. I just pushed a few more tweaks. Pause, resume, and rewind sound like good ideas. The question is: how to expose this in the UI?

walterbender commented 6 years ago

A friend of mine made a nice analogy for what we are doing. Running MB is essentially running a compiler. Running playback is executing the compiled code.

So, I will add the compiled code to the end of the .tb files so that you can rerun the compiled code without "recompiling".

pikurasa commented 6 years ago

Pause, resume, and rewind sound like good ideas. The question is: how to expose this in the UI?

With the image I uploaded, I was suggesting a Transport in the UI. Many Digital Audio Workstations (DAW) have such a feature in the UI.

This image has the transport right at the top for Ardour, a well known libre DAW: https://ardour.org/images/retina_no_plugs2.png

My best idea so far is that a button initiates this transport when the user wants it (like a widget).

If we find it helpful enough, we could even give it permanent residence in the toolbar--on the right-center side, maybe.

A friend of mine made a nice analogy for what we are doing. Running MB is essentially running a compiler. Running playback is executing the compiled code.

Yes, I agree. I also like calling in "compiler" or "compiled code". At first, I thought "wait, this is just as much compiled code as a recording is 'compiled code' of a jazz chart." However, I do think that, since it is still running the code, that it is still different from a wav or mp3 (flac, ogg, etc) recording.

So, I will add the compiled code to the end of the .tb files so that you can rerun the compiled code without "recompiling".

That seems like a nice and convenient feature.

walterbender commented 6 years ago

Added save/restore by appending the playback queue to the end of the .tb file.

walterbender commented 6 years ago

I need to wrap my head around pause and resume (it gets a bit tricky if you have multiple voices running a polyphonic rhythm). So maybe that is deferred to GCI 2017.

The other thing we could add is a mechanism for playing back with a delay between notes, or step through note-by-note. Again, a bit tricky with the polyponic rhythms.

walterbender commented 6 years ago

Need to add the speak block to the list of blocks to play back.

walterbender commented 6 years ago

(1) should mice blink in playback? (2) should status widget update in playback? (will either 1 or 2 impact quality?)

walterbender commented 6 years ago

Added a widget with a play button that appears whenever there is something "compiled" that can be played. This is where we can eventually add playback for each individual voice, plus controls for rewind, pause, resume, etc.

Not 100% certain that this won't confuse our users :( Needs lots of testing.

pikurasa commented 6 years ago

..hmmm...

The commit I am on does not get past the splashscreen.

I am on: HEAD is now at f580853... add playback widget

...and the master branch seems to work just fine.

walterbender commented 6 years ago

can you please add your console log?

pikurasa commented 6 years ago

Log from playback branch: HEAD is now at f580853... add playback widget

Tone.min.js:7  * Tone.js r10 * 
webL10n.js:144 XMLHttpRequest cannot load file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/localization.ini. Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource.xhrLoadText @ webL10n.js:144
Tone.min.js:7  * Tone.js r10 * 
file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/js/playback.js Failed to load resource: net::ERR_FILE_NOT_FOUND
require.js:1765 Uncaught Error: Script error(…)
platformstyle.js:29 On platform:  Object
dsp.js:2303 Uncaught ReferenceError: module is not defined(…)
prefixfree.min.js:5 XMLHttpRequest cannot load file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/css/activity.css. Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource.link @ prefixfree.min.js:5
prefixfree.min.js:5 XMLHttpRequest cannot load file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/css/style.css. Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource.link @ prefixfree.min.js:5
text.js:303 XMLHttpRequest cannot load file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/lib/sugar-web/graphics/activitypalette.html. Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource.text.get @ text.js:303
index.html:79 ServiceWorker registration failed:  DOMException: Failed to register a ServiceWorker: The URL protocol of the current origin ('null') is not supported.

Log from Master branch: HEAD is now at a8ab7c5... fix regression in plugin code

 * Tone.js r10 * 
XMLHttpRequest cannot load file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/localization.ini. Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource.
 * Tone.js r10 * 
On platform:  ObjectFF: falseFFOS: falseandroid: falseandroidWebkit: falsemobile: falsetablet: false__proto__: Object
Uncaught ReferenceError: module is not defined(…)
XMLHttpRequest cannot load file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/css/activity.css. Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource.
XMLHttpRequest cannot load file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/css/style.css. Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource.
XMLHttpRequest cannot load file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/lib/sugar-web/graphics/activitypalette.html. Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource.
ServiceWorker registration failed:  DOMException: Failed to register a ServiceWorker: The URL protocol of the current origin ('null') is not supported.
XMLHttpRequest cannot load file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/lib/mespeak_config.json. Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource.
Uncaught SyntaxError: Unexpected end of JSON input(…)
XMLHttpRequest cannot load file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/lib/voices/en/en.json. Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource.
Uncaught SyntaxError: Unexpected end of JSON input(…)
initing i18n for music terms
1.0666666666666667
LOAD START
loading start and a matrix
adding a new turtle start
Finished block loading

walterbender commented 6 years ago

I have no idea. Maybe something is broken in your local storage? Can you try running window.localStorage.clear(); in the console and then reload the window?

pikurasa commented 6 years ago

Can you try running window.localStorage.clear(); in the console and then reload the window?

I tried it (I had never run anything in the console, so that was a new experience), but it did not change anything. When I ran the command, it said "undefined" (I tried with and without semicolon).

I get the same results in IceCat.

I do not know if it is helpful, but the end of the console for IceCat says:

origin/playback

XML Parsing Error: syntax error
Location: file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/css/activity.css
Line Number 1, Column 1:  activity.css:1:1
XML Parsing Error: syntax error
Location: file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/css/style.css
Line Number 1, Column 1:  style.css:1:1
XML Parsing Error: junk after document element
Location: file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/lib/sugar-web/graphics/activitypalette.html
Line Number 4, Column 1:

origin/master

NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIURI.host]  load-watcher.js:109
XML Parsing Error: junk after document element
Location: file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/lib/sugar-web/graphics/activitypalette.html
Line Number 4, Column 1:  activitypalette.html:4:1
initing i18n for music terms  activity.js:191:9
XML Parsing Error: not well-formed
Location: file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/lib/mespeak_config.json
Line Number 1, Column 1:  mespeak_config.json:1:1
XML Parsing Error: not well-formed
Location: file:///home/devin/Computer-Code/musicblocks-all/walter-bender-musicblocks/lib/voices/en/en.json
Line Number 1, Column 1:  en.json:1:1
1.065  activity.js:2731:25
LOAD START  activity.js:2120:13
restoring session: [[0,["start",{"collapsed":false,"xcor":0,"ycor":0,"heading":0,"color":10,"shade":50,"pensize":5,"grey":100}],800,75,[null,248,null]],[1,["newnote",{}],932.5,296,[null,2,5,9]],[2,["divide",{}],1030,296,[1,3,4]],[3,["number",{"value":1}],1115.5,296,[2]],[4,["number",{"value":4}],1115.5,327.5,[2]],[5,["vspace",{}],946,327.5,[1,6]],[6,["pitch",{}],946,359,[5,7,8,null]],[7,["solfege",{"value":"sol"}],1019.5,359,[6]],[8,["number",{"value":4}],1019.5,390.5,[6]],[9,["hidden",{}],932.5,453.5,[1,10]],[10,["newnote",{}],932.5,453.5,[9,11,14,18]],[11,["divide",{}],1030,453.5,[10,12,13]],[12,["number",{"value":1}],1115.5,453.5,[11]],[13,["number",{"value":4}],1115.5,485,[11]],[14,["vspace",{}],946,485,[10,15]],[15,["pitch",{}],946,516.5,[14,16,17,null]],[16,["solfege",{"value":"mi"}],1019.5,516.5,[15]],[17,["number",{"value":4}],1019.5,548,[15]],[18,["hidden",{}],932.5,611,[10,19]],[19,["newnote",{}],932.5,611,[18,20,23,27]],[20,["divide",{}],1030,611,[19,21,22]],[21,["number",{"[…]  activity.js:2169:25
adding a new turtle start turtle.js:1122:13
Finished block loading

walterbender commented 6 years ago

I think I know what is wrong. I forgot to add playback.js to the git repo. Please pull again and hopefully it will work. My apologies.

pikurasa commented 6 years ago

I think I know what is wrong. I forgot to add playback.js to the git repo. Please pull again and hopefully it will work. My apologies.

No worries--it works!

This is what I suggest in terms of UI:

Instead of running the code and then having the transport pop up as an option, I think that "compile" should be an option in the toolbar. When "compile" is run, it runs quickly like when MB is generating .ly code and when it is complete the transport with options popup for the user (I have a preference that the transport stay in the toolbar--not floating--perhaps it is just from using other applications).

I think that running the code could ALSO result in the complied code for playback (as it does now), but I think that having a separate option for "compile" that runs quickly through the code (as described above) could be very useful at times.

Thoughts?

walterbender commented 6 years ago

Glad it is working.

I will explore the separate compile idea. It can be a bit non-deterministic but maybe we just live with that. I will leave the compile as we play mechanism available in any case.

Re toolbar or widget, I am struggling with that. If we eventually want advanced playback features, I think we'll need a widget. Maybe in the short term, a button will suffice. But where to put it???

pikurasa commented 6 years ago

I will explore the separate compile idea. It can be a bit non-deterministic but maybe we just live with that. I will leave the compile as we play mechanism available in any case.

At any rate, we will always have deterministic interpretation. This adds another option. Users could discover the merits and downsides of both.

Re toolbar or widget, I am struggling with that. If we eventually want advanced playback features, I think we'll need a widget. Maybe in the short term, a button will suffice. But where to put it???

Could it be a widget that can drop into the toolbar (and be pulled out to float)?

walterbender commented 6 years ago

It could be pulled out from the toolbar like the utilities panel or the save panel (where save to lilypond is found). But I still need a place to put the button. Maybe also on the save panel? Or the utilities panel? I am loathe to introduce another button on either toolbar.

walterbender commented 6 years ago

I have a solution: I will combine the Cartesian and Polar buttons and thus make a new slot on the right-hand toolbar. I can add a new panel there for compiling/playback.

walterbender commented 6 years ago

I have migrated over to a menu button that pops up a panel. Next up, "fast compile". After that I will think about how to pause/resume, rewind, etc.

walterbender commented 6 years ago

quick compile is working. I also added turtle blink. @pikurasa can you please test to see if adding turtle blink adversely impacts performance?

A few more details to work out: the playback button should gray out if nothing is compiled. Rewind is not implemented Pause/resume is not implemented

walterbender commented 6 years ago

Playback button should be maintaining state properly

walterbender commented 6 years ago

Aargh. There is a bug in the quick compile. Since we don't actually run the turtle commands, they don't update paramters such as heading and x position. If the code depends on these parameters, then it doesn't compile properly. I may have to run the turtle commands (with the pen up) during quick compile, which will slow it down.

walterbender commented 6 years ago

This is also an issue for the count notes block and the generate lilypond commands.

pikurasa commented 6 years ago

I see... lots of corner cases.

Sorry for the trouble.

At the very least, it will be helpful for testing.

I could not test today and am leaving to teach, but will let you know the results when I do.

walterbender commented 6 years ago

fixed it.

To do: shell, show, and probably some other more obscure commands need to be accounted for. note count should maintain/restore the state of the turtle

walterbender commented 6 years ago

added support for a few more blocks: show, setheading, speak

walterbender commented 6 years ago

added setvolume

walterbender commented 6 years ago

created a separate ticket #794 for the note counter, which is broken in many corner cases.

walterbender commented 6 years ago

Added pause/resume support.

walterbender commented 6 years ago

need to update sw.js

walterbender commented 6 years ago

fixed sw.js

@pikurasa at this point, I think the playback code is stable enough to move to Master. That is not to suggest we don't need to keep working on performance. Maybe a series of issues targeting specifics would be good, starting with a decent test suite so we can know the impact of any changes on performance.

walterbender commented 6 years ago

@pikurasa any suggestions/feedback re this feature would be helpful.

pikurasa commented 6 years ago

any suggestions/feedback re this feature would be helpful.

I like where you put it.

I do not really understand the interaction between the "start over" icon and the "play icon". When the performance is complete, they both do the same thing. Otherwise, I like the visual presentation of the icons--I think your sense was good.

...I still need to test functionality more. I will try some things now...

walterbender commented 6 years ago

In theory, the rewind button is gray if the performance is at the beginning. Otherwise it is active. The play button should be active or set to pause except when a performance is over.

On Sep 18, 2017 12:22 PM, "Devin Ulibarri" notifications@github.com wrote:

any suggestions/feedback re this feature would be helpful.

I like where you put it.

I do not really understand the interaction between the "start over" icon and the "play icon". When the performance is complete, they both do the same thing. Otherwise, I like the visual presentation of the icons--I think your sense was good.

...I still need to test functionality more. I will try some things now...

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/walterbender/musicblocks/issues/726#issuecomment-330276149, or mute the thread https://github.com/notifications/unsubscribe-auth/ADz74TnAbiNgy87HmV8uoXh-bvsGlCTRks5sjpitgaJpZM4Osj-E .

pikurasa commented 6 years ago

the rewind button

It would be nice if one could slide through time from beginning to end. (need to go, but can add more details later)

walterbender commented 6 years ago

Please write up the time slider (and progress bar) as a separate issue we can use in GCI. Won't fix here as we are in feature freeze.

walterbender commented 6 years ago

Fixed a few minor issues: more compact note representation in the "compiled" code; fixed a bug with playing back setcolor; saving/restoring turtle state when compiling.

pikurasa commented 6 years ago

I had an idea on the topic of performance improvement.

I have a pretty old phone (360-ish RAM) and I installed a new messanging app onto it, which happened to outperform the stock messaging app--I was impressed. I looked at the logs of this app (yes, it allows logs by default) and it revealed some clever underworkings:

it recognizes that it is running on a low ram device
it skips frames

The latter is particularly interesting.

I think that MB live performance might be improved if MB skipped frames behind the scenes if needed--both visually and sound-wise. I know that the sound is constantly adjusting to the master clock, but I suspect that there are ways to give the CPU a break by skipping processes (maybe in tone generation) in order to land on the next note in time.

Visually, the turtle may skip over some live rendering processes in an elegant way when running out of system resources.

This may be something for MB, tone.js, or perhaps Chrome/ium.

sugarlabs / musicblocks

MB Performance (speed) #726