Playback for visual music editor

[rualark]

Could you please help me choose a straightforward and robust interface for playback with the following requirements? I see that SynthController does that, but I will use my own visualization - should I still use SynthController or there is a more straightforward way?

• Initialize - .init(), then .prime() :ok:
• Play - .start() :ok:
• Pause and resume - .pause(), .resume() :ok:
• Change bpm - millisecondsPerMeasure :ok:
• Play note on click or input - .playEvent :ok:
• Start playback from arbitrary beat - is it "percent" of time or of beats in .seek()? Could you please specify this in docs?
• Receive "playback ended" event - how can I do it without SynthController audioParams.onEnded?
• Display vertical line moving with playback, like in full-synth example - I see that it can be done with CursorControl. Is there a simple way to just receive onEvent without SynthController or CursorControl so that I can draw cursor?

:warning: renderMidi gives significantly better sound than CreateSynth: sound of basic-midi.html is significantly better than sound of basic-synth.html and full-synth.html (I can create an audio recording to compare if needed). Why is that? Are there problems with renderMidi that forced moving to synth? Can renderMidi be used or is it deprecated? What is the difference between abcjs and abcjs-midi? Can aforementioned requirements be met using renderMidi?

[paulrosen]

TimingCallbacks is the basis for all that timing - when you start that, it sends callbacks for a lot of useful events - new notes, beats, subdivisions of beats, approaching the end of the line, and the end of the piece. It doesn't have any UI associated with it, so you can just hook into those callbacks and run whatever UI you want. SynthController calls TimingCallbacks so it is basically the same effect. I am trying to onion this so that you can have as much or as little functionality as is convenient.

seek takes percentage of the entire tune.

If you are watching the beats callback, you'll get one last callback at the end of the piece.

You can write your own class that takes the same entry points as CursorControl and pass that in. That was the intention. What I'd do is copy the CursorControl class to your own code and modify it as you want.

[paulrosen]

Oh - about renderMidi - there were lots of performance problems with that. It would work passably on my fast computer, but just not keep up on a lot of phones. About the sound quality, I think I know what is going on and I plan to make another pass on this code to improve it. I think the sound quality only suffers when you are playing more than one note at a time - have you noticed anything else with it? It's also a little quieter, which I'll also improve.

[rualark]

Currently renderMidi sounds much better than synth, the difference is huge. Here is my analysis of this problem.

Same music played in Chrome by renderMidi (from basic-midi.html example) and synth (from basic-synth.html example): listen

Synth seems to have gaps in audio, phasing and no stable reverb - compared to renderMidi. Let's look at it closer.

Loudness is generally the same, although renderMidi is a little louder (renderMidi on the left):

When you zoom in, you can see that there are abrupt gaps between notes played by synth, which probably kills music quality:

renderMidi does not have this problem:

If you zoom spectrum in more, you can see that even inside the note synth spectrum has severe artifacts, which probably cause this phasing sound:

renderMidi does not have this problem:

In mobile Chrome, both renderMidi and synth have audio artifacts.

I hope this information helps.

[paulrosen]

That does! I have an idea of what needs to change, but I need to figure out how to do it. This is a high priority issue for me.

[rualark]

Could you look at this issue please? I got it today from user. It seems that user input some notes (clicked note_ci) and then clicked play. All call stack with code is shown. https://sentry.io/share/issue/a2f5715aa8424e23969fd7b3e27d992e

[paulrosen]

Can you figure out what their abc string was? Or at least the instrument they were using? There is a "missing note". If I can reproduce it I can guard against it.

[rualark]

I am not sure if breadcrumbs are shown at the page that I shared above. Here are they with my annotations marked with bold font. I tried repeating these actions, but could not reproduce the error. I think we can suspend the issue if you also have no ideas.

Also, in the future I will try to send notation with errors.

Breadcrumbs

No previous state stored in this browser ok, we started with empty score with 4 parts and 4 measures in 4/4 time signature

extra {"arguments":["No previous state stored in this browser"]}

10:58:51

10:58:54

[object Object]

10:59:33

Click [object Object] 0 abcjs-staff-extra abcjs-time-signature abcjs-l0 abcjs-m0 abcjs-v1 abcjs-note_selected [object Object] undefined clicked somewhere on abcjs - can't understand where

extra {"arguments":["Click",{"type":"specified","value":"[\"[Object]\"]","abselem":"{\"abcelem\":\"[Object]\",\"bottom\":2.065290322580645,\"children\":\"[Array]\",\"duration\":0,\"durationClass\":0,\"elemset\":\"[Array]\",\"extra\":\"[Array]\",\"extraw\":0,\"heads\":\"[Array]\",\"invisible\":false,\"minspacing\":10,\"right\":\"[Array]\",\"specialY\":\"[Object]\",\"top\":9.934709677419354,\"tuneNumber\":0,\"type\":\"staff-extra time-signature\",\"w\":11.795,\"x\":88.05924999999999}"},0,"abcjs-staff-extra abcjs-time-signature abcjs-l0 abcjs-m0 abcjs-v1 abcjs-note_selected",{"line":0,"voice":1,"measure":0},"[undefined]"]}

10:59:33

10:59:33

ui.click

body.modal-open > div#abc > svg

10:59:36

ui.click seems that previous click was on clef or time signature, that opened a modal to change these. and user closed modal without changing

div.modal-content > div.modal-header > button.close[type="button"]

10:59:36

[object Object]

10:59:37

ui.click

body

10:59:38

Click [object Object] 0 abcjs-rest abcjs-d1 abcjs-l0 abcjs-m1 abcjs-v1 abcjs-n0 abcjs-note_selected [object Object] 0 user clicked a rest

extra {"arguments":["Click",{"el_type":"note","startChar":188,"endChar":191,"rest":"{\"type\":\"whole\"}","minpitch":7,"duration":1,"maxpitch":7,"averagepitch":7,"abselem":"{\"abcelem\":\"[Object]\",\"bottom\":6.396129032258065,\"children\":\"[Array]\",\"duration\":1,\"durationClass\":1,\"elemset\":\"[Array]\",\"extra\":\"[Array]\",\"extraw\":0,\"heads\":\"[Array]\",\"invisible\":false,\"minspacing\":1,\"right\":\"[Array]\",\"specialY\":\"[Object]\",\"top\":7.603870967741935,\"tuneNumber\":0,\"type\":\"rest\",\"w\":11.25,\"x\":205.846625}"},0,"abcjs-rest abcjs-d1 abcjs-l0 abcjs-m1 abcjs-v1 abcjs-n0 abcjs-note_selected",{"line":0,"voice":1,"measure":1},0]}

10:59:38

10:59:38

ui.click

body > div#abc > svg

10:59:39

ui.click

body

10:59:40

[object Object]

10:59:40

10:59:40

ui.click

img#note_ci user clicked a C note to change rest to note

10:59:42

[object Object]

10:59:43

ui.click

body > div#toolbar

10:59:43

10:59:43

ui.click user seemingly removed note that was input

a#undo.btn.btn-outline-white.p-1[title="Undo
(shortcut Ctrl+Z)"] > img#undoi

10:59:44

Click [object Object] 0 abcjs-rest abcjs-d1 abcjs-l0 abcjs-m0 abcjs-v1 abcjs-n0 abcjs-note_selected [object Object] undefined clicked rest again

extra {"arguments":["Click",{"el_type":"note","startChar":183,"endChar":186,"rest":"{\"type\":\"whole\"}","minpitch":7,"duration":1,"maxpitch":7,"averagepitch":7,"abselem":"{\"abcelem\":\"[Object]\",\"bottom\":6.396129032258065,\"children\":\"[Array]\",\"duration\":1,\"durationClass\":1,\"elemset\":\"[Array]\",\"extra\":\"[Array]\",\"extraw\":0,\"heads\":\"[Array]\",\"invisible\":false,\"minspacing\":1,\"right\":\"[Array]\",\"specialY\":\"[Object]\",\"top\":7.603870967741935,\"tuneNumber\":0,\"type\":\"rest\",\"w\":11.25,\"x\":125.370875}"},0,"abcjs-rest abcjs-d1 abcjs-l0 abcjs-m0 abcjs-v1 abcjs-n0 abcjs-note_selected",{"line":0,"voice":1,"measure":0},"[undefined]"]}

10:59:44

10:59:44

ui.click

body > div#abc > svg

10:59:45

[object Object]

10:59:45

10:59:45

ui.click input note C again

img#note_ci

10:59:48

[object Object]

10:59:49

[object Object]

10:59:52

ui.click user clicked play

img#playi

10:59:53

ui.click

img#playi

10:59:54

exception

TypeError: soundsCache[note.instrument][noteName].getChannelData is not a function

[rualark]

Analysis of current abcjs synth: Played with abcjs synth:

Played with NI Gentleman:

Sound here: https://audiomack.com/rualark/playlist/abcjs-sound-comparison

Problems:

1. Abrupt sound cutoff at the end.
2. Notes are played faster when exported from abcjs as MIDI than played with abcjs synth.
3. It looks and sounds like sound problems are not just at the beginning and the end of the note. It seems that they are during the whole note. The spectrum looks irregular, fractured.

[moonwave99]

I am facing the same issue - basically:

• renderMidi sounds way cleaner than synth;
• there is no way to retrieve the played pitches through the midi API, so for visualisation purposes I cannot use it.

[Chrome v83.0.4103.97 on OS X - abcjs v6-beta10]

What is the current status of this? Is there anything I can do in order to help you?

[thank you so much for working on this wonderful library btw!]

[paulrosen]

I think there are two problems, but when those are solved we'll see if there is more.

First, I'd like to improve the soundfont, at least for piano - that seems to be the one that can naturally sound good with midi. I'm investigating that now. If you can find a sound font where the individual notes can be split into a separate files I will incorporate that.

Second, I have an idea of something I can do when creating the sound buffers to improve it. I will do an experiment this weekend to see if that helps.

[rualark]

What is the problem with buffers?

[paulrosen]

I am doing a very simple addition to get two notes into a buffer if they are in the same voice. I am going to experiment with not doing any of my own calculations and create as many channels as needed to keep all the notes separate.

[rualark]

Playback is very much improved as of today