Sound does no longer fade

[bwl21]

I extract this topic from #83 to continue the discussion here.

Request

3. The sound does no longer fade in non loop instruments (it sounds like an organ, even with sound instr. 25 (steel guitar). It seems that there is always a sustain. Is it a result of commit 0fc096d ?

[bwl21]

@moinejf wrote

Back on 3. I could not find more information about fading after looking at the source of fluidsynth. Anyway, I compared playing a steel guitar (25) with swami and in the editor (with both Vivaldi and Palemoon). For me, they sound the same: the sound becomes inaudible after about 6s. Which browser do you use?

[bwl21]

I am using chrome, vivaldi and firefox. It does not work at all in Safari (but this is a safari problem ... see https://github.com/mdn/webaudio-examples/issues/5 )

I tried a bit in edit-1.xhtml

X:1
L:1/4
Q:1/4=120
K:C
CDEFGA

and setting the speed to very low, you can hear that it sounds very long. It sounds the same in Firefox.

from https://www.teachmeaudio.com/recording/sound-reproduction/sound-envelopes/

According to this diagram it seems that sustain is too high, decay is too short. Maybe it can be adjusted somewhere. I am not sure if these properties are set in the soundfont or in the player, or even in sf2_create within toaudio5.html

[moinejf]

Two things:

• safari From what I know, safari does not allow audio output if it is not initiated by the user (click). In the editor, I open the audio output when the user clicks on 'play'. Normally, safari should permit audio. Then, as I cannot test that, if this does not work, could you try to add a dummy playing just after audio open and see if the playing works thereafter? Here is the sequence (to put after "new (window.AudioContext")

  // create an empty buffer
  var buffer = ac.createBuffer(1, 1, 22050);
  var source = ac.createBufferSource();
  source.buffer = buffer;
  // connect to output
  source.connect(ac.destination);
  // play the file
  source.noteOn(0);

• sustain/decay This properties are defined in the SF2.
  For instance, the steel guitar (25 - as many other percussion instruments) has:

  • swami values
  • decay: 19.996s
  • sustain: 100dB
  • sf2-parser.js
  • decay: 5186 (= Math.pow(2,(5186/1200)) = 19.99637616366252s)
  • sustain: 1000 (= 1000/1000 = 100dB)

  So, the sustain volume is (100dB - 100dB) = 0. It cannot be smaller! It is reached 20s after the decay start time. As the decay is linear, half volume would occur at about 14s after decay start time.

  In fact, this is not what should occur: with swami (fluidsynth), I hear no sound after 6s.

  So, I changed the algorithm, dividing the decay time by 10. Then, using the html5 audio function setTargetAtTime(), the volume is 63% after this new decay time. Pratically, with the steel guitar:

  • on (decaytime + 0s), the volume is 1
  • on (decaytiime + 2s), the volume is 0.63
  • on (decaytime + 4s), the volume is 0.4
  • on (decaytime + 6s), the volume is 0.25
  • on (decaytime + 8s), the volume is 0.16
  • ... and that is fine for me: I do not hear any difference between swami and the editor.

[bwl21]

Safari:

To double check, I discarded all my changes - an unbelievable, it still plays on safari.

TL:DR;

I made in toaudio5.js starting with line 390 (

    // play the events
    play: function(istart, i_iend, a_e) {
        if (!a_e || !a_e.length) {
            onend()         // nothing to play
            return
        }

        // initialize the audio subsystem if not done yet
        // (needed for iPhone/iPad/...)
        if (!gain) {
            ac = conf.ac
            if (!ac)
                conf.ac = ac = new (window.AudioContext ||
                            window.webkitAudioContext);

      // create an empty buffer
      var buffer = ac.createBuffer(1, 1, 22050);
      var source = ac.createBufferSource();
      source.buffer = buffer;
// connect to output
      source.connect(ac.destination);
// play the file
      //source.start(0);

            gain = ac.createGain();
            gain.gain.value = gain_val
        }

and it plays the sound on safari !!!

To double check, I discarded all my changes - an unbelievable, it still plays on safari.

[bwl21]

I then tried with Zupfnoter, it also plays on safari Version 11.0.3 (11604.5.6.1.1). Now I am confused, as yesterday it did not work.

nevertheless, let us focus on the other stuff.

[bwl21]

Even if there is not difference betwee swami (which I cannot run on my mac) and abc2svg, the sound is not satisfying.

I will prepare an audio/video recording such that you can hear how it sounds on my machine. then we see if you hear the same. I have users complaining about the playback with the old soundfont stuff.

As I am teaching a Zufpnoter workshop tomorrow, It will take til tomorrow night.

[moinejf]

I don't follow all of what you are saying.

• safari: does it play with or without the dummy play? (source.start(0) is commented in your patch)
• decay: you did not tell me which browser(s) you are using (and which version). Do all your browsers have the same bad decay?

[moinejf]

For an example, this one is enough:

X:1
M:C
L:1/4
K:C
%%MIDI program 25
c4-|c4-|c4-|c4-|

[bwl21]

Safari (now) plays with as well as without the dummy play! I guess it is the other soundfont format which made it work again.

I am using:

chrome 65.0.3325.181 (Offizieller Build) (64-Bit) firefox 59.0.1 (64-Bit) safari Version 11.0.3 (11604.5.6.1.1)

All three sound the same. And sound to have the bad decay. But let me create the video tomorrow to see if we hear the same.

[moinejf]

About safari, I think that the fix was the commit 8afe984 (don't use window.atob()).

[moinejf]

I changed a bit the envelope, removing the call to the function setTargetAtTime(). Otherwise, I checked all what is possible with the web audio API, and I cannot do better. If you have some other ideas (not, not setValueCurveAtTime!). Here is the patch:

diff --git a/util/toaudio5.js b/util/toaudio5.js
index c129e03..835cd26 100644
--- a/util/toaudio5.js
+++ b/util/toaudio5.js
@@ -179,7 +179,7 @@ function Audio5(i_conf) {
                hold: Math.pow(2, (gen.holdVolEnv ?
                    gen.holdVolEnv.amount : -12000) / 1200),
                decay: Math.pow(2, (gen.decayVolEnv ?
-                   gen.decayVolEnv.amount : -12000) / 1200) / 10,
+                   gen.decayVolEnv.amount : -12000) / 1200) / 3,
                sustain: gen.sustainVolEnv ?
                    (gen.sustainVolEnv.amount / 1000) : 0,
 //             release: Math.pow(2, (gen.releaseVolEnv ?
@@ -191,6 +191,12 @@ function Audio5(i_conf) {
            parm.hold += parm.attack;
            parm.decay += parm.hold;

+           // sustain > 40dB is not audible
+           if (parm.sustain >= .4)
+               parm.sustain = 0.01 // must not be null
+           else
+               parm.sustain = 1 - parm.sustain / .4
+
            sample_cp(parm.buffer, sample)

            if (gen.sampleModes && (gen.sampleModes.amount & 1)) {
@@ -272,17 +278,21 @@ function Audio5(i_conf) {
 //     o.playbackRate.setValueAtTime(parm.rate, ac.currentTime);
        o.playbackRate.value = rates[instr][key];

-       var vol = .5;
        g = ac.createGain();
-       g.gain.setValueAtTime(0, t);
-       g.gain.linearRampToValueAtTime(vol, t + parm.attack);
-       g.gain.setTargetAtTime((1 - parm.sustain) * vol,
-                   t + parm.hold, parm.decay);
+       if (parm.hold < 0.002) {
+           g.gain.setValueAtTime(1, t)
+       } else {
+           if (parm.attack < 0.002) {
+               g.gain.setValueAtTime(1, t)
+           } else {
+               g.gain.setValueAtTime(0, t);
+               g.gain.linearRampToValueAtTime(1, t + parm.attack)
+           }
+           g.gain.setValueAtTime(1, t + parm.hold)
+       }

-//fixme: does not work
-//     g.gain.setValueAtTime((1 - parm.sustain) * vol, t + d);
-//     g.gain.linearRampToValueAtTime(0,
-//                 t + d);
+       g.gain.exponentialRampToValueAtTime(parm.sustain,
+                   t + parm.decay);

        o.connect(g);
        g.connect(gain);
@@ -462,6 +472,6 @@ function Audio5(i_conf) {
        else
            gain_val = v;
        set_cookie("volume", v.toFixed(2))
-   }, // set_vol()
+   } // set_vol()
     }
 } // end Audio5

[bwl21]

I tried to make the promised video today, but the quality of sound recording is not such that one can hear the difference well. So I made an audio recording in the attached file.

I thas

1. The testcase with piano on 1.16 - your latest commit
2. The testcase with piano on 1.15
3. Real piece (multiple voices) on 1.16 with piano
4. Real piece (multiple voices) on 1.16 with acoustic guitar steel strings
5. Real piece (multiple voices) on 1.15 with piano
6. Real piece (multiple voices) on 1.15 with acoustic guitar steel strings

you can hear that

1. 1.15 has lower volume and the sample is much shorter
2. the "organ noise" in 1.16 is much more, in particular on the acoustic guitar with steel strings

I had a talk with a friend about the issue. He assumes that the problem is in the samples. In order to save memory, sometimes only the attack is sampled, but the sustain repeats the same sample.

However it is, I see that e.g. 0.js is much smaller (256 k) than acoustic_grand_piano-ogg.js (1.5MB). Maybe this is the reason for the difference.

How do you produce the sample files? I see they hold an abc2svg comment in it. Could we try with samples of higher quality?

abc2svg_sounds.zip

[moinejf]

It seems that you don't know about SF2. First, about hearing.

1. 1.16 has a higher volume because I removed the 'vol' constant (.5).
   The sample is exactly the same.
2. The noise is greater because the volume is greater. SF2 has a filter mechanism, but I did not implement it yet.

About SF2.

• the duration of the samples depends on the soundfont quality. For instance, the acoustic grand piano in the Scc1t2 soundfont has a sample duration about 1/4s. I don't know the value for the Fluid R3 soundfont which was used previously (the Fluid R3 is 130Mb, Scc1t2 is only 3.2Mb).
• yes, when the sample is exhausted, a generation loop occurs.
  In the previous version, the loop was activated only for non-percussion instrument. There was no loop for the piano nor for the steel guitar. As the sample duration was 2.5s, the sound was stopping at this date. In the actual version, the loop is defined in the SF2. It may occur for any instrument type. The loop points (start and end) are carefully set. For percussion instruments, the decay (time) and sustain (volume) parameters define how the sound must fade down.
• the previous version was using a single sample for each note without any loop indication. This was implying:
  • all percussion instruments had a sound cut-off at 2.5s,
  • for the other instruments, the loops were created randomly, and this was always generating some noise at the loop time.

About the sizes.

• the previous version was using one sample per note. These sample were generated from the Fluid R3 soundfont with fluidsynth for a fix duration 2.5s and compressed either in MP3 or OGG. The size of each sample was about 12Kb, and 1.4Mb for a whole instrument. This mechanism was working fine when the duration of the notes were smaller than 2.5s. When greater, there was a cut-off or a looping noise.
• the actual version uses a native soundfont, but all the SF2 components have not (yet?) been implemented.
  I tried TimGM6mb (distributed with Timidity - size 5.7Mb), but there were many bugs, and the quality was not so good. So, I moved to Scc1t2, a soundfont adapted from an old Roland synthesizer, which is small enough (3.2Mb) and MIDI GS. As you may see, when extracted and converted to base64, the biggest instruments are the percussion (390Kb) and the acoustic pianos (250Kb). The other instruments have 50Kb as their mean sizes.

About the generation in javascript.
The web audio API offers only a few functions for the envelopes.
While SF2 says the envelopes use linear ramps, in practice, the web audio linear ramps do not render the same as what can be heard from swami (by fluidsynth). Also, with the web browsers I have, I could not hear a real difference between linear and exponential ramps.
So, I told before, I cannot do better.

About other soundfonts. You may easily use your own soundfonts. Here is how:

• get a soundfont as SF2 only. Other formats would not work.
• with a soundfont editor (as swami), extract each instrument you want into a separate soundfont file (always format SF2).
• convert these individual SF2 files to base64 and create the files instrument_number.js with a content:

  abcsf2[instrument_number] = 'base64_encoded_SF2_of_the_instrument'

  You may then put all these xx.js files in any directory, and set the SFU parameter to this one.
  The instrument number is computed in ABC from %%MIDI commands:

  %%MIDI control 0 bh % MSB bank number
  %%MIDI control 0 bl % LSB bank number
  %%MIDI program p
  % instrument_number = (bh * 128 + bl) * 128 + p

[bwl21]

It seems that you don't know about SF2.

Oh yes, this is fully true .. sorry; I tried to find a page with good explanation, but was not successful.

About SF2.

Thanks for the explanation. It helps for my understanding and confirms the assumption of my friend.

About other soundfonts.

swami seems not to be available on MacOs. But I found https://polyphone-soundfonts.com/en/. This site provides a soundfont editor (called polyphone, available on MacOs, Linux, Windows) and also some Soundfonts. So I will play around with this to get some knowledge. Maybe at the end I will manage to create my own soundfont optimized for table harp :-)

About the generation in Javascript ... So, I told before, I cannot do better.

Commit ca0ed08 provides a much better result at least for the piano (0). Thanks a lot. It provides an acceptable playback and I am not stuck to abc2svg 1.15. Instrument 25 still does not sonund well. Instrument 25 does not ramp down fast enough. But as far as I understood for now, you only have linear ramps. So for the time being, I will stop using 25 to emulate the table harp until I have a better understanding of all that stuff.

maybe we keep this ticket open for a while if you don't mind. Thanks a lot for your work and patience.

[bwl21]

Yesterday I have read a lot about soundfonts and all that stuff.

I found these pages very useful:

• https://theproaudiofiles.com/synthesis-101-envelope-parameters-uses/
• https://mdn.github.io/webaudio-examples/audio-param/

I then did some experiments. My conclusion is, that I eventually I have to create my own optimized soundfonts.

TL:DR;

I tried to understand, what you are doing. The essential part is the following line which controls the decay.

        g.gain.exponentialRampToValueAtTime(parm.sustain,
                    t + parm.decay);

Sith the following line, piano sounds much better for me - yes it is a matter of taste. So I do not expect that you set these parameters. I rather think, I need to create an optimized soundfont.

        g.gain.exponentialRampToValueAtTime(parm.sustain,
                    t + parm.decay * 0.7 );

or even a combination which was even a little better

    g.gain.exponentialRampToValueAtTime(parm.sustain,
      t + parm.decay * 0.5);

    g.gain.exponentialRampToValueAtTime(parm.sustain * 0.000001,
      t + parm.decay);

For the steel guitar (25) the following setting sounds best for me:

    g.gain.exponentialRampToValueAtTime(parm.sustain,
      t + parm.decay * 0.2);

    g.gain.exponentialRampToValueAtTime(parm.sustain * 0.000001,
      t + parm.decay * 0.9);

Of course it is not possible to optimize this in the player, as it depends on the instrument. Maybe we could pass some extra parameters to the player, but I think this is the wrong direction. It needs to be handled in the soundfonts.

[moinejf]

For your information, 'decay' is 20s and 'sustain' is 0 (100dB) for both the piano and the steel guitar. Then, your second ramp is of no use.

[bwl21]

Hm, I still don't really know, what I am doing. I tried again and now I also do not find a diffence if the second ramp is there.

Nevertheless, i tried to manipulate the soundfonts today using polyphone https://polyphone-soundfonts.com/en/ . As a pity, the .sf2 - Files created by Polyphone did not work. It appears to me that Polyphone creates additional infos without samples. So I patched toaudio5.js to skip infos without sampleID.

@@ -170,6 +170,8 @@ function Audio5(i_conf) {
        rates[instr] = []
        for (i = 0; i < infos.length; i++) {
            gen = infos[i].generator;
            if (! gen.sampleID)
                continue;
            sid = gen.sampleID.amount;
            sampleRate = parser.sampleHeader[sid].sampleRate;
            sample = parser.sample[sid];

Here I provide the soundfonts as I generated them for Zupfnoter: I have set decay to 8sec and sustain to 120db. The files only work with the mentioned patch. zupfnoter.zip

[bwl21]

TL:DR;

In case other readers are interested: I created it in in following steps:

1. download from https://stash.reaper.fm/v/23360/Scc1t2.sf2
2. open the file in polyphone
3. export instrument piano1 to 0_GS sound set (16 bit).sf2
4. export instrument Steel guitar to 25_GS sound set (16 bit).sf2
5. Open the tow files again in polyphone
6. go to the instrument and set decay to 8sec in all columns of the table, set sustain to 120db
7. save the files
8. convert to js files as specified by @moinejf in https://github.com/moinejf/abc2svg/issues/84#issuecomment-375979049 . I did this using a rake task.

desc "compile zupfnoter soundfonts"
task :buildSoundfonts do
  sources = Dir["../sf2_sources/*.sf2"]
  i=0
  sources.each do |sf|
     outnumber = File.basename(sf).split("_").first
     sfbase64 = Base64.encode64(File.read(sf)).split("\n").join("\\\n")
     File.open("../public/soundfont/zupfnoter/#{outnumber}.js", "w") do |outfile|
       outfile.puts (%Q{//ABC2SVG soundfont prepared for Zupfnoter
//created #{Time.now} from #{sf} by zupfnoter rake
abcsf2[#{outnumber}]='\\
#{sfbase64}'})
     end
  end
end

[moinejf]

In 6., you don't need to set the sustain: in toaudio5, any value greater to 40dB sets the volume level to 0. It would have been interested to do this job with some other soundfont. For instance, the decay time is smaller in TimGM6mb for these two instruments...

[bwl21]

I did it with http://polyphone-soundfonts.com/en/files/10-pianos/305-motif-es6-concert-piano-v2 without touching the parameters. It sound good but is 12 MB :-)

Next step will be to create my own harp samples.

[bwl21]

I recorded some wav sample from my table harp and managed to create an SF2 file. It has 384 kb and sounds well.

I still need the change from https://github.com/moinejf/abc2svg/issues/84#issuecomment-376485119 to use it with abc2svg. Could you please add this patch.

[moinejf]

May you send me this file for I have a look inside?

[bwl21]

you can find testfiles in the https://github.com/moinejf/abc2svg/files/1851584/zupfnoter.zip

[moinejf]

OK, seen, the first generator is empty. I will apply your patch. BTW, I wonder why you dit not use the harp (GM 46).

[bwl21]

Simply since I did not know. And the sound of a vertical harp is different from a table harp. I am excitedly playing around with with recording and creating soundfonts. I recorded exactly one string of a table harp with good results. I now play around (just for fun and learning) with recoring my own voice and create a soundfont from that :-)

Thanks for applying the patch.

[bwl21]

I think, we can close this. As I am now able to manipulate the soundfonts, I can optimize the playback there. Thanks for the helpful discussion and fixes.