more of a question - Githubissues

UNIcodeX commented 4 years ago

I was trying to adapt the sine wave example to play the Mario tune, but my timing is all off...

Would you mind taking a look and advising me on where I've gone wrong?

The following code is partially adapted from https://www.princetronics.com/supermariothemesong/

import os
import sequtils
import strformat

import rapid/audio/samplers/osc
import rapid/audio/device

const
  NOTE_B0   = 31.float
  NOTE_C1   = 33.float
  NOTE_CS1  = 35.float
  NOTE_D1   = 37.float
  NOTE_DS1  = 39.float
  NOTE_E1   = 41.float
  NOTE_F1   = 44.float
  NOTE_FS1  = 46.float
  NOTE_G1   = 49.float
  NOTE_GS1  = 52.float
  NOTE_A1   = 55.float
  NOTE_AS1  = 58.float
  NOTE_B1   = 62.float
  NOTE_C2   = 65.float
  NOTE_CS2  = 69.float
  NOTE_D2   = 73.float
  NOTE_DS2  = 78.float
  NOTE_E2   = 82.float
  NOTE_F2   = 87.float
  NOTE_FS2  = 93.float
  NOTE_G2   = 98.float
  NOTE_GS2  = 104.float
  NOTE_A2   = 110.float
  NOTE_AS2  = 117.float
  NOTE_B2   = 123.float
  NOTE_C3   = 131.float
  NOTE_CS3  = 139.float
  NOTE_D3   = 147.float
  NOTE_DS3  = 156.float
  NOTE_E3   = 165.float
  NOTE_F3   = 175.float
  NOTE_FS3  = 185.float
  NOTE_G3   = 196.float
  NOTE_GS3  = 208.float
  NOTE_A3   = 220.float
  NOTE_AS3  = 233.float
  NOTE_B3   = 247.float
  NOTE_C4   = 262.float
  NOTE_CS4  = 277.float
  NOTE_D4   = 294.float
  NOTE_DS4  = 311.float
  NOTE_E4   = 330.float
  NOTE_F4   = 349.float
  NOTE_FS4  = 370.float
  NOTE_G4   = 392.float
  NOTE_GS4  = 415.float
  NOTE_A4   = 440.float
  NOTE_AS4  = 466.float
  NOTE_B4   = 494.float
  NOTE_C5   = 523.float
  NOTE_CS5  = 554.float
  NOTE_D5   = 587.float
  NOTE_DS5  = 622.float
  NOTE_E5   = 659.float
  NOTE_F5   = 698.float
  NOTE_FS5  = 740.float
  NOTE_G5   = 784.float
  NOTE_GS5  = 831.float
  NOTE_A5   = 880.float
  NOTE_AS5  = 932.float
  NOTE_B5   = 988.float
  NOTE_C6   = 1047.float
  NOTE_CS6  = 1109.float
  NOTE_D6   = 1175.float
  NOTE_DS6  = 1245.float
  NOTE_E6   = 1319.float
  NOTE_F6   = 1397.float
  NOTE_FS6  = 1480.float
  NOTE_G6   = 1568.float
  NOTE_GS6  = 1661.float
  NOTE_A6   = 1760.float
  NOTE_AS6  = 1865.float
  NOTE_B6   = 1976.float
  NOTE_C7   = 2093.float
  NOTE_CS7  = 2217.float
  NOTE_D7   = 2349.float
  NOTE_DS7  = 2489.float
  NOTE_E7   = 2637.float
  NOTE_F7   = 2794.float
  NOTE_FS7  = 2960.float
  NOTE_G7   = 3136.float
  NOTE_GS7  = 3322.float
  NOTE_A7   = 3520.float
  NOTE_AS7  = 3729.float
  NOTE_B7   = 3951.float
  NOTE_C8   = 4186.float
  NOTE_CS8  = 4435.float
  NOTE_D8   = 4699.float
  NOTE_DS8  = 4978.float

  melody = @[
    NOTE_E7, NOTE_E7, 0, NOTE_E7, 0, NOTE_C7, NOTE_E7, 0, NOTE_G7, 0, 0,  0, NOTE_G6, 0, 0, 0, NOTE_C7, 0, 0, NOTE_G6,
    0, 0, NOTE_E6, 0, 0, NOTE_A6, 0, NOTE_B6, 0, NOTE_AS6, NOTE_A6, 0, NOTE_G6, NOTE_E7, NOTE_G7, NOTE_A7, 0, NOTE_F7, NOTE_G7,
    0, NOTE_E7, 0, NOTE_C7, NOTE_D7, NOTE_B6, 0, 0, NOTE_C7, 0, 0, NOTE_G6, 0, 0, NOTE_E6, 0, 0, NOTE_A6, 0, NOTE_B6,
    0, NOTE_AS6, NOTE_A6, 0, NOTE_G6, NOTE_E7, NOTE_G7, NOTE_A7, 0, NOTE_F7, NOTE_G7, 0, NOTE_E7, 0, NOTE_C7, NOTE_D7, NOTE_B6, 0, 0
    ]

  tempo = @[
    12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
    9, 9, 9, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
    9, 9, 9, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
  ]

var
  dev = newRAudioDevice()
  oscl = newROsc(oscSine)
dev.attach(oscl)
dev.start()

for (n, t) in zip(melody, tempo):
  var
    noteDuration = 1000 / t
    # delayValue = int((1_000_000 / n) / 8)
    # delayBetween = int(noteDuration * 1.3)
    numCycles    = int(n * noteDuration / 1000)
  echo(fmt"{noteDuration}  {numCycles}")

  oscl.play(n)
  sleep(numCycles)

  oscl.play(0)
  # sleep(delayBetween)

# sleep(1000)

oscl.stop()

liquidev commented 4 years ago

Sorry, I'm not able to test this right now. The latest version of Nimterop introduced an ABI break, and causes rapid/audio to segfault (because field order matters! and Nim does not seem to order fields the same way C does, or GCC does some annoying reordering). I'm contacting the author of the library right now to see if they can do something about this. It probably works on your machine because you're using an older version of Nimterop, so don't update. Meanwhile, to push the issue a bit further, can you send me a sample of how it sounds and how it should sound?

UNIcodeX commented 4 years ago

Rapid-Attempt_at_Mario_Bros_Tune.zip

It should sound like the main Mario Bros theme song. --> https://youtu.be/nYwM_OFiFDY?t=21

These are the libraries and versions I have installed:

ajax  [0.1.0]
c2nim  [0.9.14]
cligen  [0.9.45]
gifenc  [0.1.1]
glm  [1.1.1]
gr  [0.1.0]
hmac  [0.1.9]
html5_canvas  [1.3]
nico  [0.2.0, 0.2.1]
nimaws  [0.3.2]
nimPNG  [0.2.6]
nimpy  [0.1.0]
nimSHA2  [0.1.1]
nimterop  [0.4.4]
pylib  [0.1.0]
rapid  [0.1.0]
regex  [0.14.1]
sdl2  [2.0.2]
sdl2_nim  [2.0.12.0]
segmentation  [0.1.0]
sha1  [1.1]
sndfile  [0.1.0]
unicodedb  [0.9.0]
unicodeplus  [0.6.0]
webaudio  [0.2.0]

liquidev commented 4 years ago

Just so you know, oscl.play(0) does not stop playback. It sets the frequency to 0 which probably causes some weird math related issues during sound generation, so don't actually do this :P Use oscl.stop() instead. In the for loop, the sleep duration is determined using this formula: n * noteDuration / 1000. However, looking at the for loop definition, I can see that n is actually the note you want to play, and not the duration you want to play it for! You probably meant to use t here. But anyways, here's an improved version of your code. You can adjust how long the note plays by altering the NoteLength const.

import os
import sequtils

import rapid/audio/samplers/[mixer, osc]
import rapid/audio/device

# NOTE_* definitions omitted

const
  melody = @[
    NOTE_E7, NOTE_E7, 0, NOTE_E7, 0, NOTE_C7, NOTE_E7, 0, NOTE_G7, 0, 0,  0, NOTE_G6, 0, 0, 0, NOTE_C7, 0, 0, NOTE_G6,
    0, 0, NOTE_E6, 0, 0, NOTE_A6, 0, NOTE_B6, 0, NOTE_AS6, NOTE_A6, 0, NOTE_G6, NOTE_E7, NOTE_G7, NOTE_A7, 0, NOTE_F7, NOTE_G7,
    0, NOTE_E7, 0, NOTE_C7, NOTE_D7, NOTE_B6, 0, 0, NOTE_C7, 0, 0, NOTE_G6, 0, 0, NOTE_E6, 0, 0, NOTE_A6, 0, NOTE_B6,
    0, NOTE_AS6, NOTE_A6, 0, NOTE_G6, NOTE_E7, NOTE_G7, NOTE_A7, 0, NOTE_F7, NOTE_G7, 0, NOTE_E7, 0, NOTE_C7, NOTE_D7, NOTE_B6, 0, 0
  ]

  lengths = @[
    12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
    15, 15, 15, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
    15, 15, 15, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
  ]

var
  dev = newRAudioDevice()
  oscl = newROsc(oscSine)
  mix = newRMixer()
  osclTrack = mix.add(oscl)

# I altered the volume here to not blow my ears off. the default sine oscillator is LOUD
osclTrack.volume = 0.2

dev.attach(mix)
dev.start()

# we sleep for 500 millis to prevent the oscillator from playing before the device starts
sleep(500)

const NoteLength = 0.8

for (note, time) in zip(melody, lengths):
  var
    duration = float(time * 10)
    onTime = duration * NoteLength
    offTime = duration - onTime
  echo (note: note, time: time)

  if note != 0:
    oscl.play(note)
    sleep(onTime.int)

    oscl.stop()
    sleep(offTime.int)
  else:
    # prevent frequency 0, because god knows what happens when you use that
    sleep(duration.int)

One piece of advice I can give to you is to not use cryptic names like (n, t) when what the variable stores is clear. It will save you a lot of headaches!

UNIcodeX commented 4 years ago

Thank you for your advice. I am still having trouble getting it to play right. I'm on a windows machine.

I've tried changing the timing to this:

while true:
  for (note, time) in zip(melody, lengths):
    var
      timing = time / 12
      onTime = int((2000 / timing) / 10)
      offTime = int(onTime * 1.3)
    echo (note: note, time: time, onTime: onTime, offTime: offTime)

    if note != 0:
      oscl.play(note)
      sleep(onTime)

      oscl.stop()
      sleep(offTime)
    else:
      # prevent frequency 0, because god knows what happens when you use that
      oscl.stop()
      sleep(offTime)

The notes seem to play in the right order, but if I try to tighten it up, then it seems like maybe the oscilator doesn't have a chance to actually initialize or change it's tone in time before the next instruction comes in, and it only plays a few random notes.

I don't know what I'm doing wrong. I guess I'm just not using it right...

liquidev commented 4 years ago

The problem is that sleep can only handle precision up to milliseconds, so some values get rounded down to 0, so basically some notes are played and then immediately stopped. Nim would need a more precise sleep proc that can handle nanoseconds, as far as I know there's sleepAsync, but I haven't tested if it does actually handle nanoseconds or not. Looking at the source code, I'd say it does, so try replacing the sleep(time)s with waitFor sleepAsync(time), and remove all the rounding to ints.

UNIcodeX commented 4 years ago

hmm. I'll have to give that a try. I was just trying to get this working as something fun for my kids as well as to understand how to generate music for a game using tones, rather than pre-recorded music files.

On Tue, May 5, 2020 at 11:07 AM lqdev notifications@github.com wrote:

The problem is that sleep can only handle precision up to milliseconds, so some values get rounded down to 0, so basically some notes are played and then immediately stopped. Nim would need a more precise sleep proc that can handle nanoseconds, as far as I know there's sleepAsync, but I haven't tested if it does actually handle nanoseconds or not. Looking at the source code, I'd say it does, so try replacing the sleep(time)s with waitFor sleepAsync(time), and remove all the rounding to ints.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/liquid600pgm/rapid/issues/20#issuecomment-624147106, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB62P6TUTQIKSVOBEK7MGTLRQA2VBANCNFSM4MZERDOQ .

liquidev commented 4 years ago

That's cool! Maybe you can implement this as a precise sampler, here's the gist: Samplers are actually what generate or process audio, so every ROsc, RWave, RMixer is a sampler. This is the usual pattern for implementing a custom sampler:

import rapid/audio/sampler

type
  SongPlayer* = ref object of RSampler

method sample*(player: SongPlayer, dest: var SampleBuffer, count: int) =
  # this is a barebones sampler that outputs silence
  dest.add(0.0, count * 2)

proc initSongPlayer*(player: SongPlayer) =
  discard  # init logic goes here

proc newSongPlayer*(): SongPlayer =
  result = SongPlayer()
  result.initSongPlayer()

In the sample method, your custom sampler must output samples for left and right channels interleaved (so the order should be [L, R, L, R, L, R, …]). The number of samples you output must always be count * 2. A sampler can also call back to other samplers, so for example you may store a voice: ROsc in your SongPlayer and call player.voice.sample(dest, count) to defer the actual sample generation to the oscillator. You can also call player.voice.sample(dest, 1) to generate only one audio sample, and then use it in a loop. For example:

type
  SongPlayer* = ref object of RSampler
    voice: ROsc
    time: uint

method sample*(player: SongPlayer, dest: var SampleBuffer, count: int) =
  for _ in 0..<count:
    player.voice.sample(dest, 1)
    inc(player.time)

Then you can store your song in a different format, like this:

import std/tables

type
  NoteCommand = object
    on: bool
    freq: float
  Song = Table[uint, NoteCommand]

Then alter the song player like this:

type
  SongPlayer* = ref object of RSampler
    song: Song
    voice: ROsc
    time: uint

method sample*(player: SongPlayer, dest: var SampleBuffer, count: int) =
  for _ in 0..<count:
    if player.time in player.song:
      let note = player.song[player.time]
      if note.on:
        player.voice.play(note.freq)
      else:
        player.voice.stop()
    player.voice.sample(dest, 1)
    inc(player.time)

You'd then have to store the notes as a sequence of on|off commands, maybe even create a file format for that. Or maybe even a nice editor :) the sky is the limit. But this way your note playback would be precise to the single audio sample. Here are some things to keep in mind:

The sample callback must be really fast. Otherwise you may get stuttering audio due to underflows. You can make it go faster by wrapping it in a {.push stacktrace: off.} pragma, but then you lose on debugging capabilities.
Don't do system calls like filesystem access inside the sample method. They're very slow. Allocating memory on the heap should be fine as the Nim GC simply reuses any free memory it has, but prefer stack allocation wherever possible.
The sample method runs on a separate thread, so be cautious while accessing global state. Preferably store everything inside of the sampler object itself to avoid headaches.

At this point I'm starting to think I should turn this into a tutorial. Anyways, happy coding!

UNIcodeX commented 4 years ago

Wow! Thank you for the very thorough write up. I'll have to go over it at least one more time to fully digest it. XD

liquidev / rapid

more of a question #20