Will this repos be commited to SDK, or any licenses?

[meganetaaan]

Hello!

I'd like to use your soundstream functions to my Stack-chan project. (Yes, he will finally have the lipsync feature with his speech!) Will the function be commited into ModdableSDK? If not, does this repository have any OSS licenses?

[phoddie]

Thanks for the reminder.

This began as a quick example. I am happy it is useful. There is some work remaining to make it generally useful:

1. WAVE header needs to be skipped (or there could be some glitches at the start... though I didn't noticed this in my testing)
2. Last audio fragment is dropped (again, maybe not a huge issue if that block happens to contain silence but could be noticeable depending on the audio content and network buffer behavior)
3. Network underflow -- what happens if network delivers data slower than realtime?
4. Organize into a class for re-use

(3) is the most difficult, I think.

I would be happy to merge it into the Moddable SDK as an example once they are resolved. Maybe you have tackled some of those or have some ideas?

[phoddie]

Small update. I've done the following:

• Updated project to work with latest ECMA-419 HTTPClient (small changes)
• Parse and skip WAVE header
• Validate WAVE matches expected playback format
• Queue last block (uses sample count from WAVE header)
• Simple technique to detect and record from underflow stalls

These are changes are uncommitted, as they depend on an update to the Moddable SDK (WAVE parsing uses WavReader which was in wav2maud tool but now is in a stand-alone module). That should be available Friday.

I think my next step is to wrap this in a re-usable class.

[phoddie]

I've committed an update with the changes above all wrapped in a WavStream class. There's also some documentation.

The update depends on the next Moddable SDK update to run.

Once we have a little more experience using this, I'll migrate it over to the Moddable SDK repository.

[meganetaaan]

Sorry for my late response. And thank you so much for your implementation! The specification including the behavior on the network stalls is very reasonable and straightforward for me. I'm going to make time to try it by this weekend.

[phoddie]

Great, thanks. Please let me know how it goes. Just FYI, the supporting Moddable SDK changes landed yesterday.

[meganetaaan]

Tested and worked!

https://twitter.com/meganetaaan/status/1595790350106648576

I wrapped the example code into an async function as below. It worked without a problem. I implemented a WavStreamer instance to be closed and recreated on each API access. Is this the right direction?

import AudioOut from "pins/audioout"
import WavStreamer from "wavstreamer";

function calculatePower(samplea) @ "xs_calculatePower";

const audio = new AudioOut({});
let streamer

async function stream() {
    return new Promise((resolve, reject) => {
        if (streamer != null) {
            reject(new Error("already playing"));
            return
        }
        streamer = new WavStreamer({
        http: device.network.http,
        host: '192.168.7.112',
        port: 8080,
        path: "/googletts_16bit_mono_11025hz.wav",
        audio: {
            out: audio,
            stream: 0,
            sampleRate: 11025
        },
        onPlayed(buffer) {
            const power = calculatePower(buffer);
            trace("power " + Math.round(power) + "\n");
        },
        onReady(state) {
            trace(`Ready: ${state}\n`);
            if (state)
                audio.start();
            else
                audio.stop();
        },
        onError(e) {
            trace("ERROR: ", e, "\n");
            streamer = null
            reject(new Error("unknown error occured"))
        },
        onDone() {
            trace("DONE\n");
            streamer.close()
            streamer = null
            resolve()
        }
    });
})
}

(async function main() {
    let count = 0
    while (true){
        trace(`play: ${count++}\n`)
        let willPlay = stream();
        let willFail = stream().catch(e => {trace(`error: ${e.message}\n`)}) // error: already playing
        await Promise.allSettled([willPlay, willFail])
    }
})()

[phoddie]

Thanks for trying it out! Nice that it works.

[meganetaaan]

FWIW I added ResourceStreamer class modifying the resource-stream example to have the same interface as WavStreamer. The motivation is that I want to implement lipsync feature using onPlayed callback when playing offline.

[phoddie]

That's a nice idea. Do you want to do a Moddable SDK PR for that? I put the http streaming example at $MODDABLE/examples/pins/audioout/http-stream for our December release. The ResourceStreamer class could go beside it.

BTW – For audio in flash, you should really look into our SBC audio support. It is much smaller with very good quality, almost no RAM use, and trivial CPU load. I suppose the challenge here is that you want the power level, which cannot be calculated from the compressed samples. You could fire up an instance of the Mixer to do that, but it would be some more work.

[meganetaaan]

Do you want to do a Moddable SDK PR for that?

I would love to! Do I wait for the next release?

For audio in flash, you should really look into our SBC audio support.

Alright. I'll try look at SBC audio support after pushing my current source.

[phoddie]

Any time is fine. No need to wait.

If I can help with the SBC decode, please let me know.