search

microbit-foundation / micropython-microbit-v2

Temporary home for MicroPython for micro:bit v2 as we stablise it before pushing upstream
MIT License
41 stars 22 forks source link

WIP: Audio recording and playback #163

Open dpgeorge opened 7 months ago

dpgeorge commented 7 months ago

This PR adds audio/microphone recording and playback capabilities, as per the docs proposal: https://github.com/bbcmicrobit/micropython/pull/791

It currently supports:

There is a test program called src/test_record.py which uses all the above to show how it works.

dpgeorge commented 7 months ago

@microbit-carlos Please try out the test program and see how it goes.

dpgeorge commented 5 months ago

I have updated this PR with the following additions/changes:

dpgeorge commented 5 months ago

Some tweaks have been made to how the "used_size" entry in AudioFrame is used/set, the default microphone gain set to 0.2, and this PR has been rebased on the latest master which includes an update to the latest MicroPython version.

@microbit-carlos this is now ready for wider testing.

microbit-carlos commented 4 months ago

Thanks Damien! Btw, the next CODAL tag has been released yesterday, which should resolve some of the issues with changing the sampling rate to non-proportional values: https://github.com/lancaster-university/codal-microbit-v2/releases/tag/v0.2.65

Diff:

dpgeorge commented 4 months ago

@microbit-carlos I have now updated this PR to use CODAL v0.2.66. It's working well.

microbit-carlos commented 4 months ago

With this example the playback rate doesn't seem to change for playback.

from microbit import *

RECORDING_SAMPLING_RATE = 7812

while True:
    if pin_logo.is_touched():
        # Record and play back at the same rate
        my_recording = microphone.record(duration=3000, rate=RECORDING_SAMPLING_RATE)
        audio.play(my_recording)
        del my_recording

    if button_a.is_pressed():
        # Play back at half the sampling rate
        my_recording = microphone.record(duration=3000, rate=RECORDING_SAMPLING_RATE)
        audio.set_rate(RECORDING_SAMPLING_RATE // 2)
        audio.play(my_recording)
        del my_recording

    if button_b.is_pressed():
        # Play back at twice the sampling rate
        my_recording = microphone.record(duration=3000, rate=RECORDING_SAMPLING_RATE)
        audio.set_rate(RECORDING_SAMPLING_RATE * 2)
        audio.play(my_recording)
        del my_recording

    sleep(200)

Edit: Ah! the type was that this example was using audio.set_rate(), which I guess we need to remove from this branch?

microbit-carlos commented 4 months ago

Also, I think the microphone.set_sensitivity(gain) still needs to be implemented.

microbit-carlos commented 4 months ago

Another small one, if the AudioFrame rate is set to a negative number it overflows the calculation (well, maybe it's converting it to an unsigned int at some point):

>>> AudioFrame(1000, -1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError: memory allocation failed, allocating 4294990 bytes

It could throw a ValueError instead, like setting it to zero:

>>> AudioFrame(1000, 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: rate out of bounds
microbit-carlos commented 4 months ago

Looks like set_rate() overflows/converts as well:

>>> af = AudioFrame()
>>> af.set_rate(2**16 - 1)
>>> af.get_rate()
65535
>>> af.set_rate(2**16)
>>> af.get_rate()
0
>>> af.set_rate(2**16 + 1)
>>> af.get_rate()
1

And on that note, should the rate be limited to 16 bits? Should it be increased to 32?

microbit-carlos commented 4 months ago

Should copyfrom() also duplicate the length from the source? Before these changes AudioFrame was always 32 bytes, but now we can have different size AudioFrames.

I was also wondering if we should also make it a static function to be able to do new = AudioFrame.copyfrom(old). Otherwise we now we have to figure out a way to match the length of the old instance somehow when creating the new, as we cannot even do something like new = AudioFrame(size=len(old)); new.copyfrom(old).

microbit-carlos commented 4 months ago

Does the AudioFrame returned by microphone.record() purposely does not have a size that is a multiple of 32 bytes, or is len(AudioFrame()) returning the "used size" instead of the "total size"?

>>> len(microphone.record(duration=1000))
7812
>>> 7812 / 32
244.125
microbit-carlos commented 4 months ago

Passing a negative value to the microphone.record() duration or rate parameters seems to lock up and never return. So microphone.record(-1) or microphone.record(1000, -1)

And setting the rate to zero returns a zero-length AudioFrame:

>>> len(microphone.record(1000, 0))
0
dpgeorge commented 3 months ago

Ah! the type was that this example was using audio.set_rate(), which I guess we need to remove from this branch?

I have now removed audio.set_rate().

dpgeorge commented 3 months ago

I think the microphone.set_sensitivity(gain) still needs to be implemented.

Yes, correct, it still needs to be implemented.

Would this just call CODAL's uBit.audio.processor->setGain()? If so, what is the argument to set_sensitivity(), is it a float or integer, and is the value inverted before passing through to setGain()? And what values should we use for the SENSITIVITY_xxx constants?

dpgeorge commented 3 months ago

Another small one, if the AudioFrame rate is set to a negative number it overflows the calculation

This was a mistake, it was supposed to be checking for negative rates. Now fixed.

Looks like set_rate() overflows/converts as well:

I also fixed this, and increased the internal rate variable to 32-bits (I thought 16-bit would be enough for the rate, but it doesn't cost anything to make it 32-bits).

microbit-carlos commented 3 months ago

Would this just call CODAL's uBit.audio.processor->setGain()? If so, what is the argument to set_sensitivity(), is it a float or integer, and is the value inverted before passing through to setGain()? And what values should we use for the SENSITIVITY_xxx constants?

Yes, only need to call the uBit.audio.processor->setGain() and the values are floats and we can use the same three levels as MakeCode:

microbit-matt-hillsdon commented 3 months ago

The existing sim implementation of playing AudioBuffers has always been problematic because of the 4ms chunk size. For audio frames of longer duration I had more hope, but at the moment the chunk size used is the same even though we likely have a much larger buffer in the frame itself.

Our work-in-progress record/playback sim implementation works OK if you increase LOG_AUDIO_CHUNK_SIZE from 5 to 6. (it's possible that on slower computers a bigger buffer still might help, I've not had a chance to test yet). We can't just do that as I think it means regular audio frames are pitch/rate shifted but it might show a way forward for the sim.

The sim changes without the LOG_AUDIO_CHUNK_SIZE change can be seen here: https://review-python-simulator.usermbit.org/beta-updates/demo.html (https://github.com/microbit-foundation/micropython-microbit-v2-simulator/pull/113) - see sample "Record". For now you'll have to rebuild locally to see the benefit of the buffer size change. We've not yet looked at edge cases etc. but I think this problem will remain.

microbit-carlos commented 2 months ago

@dpgeorge We've been testing the latest version from this PR and came up with a few discussion topics and bug reports. As there is a few of them, I've created individual GH issues to be able to track each of them individually, and all of them are in the v2.2.0-beta.1 milestone.

Some of the issues are bug reports or fairly straightforward questions, but the following issues require further discussion:

@jaustin @microbit-matt-hillsdon your input on these would also be very welcomed.

dpgeorge commented 2 months ago

The following changes have been made to this branch/PR:

dpgeorge commented 2 months ago

Should copyfrom() also duplicate the length from the source?

This needs discussion. copyfrom() actually allows copying from any object with the buffer protocol (eg bytes, memoryview) and they won't necessarily have the used_size attribute.

This method should probably just set used_size to the total bytes copied fro the buffer-like object. The only issue there is copying from another AudioFrame it will copy the total allocated size, not the used_size, of the other AudioFrame.

I was also wondering if we should also make it a static function to be able to do new = AudioFrame.copyfrom(old). Otherwise we now we have to figure out a way to match the length of the old instance somehow when creating the new, as we cannot even do something like new = AudioFrame(size=len(old)); new.copyfrom(old).

How about allowing passing an AudioFrame to the AudioFrame constructor? This is how list, dict etc work in Python. (Did we already discuss point?)

microbit-carlos commented 2 months ago

Should copyfrom() also duplicate the length from the source?

This needs discussion. copyfrom() actually allows copying from any object with the buffer protocol (eg bytes, memoryview) and they won't necessarily have the used_size attribute.

This method should probably just set used_size to the total bytes copied fro the buffer-like object. The only issue there is copying from another AudioFrame it will copy the total allocated size, not the used_size, of the other AudioFrame.

This conversation has already been moved to:

I was also wondering if we should also make it a static function to be able to do new = AudioFrame.copyfrom(old). Otherwise we now we have to figure out a way to match the length of the old instance somehow when creating the new, as we cannot even do something like new = AudioFrame(size=len(old)); new.copyfrom(old).

How about allowing passing an AudioFrame to the AudioFrame constructor? This is how list, dict etc work in Python. (Did we already discuss point?)

We can move this one to:

zazer0 commented 1 day ago

Hi, is this currently usable? Keen to use my microbit v2 as microphone in + audio out for an art project if possible :)

dpgeorge commented 2 hours ago

Hi, is this currently usable?

Yes this branch is usable. We are currently fine-tuning the Python-level API (function names and behaviour) so things will change, but the overall functionality will remain.