Make time stretching optional for AudioClips

chrisbc commented 1 year ago

Problem statement

Recently I wanted to experiment with recorded audio, and some random, not-so-beat-centric musical ideas. Currently deluge is very eager to do the stretching and make everything fit the grid neatly. I searched the forums for answers/workarounds (see forum links below) but finally, after recording clips in deluge I had to export them and use SooperLooper to achieve what I wanted.

related Synthstrom Forum threads:

Outline

Here we'll try to switch off time stretching to see how useful this option is. Basically its an excuse to get my hands dirty with the newly opensource Deluge Firmware.

Done criteria (OG)

[x] https://github.com/mupaduw/DelugeFirmware/issues/2 Does it work? Does it sound good? what are impacts? Try recording, overdubbing, importing audiofiles.
[x] a way option to toggle this behaviour on/off without reloading firmware.

Done criteria (from feedback)

~https://github.com/mupaduw/DelugeFirmware/issues/4~ WONTFIX
[x] https://github.com/mupaduw/DelugeFirmware/issues/6
[x] https://github.com/mupaduw/DelugeFirmware/issues/7
[x] persist settings to song XML file - with backwards compatability.
~setting to determine end-of-loop syncing behaviour~ deferred

ok-reza commented 1 year ago

How is disabling time-stretching in audio clips advantageous to just loading the sound into a kit? Would be curious how many use cases there are. The following are the ones I could think of and they revolve around clarity in Arranger view.

Audio clip length will be exactly as long as the sample. To do this in a Kit clip, one would have to set the sample to 'CUT' and place it on the first pad of the sequence. Then they would have to zoom in to the end of the sample's tail and bring the end of the clip length's as close as possible to that point.
When adjusting Audio Clip's pitch, the clip length will dynamically change length.

Benefit: Dragging a time-stretching disabled Audio Clip into Arranger will visually allow you to know exactly how long the sample plays.

Glad to hear about the progress!

chrisbc commented 1 year ago

@ok-reza thanks for the feedback. I don't quite grok the kit workflow - it sounds doable but I find too many steps can stifle the music. However, I will try it out and compare that with what I have got here.

FWIW, I'm finding that working with unstretched audio tracks in song & arranger views is both easy and fun. I found some nice tempo scaling effects are now possible between gridded material and the unstretched audio-clips, as changing the machine tempo has no effect on the unstretched loop(s).

A few notes on my use case(s)

I'm not doing pitch shifting here, these are human played-in parts (eg hydrasynth, or mic) esp where I want a loose, dreamy, non grid-locked quality.
I want to quickly try different combinations of these sounds and look for odd interactions, chuck out disasters etc
I may still transfer these tracks (via SD for now) to an external looper and/or DAW for final assembly, but I want to stay in deluge-land as much as possible
if there's interest, I might post a short video explaining how I'm using it.

Implementation notes

this is currently a global on/off switch, it's not clip specific - it's applied to all playing AudioClip instances. That could be handled I'm sure, but I just don't need it yet.
I've co-opted a legacy UI feature (the sync-scaling LED). I'm sure there are other options I could have chosen, but right now this build effectively disables the official sync-scaling behaviour. I never used that myself and haven't got any decent insight about the importance of this to other users.
- I've kept it super simple here (no XML or flash changes) because: a) its an experiment and b) that'd lead to a bunch of compatibility issues (see below).

community related thoughts

I'm certain that what I've done here is not for everyone. And that's gotta a be a common issue with a platform that's got so many options and musical tastes involved and open-sourced firmware.
How to support optional features safely for community users. Maybe categories for different compatibility levels ??
It'd be cool to set up some guidelines for developer and users. re accessing, supporting, road-mapping, classifying the various deluge firmware branches that might emerge over time.
changing behaviour might affect buttons, menus, user and midi messages, LEDs ..... How to co-ordinate?and communicate this ...maybe the Wiki to start with

sichtbeton commented 1 year ago

How is disabling time-stretching in audio clips advantageous to just loading the sound into a kit? Would be curious how many use cases there are. The following are the ones I could think of and they revolve around clarity in Arranger view.

Audio clip length will be exactly as long as the sample. To do this in a Kit clip, one would have to set the sample to 'CUT' and place it on the first pad of the sequence. Then they would have to zoom in to the end of the sample's tail and bring the end of the clip length's as close as possible to that point.

When adjusting Audio Clip's pitch, the clip length will dynamically change length.

Benefit: Dragging a time-stretching disabled Audio Clip into Arranger will visually allow you to know exactly how long the sample plays.

Glad to hear about the progress!

One of the uses for example as a quick way to throw an acapella into the arranger. Using a kit is a clunky workaround. Also for ambient and other types of non beat oriented music this function would be great.

designerfuzzi commented 1 year ago

A audio clip is a chunk of data with a start and end, aka has a length. Timestretch is the same chunk spread or shrinked over a different length. If the audio is shorter than the timespan there are multiple ways of interpretation. It might loop, play and fade to silence and on and on.

But there is also "phase" where actually not much changes, just that start has an offset and same offset applied to the end allows different ways of application as well. As of now i dont know how to apply phase on an audio clip but i can phase shift Synth/Kit/Midi patterns. And Audio clips tend to be somehow offset to those points in time (notesOn) no matter how you record them unless you use an editing method to correct the start point or cut the audio. If the end + offset would be allowed to flip back to start and apply the missing offset we would get phase shift of audio clips and with it the very same workflow as shifting any pattern. That this is missing was surprising to me when the looper feature came along because the machine offers replay clips from start independent from main sequencing anyway.

So yes timestretching or rather interpretation of samples of audio chunk with (startpoint+offset) until (endpoint + offset) and stretch but also phase shift it is something to improve.

ok-reza commented 1 year ago

The value of this is definitely a lot clearer to me now. I'm so beat/grid centric that it didn't occur me the workflow improvements this could make, particular when 'off the grid'. I would love to see this feature implemented.

My thoughts on getting something like this approved for community release:

Make time-stretching (on/off) audio clip specific, instead of global.
Find a new shortcut that doesn't trump an official feature (SYNC-SCALING)
Allow XML song files made previous to this firmware automatically assign time-stretching to 'on' so previous songs are compatible.
When audio clip 'Pitch/Speed' settings are set to 'Linked', pitch shifting should automatically change the length of the Audio Clip to represent it's length based on the BPM (maybe this already happens??)
Adequate testing when adjusting Pitch/Speed, pitch, clip length (which I guess would affect pitch also), editing waveform etc.

Although probably more complex to implement, I think the shortcut could live in 'Shift+SAMPLE 1 MODE' (pad on bottom left corner of Deluge). There currently isn't any function to that pad and it would be somewhat intuitive for users to look. From there, the two options would be ON or OFF, not sure the right words to accommodate 7SEG. In KIT, there is the sample mode "STRE" which is for stretch so maybe "STRE" and "OFF"?

sichtbeton commented 1 year ago

The value of this is definitely a lot clearer to me now. I'm so beat/grid centric that it didn't occur me the workflow improvements this could make, particular when 'off the grid'. I would love to see this feature implemented.

My thoughts on getting something like this approved for community release:

Make time-stretching (on/off) audio clip specific, instead of global.

Find a new shortcut that doesn't trump an official feature (SYNC-SCALING)

Allow XML song files made previous to this firmware automatically assign time-stretching to 'on' so previous songs are compatible.

When audio clip 'Pitch/Speed' settings are set to 'Linked', pitch shifting should automatically change the length of the Audio Clip to represent it's length based on the BPM (maybe this already happens??)

Adequate testing when adjusting Pitch/Speed, pitch, clip length (which I guess would affect pitch also), editing waveform etc.

Although probably more complex to implement, I think the shortcut could live in 'Shift+SAMPLE 1 MODE' (pad on bottom left corner of Deluge). There currently isn't any function to that pad and it would be somewhat intuitive for users to look. From there, the two options would be ON or OFF, not sure the right words to accommodate 7SEG. In KIT, there is the sample mode "STRE" which is for stretch so maybe "STRE" and "OFF"?

This all makes a lot of sense! Thank you @ok-reza. Exactly what I thought too.

Will this be incorporated into the community branch then? Would be awesome to have this feature.

chrisbc commented 1 year ago

@ok-reza & @sichtbeton awesome - thanks for the feedback. I'd be happy to take on those suggested changes/features. I'll raise some tickets in my mupaduw forks, and perhaps I can get you involved as reviewers/contributors/testers?

chrisbc commented 1 year ago

A couple of other ideas for features...

per-clip setting to determine end-of-loop syncing behaviour (currently shorter loops will just loop, and then reset with the longest loop, which is a bit strange. Maybe this relates to your point(s) @designerfuzzi ??
per-clip half-speed, double-speed playback (with pitch change) as a performance option (fun times in sooperlooper)

Also ....I am by no means an deluge 'expert user' and might easily miss or misunderstand what's already there. I won't be offended if anyone suggests better ways/ideas !

sichtbeton commented 1 year ago

A couple of other ideas for features...

per-clip setting to determine end-of-loop syncing behaviour (currently shorter loops will just loop, and then reset with the longest loop, which is a bit strange. Maybe this relates to your point(s) @designerfuzzi ??

per-clip half-speed, double-speed playback (with pitch change) as a performance option (fun times in sooperlooper)

Also ....I am by no means an deluge 'expert user' and might easily miss or misunderstand what's already there. I won't be offended if anyone suggests better ways/ideas !

A simple switch to turn it off per clip would augment the deluge quite a lot as numerous people have asked for this. especially in the arranger this function makes total sense. so only having this would make a lot of people happy.

the fancy bells and whistels (looping behaviour etc) are maybe cherries, but not essential to the cake.

thank you for your efforts!

sichtbeton commented 1 year ago

Could this be converted to a draft PR as this is not an issue but a feature WIP.

m-m-adams commented 11 months ago

Closing as it does not appear to have ongoing work

SynthstromAudible / DelugeFirmware