Input data shape

[ZFTurbo]

Hello, there is a problem with input data. You write

x = torch.randn(2, 131680)
target = torch.randn(2, 131680)

loss = model(x, target = target)
loss.backward()

# after much training

out = model(x)

But in reality there must be also some batch size:

x = torch.randn(10, 2, 131680)
target = torch.randn(10, 2, 131680)

loss = model(x, target = target)
loss.backward()

# after much training

out = model(x)

If I use like in bottom example I have a problem with stft.

RuntimeError: stft(torch.cuda.FloatTensor[6, 2, 133728], n_fft=2048, hop_length=512, win_length=2048, window=None, normalized=0, onesided=None, return_complex=1) : expected a 1D or 2D tensor

[lucidrains]

@ZFTurbo i'm using 2 as the batch size, with an implicit channel dimension of 1

[ZFTurbo]

Is it possible to remade for stereo input?

[lucidrains]

@ZFTurbo ah i see

in stereo, the channel dimension would be 2?

[ZFTurbo]

@ZFTurbo ah i see

in stereo, the channel dimension would be 2?

Yes )

[lucidrains]

ok, so i just do stft on each dimension and concat? sorry but i'm actually not a domain expert (besides the transformer part)

[ZFTurbo]

I think here is the simple example how it can be done: https://github.com/kuielab/sdx23/blob/mdx_AB/my_submission/src/tfc_tdf_v3.py

[lucidrains]

@ZFTurbo ahh got it, yea, like i guessed

yup, i can do this, and i guess the multi-stft loss can also be done per channel dimension and just summed

[lucidrains]

tada

try 0.1.0

import torch
from bs_roformer import BSRoformer

model = BSRoformer(
    dim = 512,
    depth = 12,
    stereo = True,
    time_transformer_depth = 1,
    freq_transformer_depth = 1,
    freqs_per_bands = (512, 513)  # in the paper, they divide into ~60 bands, test with 1 for starters
)

x = torch.randn(2, 2, 131680)
target = torch.randn(2, 2, 131680)

loss = model(x, target = target)
loss.backward()

# after much training

out = model(x)

[lucidrains]

ok, calling it a day, happy friday!

[ZFTurbo]

Thank you. It works for me now.

One more proposal. For MSS task you sometimes need more than one stems on output.

so target can have shape like (BS, Stems number, 2, length)

target = torch.randn(2, 4, 2, 131680)

May be it's good idea to add to support for example 4 stems separation.

[lucidrains]

ok, very last commit for the day

import torch
from bs_roformer import BSRoformer

model = BSRoformer(
    dim = 512,
    depth = 12,
    stereo = True,
    num_stems = 2,
    time_transformer_depth = 1,
    freq_transformer_depth = 1,
    freqs_per_bands = (512, 513)  # in the paper, they divide into ~60 bands, test with 1 for starters
)

x = torch.randn(2, 2, 131680)
target = torch.randn(2, 2, 2, 131680)

loss = model(x, target = target)
loss.backward()

# after much training

out = model(x)

[lucidrains]

if that doesn't look right, i can address it next Monday

[ZFTurbo]

Thank you I will test it!

[lucidrains]

@ZFTurbo if stems is just different targets, you could also just train 4 separate models (in your example), or am i misunderstanding?

[ZFTurbo]

@ZFTurbo if stems is just different targets, you could also just train 4 separate models (in your example), or am i misunderstanding?

Yes, you are right. But sometimes it's better to have all in single model.

[lucidrains]

ohh got it, so 'stems' just another synonym for adapter heads

yea, this should work then

[lucidrains]

i guess you could also have a separate axial transformer block per stem head, but can leave that for another day

[lucidrains]

main issue has been addressed