Open migperfer opened 1 year ago
I've been trying to track which function might the one breaking these tests, and I'm ultra confused now. I rewrote the test a bit, so only the complex CQT ground truth is loaded; and magnitude and phase ground truths are extracted from there.
I'm even more confused now:
@pytest.mark.parametrize("device", [*device_args])
def test_cqt_1992_v2_linear(device):
# Linear sweep case
fs = 44100
t = 1
f0 = 55
f1 = 22050
s = np.linspace(0, t, fs * t)
x = chirp(s, f0, 1, f1, method="linear")
x = x.astype(dtype=np.float32)
# Get the complex ground truth for CQT1992v2, magnitude and phase can be derived from it
complex_ground_truth = np.load(
os.path.join(
dir_path, "ground-truths/linear-sweep-cqt-1992-complex-ground-truth.npy"
)
)
ground_truth_complex_real = complex_ground_truth[..., 0]
ground_truth_complex_img = complex_ground_truth[..., 1]
magnitude_ground_truth = np.log(np.sqrt(np.power(ground_truth_complex_real, 2) + np.power(ground_truth_complex_img, 2)) + 1e-5)
phase_ground_truth_atan2 = np.arctan2(ground_truth_complex_img, ground_truth_complex_real)
phase_ground_truth_real = np.cos(phase_ground_truth_atan2)
phase_ground_truth_img = np.sin(phase_ground_truth_atan2)
phase_ground_truth = np.stack([phase_ground_truth_real, phase_ground_truth_img], axis=-1)
# Magnitude
stft = CQT1992v2(
sr=fs, fmin=55, output_format="Magnitude", n_bins=207, bins_per_octave=24
).to(device)
X = stft(torch.tensor(x, device=device).unsqueeze(0))
X = torch.log(X + 1e-5)
assert np.allclose(X.cpu(), magnitude_ground_truth, rtol=1e-3, atol=1e-3)
# Complex
stft = CQT1992v2(
sr=fs, fmin=55, output_format="Complex", n_bins=207, bins_per_octave=24
).to(device)
X = stft(torch.tensor(x, device=device).unsqueeze(0))
assert np.allclose(X.cpu(), complex_ground_truth, rtol=1e-3, atol=1e-3)
assert np.allclose(np.arctan2(X[..., 1].cpu(), X[..., 0].cpu()), phase_ground_truth_atan2, rtol=1e-3, atol=1e-3)
assert np.allclose(torch.atan2(X[..., 1].cpu(), X[..., 0].cpu()), phase_ground_truth_atan2, rtol=1e-3, atol=1e-3)
# Phase
stft = CQT1992v2(
sr=fs, fmin=55, output_format="Phase", n_bins=207, bins_per_octave=24
).to(device)
X = stft(torch.tensor(x, device=device).unsqueeze(0))
assert np.allclose(X.cpu(), phase_ground_truth, rtol=1e-3, atol=1e-3)
This is quite surprising, as the complex CQT seems to be extracted correctly. Am I doing something wrong when calculating the phase?
How can it be possible that not even np.arctan2
gets the same result? (Look at second assertion of the # Complex
section). But what is surprising to me is that it fails only on the CPU; I'm testing this on a MacBook with MPS, and when I use MPS the test pass perfectly.
Thank you @migperfer for the investigation.
May I know which pytorch and numpy version are you using?
I tried torch 1.8.1 and numpy=1.19.5. It seems the results for CQT are all fine now. So I believe that it is related to the notorious floating-point error. Can you try adjusting the tolerance a bit such as rtol=1e-2
and atol=1e-2
?
I have attached the full unit test result below unit_test_report.txt
There are still problems for CFP
, STFT
, and VQT
. I am investigating the issues now.
I managed to resolve all failed test cases. Turns out it's all floating-point error. I make the conditions less strict and all tests pass now.
They are working indeed!
All tests are working for me except the CQT ones.
I think there could be a potential issue with MacBooks. I'm using a MacBook, phase is still wrong even with tolerances 1e-1
.
I see that #126 do not change at all the CQT and still is working on Github actions (I Imagine they run on linux), so I'd say the Macbook case a very nitpick one.
Are you using a MacBook with M1/M2 processor? Not sure if it induces extra problems. Well I would say floating-point error is really a big headache.
An M2 indeed. And yes, it is a headache 😂
Hi! As discussed in #123 some tests are failing. It is for now unclear what could lead to these errors.
So far the tests that are failing are:
CFP
CQT
STFT
VQT
These are failing at least for CPU. If someone with access to a GPU wants to collaborate please feel free to post the results of the tests for the GPU.