Submission to the HEAR 2021 Challenge

For model evaluation, python=3.7 and cuda10.2 with cudnn7.6.5 have been tested.

The work uses a mixed supervised and self-supervised training regime.

Usage

First install the package:

python3 -m pip install git+https://github.com/richermans/HEAR2021_EfficientLatent.git

Then just use it:

import torch
import efficient_latent

model = efficient_latent.load_model()

audio = torch.randn(1, 16000) # Sampling rate is 16000

#Manual Time and Clip embeddings
time_embeddings = efficient_latent.get_timestamp_embeddings(audio, model)
clip_embeddings = efficient_latent.get_scene_embeddings(audio, model)
# Or for getting both embeddings in one pass
clip_embeddings, time_embeddings, time_stamps = efficient_latent.get_embeddings(audio, model)

Results

Notable results of this model in the challenge are:

DCASE2016 Task 2

Team Name   Submission  Event Onset FMS Segment Error Rate
CP-JKU  base2levelmel   0.925   0.099
CP-JKU  base2level  0.913   0.102
MARL + Soundsensing openl3_hear 0.833   0.174
NTU-GURA    fusion_cat_xwc  0.826   0.145
NTU-GURA    fusion_cat_xwc_time 0.826   0.145
NTU-GURA    fusion_hubert_xlarge    0.826   0.150
NTU-GURA    fusion_wav2vec2 0.798   0.163
*RedRice*   efficient_latent    0.790   0.231

Beijing Opera

Team Name   Submission  Accuracy
MARL + Soundsensing openl3_hear 0.975
NTU-GURA    fusion_cat_xwc  0.966
CP-JKU  base    0.966
CP-JKU  base2level  0.966
CP-JKU  base2levelmel   0.966
NTU-GURA    fusion_cat_xwc_time 0.962
RedRice efficient_latent    0.953

CREMA-D

Team Name   Submission  Accuracy
Logitech AI serab_byols 0.535
*RedRice*   efficient_latent    0.502
CVSSP   panns_hear  0.440
HEAR    wav2vec2    0.434

ESC-50

Team Name   Submission  Accuracy
CP-JKU  base    0.947
CP-JKU  base2level  0.947
CP-JKU  base2levelmel   0.947
*RedRice*   efficient_latent    0.935
CVSSP   panns_hear  0.909
Soundsensing    yamnet_hear 0.838

FSD50k

Team Name   Submission  mAP d'
CP-JKU  base    0.640   2.643
*RedRice*   efficient_latent    0.607   2.538
CP-JKU  base2levelmel   0.558   2.312
CP-JKU  base2level  0.537   2.292
Logitech AI serab_byols 0.509   2.218
MARL + Soundsensing openl3_hear 0.447   2.117

GTZAN Genre

Team Name   Submission  Accuracy
MARL + Soundsensing openl3_hear 0.796
RedRice efficient_latent    0.782
Logitech AI serab_byols 0.723
CVSSP   panns_hear  0.660

RicherMans / HEAR2021_EfficientLatent

readme