search

hangzhaomit / Sound-of-Pixels

Codebase for ECCV18 "The Sound of Pixels"
http://sound-of-pixels.csail.mit.edu
MIT License
370 stars 74 forks source link

where is the pixelwise sound #14

Open TaoZheng9 opened 3 years ago

TaoZheng9 commented 3 years ago

Hi, I saw the func: forward_pixelwise in the code synthesizer, this is the one version of forward function that produce pixel-wise mask. However, throughout the code, and I found only the foward func is invoked but it is not the one of pixel-wise sound. Is there any demo that can produce pixel-wise sound?

thirteen-bears commented 1 week ago

The function forward_pixelwise(self, feats_img, feat_sound) helps compute the audio spectrum for all points on the feature map. Specifically, it calculates the correlation between the visual features (from an image or video frame) and the audio features (from a spectrogram), producing a result that associates each feature map point with its corresponding audio spectrum.

If upsampling is applied after this function, the output will correspond to the audio spectrum of each individual pixel in the original image. This means that each point in the image will have its own corresponding audio spectrum, which can be used for tasks such as sound source localization.