search

NUS-HPC-AI-Lab / Neural-Network-Parameter-Diffusion

We introduce a novel approach for parameter generation, named neural network parameter diffusion (p-diff), which employs a standard latent diffusion model to synthesize a new set of parameters
787 stars 38 forks source link

Discrepancy between average and median for linear layer parameters generation? #8

Closed SKDDJ closed 3 months ago

SKDDJ commented 3 months ago
image

In the table of the paper, there seems to be a discrepancy between the average and median values reported for the linear layer parameters. Specifically, the average is listed as 47.3, while the median is reported as 71.1.

This disparity raises the question: Is this a typographical or numerical error in the reported values, or does it accurately reflect the underlying data distribution? In the latter case, the lower average compared to the median could potentially indicate that the model generates a significant number(almost half) of low quality parameters, thereby skewing the average downward while the median remains higher.

Could you please clarify whether the values presented are correct, and if so, provide some insight into the possible reasons behind this divergence between the average and median for the linear layer parameters?

Thanks.

1zeryu commented 3 months ago

Thank you for your feedback. Firstly, the average and median of linear layer experiment is right. The reason for this is that parameter generation is random and accidental. The average is susceptible to extremely low values, in other words, if one or more examples produce very poor results, the average will drag down.