Possible loss functions for a covariance matrix Sigma, based on observing an iid sample of a vector X,
Squared error losses:
L(Sigma)(X)=sum{kl}(X{kl}-mu_{kl})^2 f(k,l) with weight function f(k,l), where mu(k,l)=EX_kX_l. In particular, f(k,l) = 1/p provides a theoretically sound loss when the true covariance matrix is assumed to be sparse, and can be easily decomposed into a useable empirical loss. When f(k,l) = 1/p^2, a similarly theoretically sound empirical loss function can be derived, though it is best suited for situations where the true covariance is not assumed to be sparse, such as in random effect models / latent variable models.
L(Sigma)(X)= (x{kl}-mu(k,l):k,l)^t (rho)^{-1} (x{kl}-mu(k,l):k,l), where rho would represent an estimator of covariance matrix of vector (x_{kl}: k,l)
Operator norm: An operator norm loss will be ideal for when the estimand is not the covariance matrix, but the leading eigenvalues and eigenvectors of the true covariance matrix.
Possible loss functions for a covariance matrix Sigma, based on observing an iid sample of a vector X,