Roman223
commented
3 weeks ago Intro
The foundation for solution supposed to be integrated are based on article (Sha Lu, et all. Effective Outlier Detection based on Bayesian Network and Proximity), namely ODBP . In this algorithm bn must be linear in order to estimate mean of mutivariate random variable (MRV) as linear combination of parents' values.
Algorithm
If we want to use for oulier detection not only density estimation but also a proximity estimation we need a way to combine them. In this work authors suggested just sum up all outlier factors from detectors. Greater the factor greater the chance that given sample is outlier.
Foundation
Density estimators stores information about how dense points in space are. In this terms, outlier is a point that has too deviant values from expectation of MRV. Note that if values are on boundary of point cloud, it is hard to detect such a point. Authors solve this issue by adjusting proximity metric.
Essence
Model estimation
Theirs: Given data D and corresponding bn. For every node that has any parents, estimate its mean as linear combination depending on parents values. It is supposed here that for sample of fixed child variable, mean must converge ;into its true value whereas variance converges into zero. Any step away is treated as anomaly either in child variable or in parents variables (or both). Final outlier factor is just Z-score for given child node. Therefore, another limitation comes into game: we must guarantee a normal distribution for out estimations. Despite no proof was provided, this statement probably holds by virtue of linearity of BN. Ours: Since we have mixed data, we have no guaratee of linearity at all. Moreover, we cannot apply Z-score when use mean estimation in general, because for conditional node for every combination of parents we store a different model (not only Gaussian, it can be any), but we still can estimate mean and thus can find deviation from it with some errors (such MAE or MSE). Discrete RV are coded and categories are used as new RV. Here what is done:
node = self.estimator.bn[node_name]
diff = []
dist = self.estimator.bn.distributions[node_name]
parents = node.cont_parents + node.disc_parents
for _, row in X.iterrows():
pvalues = row[parents].to_dict()
pvals_bamt_style = [pvalues[parent] for parent in parents]
cond_dist = self.estimator.bn.get_dist(node_name, pvals=pvalues)
# not good written, subject to change after bamt 2 release.
if isinstance(cond_dist, tuple):
# if returns mean and variances
cond_mean = cond_dist[0]
else:
# otherwise
dispvals = []
for pval in pvals_bamt_style:
if isinstance(pval, str):
dispvals.append(pval)
if "vals" in dist.keys():
classes = dist["vals"]
elif "classes" in dist.keys():
classes = dist["classes"]
elif "hybcprob" in dist.keys():
if "classes" in dist["hybcprob"][str(dispvals)]:
classes = dist["hybcprob"][str(dispvals)]["classes"]
else:
raise Exception()
else:
raise Exception()
# get list of classes for random variable to calculate weighted sum (expectation)
classes_coded = np.asarray([self.encoding[node_name][class_name] for class_name in classes])
cond_mean = classes_coded @ np.asarray(cond_dist).T
if isinstance(row[node_name], str):
# MAE
diff.append(abs(cond_mean - self.encoding[node_name][row[node_name]]))
else:
# MAE
diff.append(abs(cond_mean - row[node_name]))
# Subject to vary, explaination in text
scaler = StandardScaler()
diff_scaled = scaler.fit_transform(np.asarray(diff).reshape(-1, 1))
return diff_scaledProximity estimation
Theirs: The idea is simple, just choose randomly t columns from initial dataset, then find Local Outlier Factor (LOF) for given subsample.
LOF has values less then 1 if local density (density of neighbors) is high (so point is not anomalous), 1 if it is the same and greater than 1 if local density is low (outliners).
Again here it is assumed that obtained values will be distributed normally and Z-score is possible to apply.
Problem here arises when columns are discrete, LOF cannot be calculated here. This operation is a body of loop, single loop is called "detector" by the authors. Its number is hyperparameter. PS: Simple explaination on LOF here. Ours: For now we just drop discrete columns if they apears in random subsample.
On outlier score
After obtainig both local scores we need to merge them into one. Authors' suggestion is just sum up all non zero $score_{ij}$ (i - number of sample, j - number of detector).
On shapes
All local scores has shape (n, d), where n - number of sample, d - number of detectors / number of child nodes. There are two summation: first -- over d-axis, next -- over proximity scores (n,) and model scores (n,). Final score has shape (n,).
First experiments will be posted as single posts with corresponding discussion.
Для табличный данных скорее всего будет реализован простой алгоритм детекции аномалий. Для временных рядов скорее всего придётся дорабатывать существенно алгоритм обучения ДБС, также можно рассмотреть задачу поиска аномалий на основе изменений в структуре ДБС. Можно рассматривать БС как вспомогательный инструмент для поиска аномалий.