Abstract
Increasingly high-stakes decisions are made using neural networks in order to make predictions. Specifically, meteorologists and hedge funds apply these techniques to time series data. When it comes to prediction, there are certain limitations for machine learning models (such as lack of expressiveness, vulnerability of domain shifts and overconfidence) which can be solved using uncertainty estimation. There is a set of expectations regarding how uncertainty should ``behave". For instance, a wider prediction horizon should lead to more uncertainty or the model's confidence should be proportional to its accuracy. In this paper, different uncertainty estimation methods are compared to forecast meteorological time series data and evaluate these expectations. The results show how each uncertainty estimation method performs on the forecasting task, which partially evaluates the robustness of predicted uncertainty.
Keyword: confidence
Test-time adversarial detection and robustness for localizing humans using ultra wide band channel impulse responses
Authors: Abhiram Kolli, Muhammad Jehanzeb Mirza, Horst Possegger, Horst Bischof
Abstract
Keyless entry systems in cars are adopting neural networks for localizing its operators. Using test-time adversarial defences equip such systems with the ability to defend against adversarial attacks without prior training on adversarial samples. We propose a test-time adversarial example detector which detects the input adversarial example through quantifying the localized intermediate responses of a pre-trained neural network and confidence scores of an auxiliary softmax layer. Furthermore, in order to make the network robust, we extenuate the non-relevant features by non-iterative input sample clipping. Using our approach, mean performance over 15 levels of adversarial perturbations is increased by 55.33% for the fast gradient sign method (FGSM) and 6.3% for both the basic iterative method (BIM) and the projected gradient method (PGD).
MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge Distillation
Authors: Lingtong Kong, Jie Yang
Subjects: Computer Vision and Pattern Recognition (cs.CV)
Abstract
Recent works have shown that optical flow can be learned by deep networks from unlabelled image pairs based on brightness constancy assumption and smoothness prior. Current approaches additionally impose an augmentation regularization term for continual self-supervision, which has been proved to be effective on difficult matching regions. However, this method also amplify the inevitable mismatch in unsupervised setting, blocking the learning process towards optimal solution. To break the dilemma, we propose a novel mutual distillation framework to transfer reliable knowledge back and forth between the teacher and student networks for alternate improvement. Concretely, taking estimation of off-the-shelf unsupervised approach as pseudo labels, our insight locates at defining a confidence selection mechanism to extract relative good matches, and then add diverse data augmentation for distilling adequate and reliable knowledge from teacher to student. Thanks to the decouple nature of our method, we can choose a stronger student architecture for sufficient learning. Finally, better student prediction is adopted to transfer knowledge back to the efficient teacher without additional costs in real deployment. Rather than formulating it as a supervised task, we find that introducing an extra unsupervised term for multi-target learning achieves best final results. Extensive experiments show that our approach, termed MDFlow, achieves state-of-the-art real-time accuracy and generalization ability on challenging benchmarks. Code is available at https://github.com/ltkong218/MDFlow.
Weighing votes in human-machine collaboration for hazard recognition: Inferring hazard perceptual threshold and decision confidence from electroencephalogram wavelets
Abstract
Purpose: Human-machine collaboration is a promising strategy to improve hazard inspection. However, research on the effective integration of opinions from humans with machines for optimal group decision making is lacking. Hence, considering the benefits of a brain-computer interface (BCI) to enable intuitive commutation, this study proposes a novel method to predict human hazard response choices and decision confidence from brain activities for a superior confidence-weighted voting strategy. Methodology: First, we developed a Bayesian inference-based algorithm to ascertain the decision threshold above which a hazard is reported from human brain signals. This method was tested empirically with electroencephalogram (EEG) data collected in a laboratory setting and cross-validated using behavioral indices of the signal detection theory. Subsequently, based on numerical simulations, the decision criteria for low-, medium-, and high-confidence level differentiations characterized by parietal alpha-band EEG power were determined. Findings : The investigated hazard recognition task was described as a process of probabilistic inference involving a decision uncertainty evaluation. The results demonstrated the feasibility of EEG measurements in observing human internal representations of hazard discrimination. Moreover, the optimal criteria to differentiate between low-, medium-, and high-confidence levels were obtained by benchmarking against an optimal Bayesian observer. Originality: This research demonstrates the potential of a BCI as an effective channel for telecommunication, laying the foundation for the design of future hazard detection techniques in the collaborative human-machine systems research field.
Comparison of Uncertainty Quantification with Deep Learning in Time Series Regression
Abstract
Increasingly high-stakes decisions are made using neural networks in order to make predictions. Specifically, meteorologists and hedge funds apply these techniques to time series data. When it comes to prediction, there are certain limitations for machine learning models (such as lack of expressiveness, vulnerability of domain shifts and overconfidence) which can be solved using uncertainty estimation. There is a set of expectations regarding how uncertainty should ``behave". For instance, a wider prediction horizon should lead to more uncertainty or the model's confidence should be proportional to its accuracy. In this paper, different uncertainty estimation methods are compared to forecast meteorological time series data and evaluate these expectations. The results show how each uncertainty estimation method performs on the forecasting task, which partially evaluates the robustness of predicted uncertainty.
Leveraging the Hints: Adaptive Bidding in Repeated First-Price Auctions
Abstract
With the advent and increasing consolidation of e-commerce, digital advertising has very recently replaced traditional advertising as the main marketing force in the economy. In the past four years, a particularly important development in the digital advertising industry is the shift from second-price auctions to first-price auctions for online display ads. This shift immediately motivated the intellectually challenging question of how to bid in first-price auctions, because unlike in second-price auctions, bidding one's private value truthfully is no longer optimal. Following a series of recent works in this area, we consider a differentiated setup: we do not make any assumption about other bidders' maximum bid (i.e. it can be adversarial over time), and instead assume that we have access to a hint that serves as a prediction of other bidders' maximum bid, where the prediction is learned through some blackbox machine learning model. We consider two types of hints: one where a single point-prediction is available, and the other where a hint interval (representing a type of confidence region into which others' maximum bid falls) is available. We establish minimax optimal regret bounds for both cases and highlight the quantitatively different behavior between the two settings. We also provide improved regret bounds when the others' maximum bid exhibits the further structure of sparsity. Finally, we complement the theoretical results with demonstrations using real bidding data.
Õptimal Differentially Private Learning of Thresholds and Quasi-Concave Optimization
Authors: Edith Cohen, Xin Lyu, Jelani Nelson, Tamás Sarlós, Uri Stemmer
Subjects: Machine Learning (cs.LG); Cryptography and Security (cs.CR); Data Structures and Algorithms (cs.DS)
Abstract
The problem of learning threshold functions is a fundamental one in machine learning. Classical learning theory implies sample complexity of $O(\xi^{-1} \log(1/\beta))$ (for generalization error $\xi$ with confidence $1-\beta$). The private version of the problem, however, is more challenging and in particular, the sample complexity must depend on the size $|X|$ of the domain. Progress on quantifying this dependence, via lower and upper bounds, was made in a line of works over the past decade. In this paper, we finally close the gap for approximate-DP and provide a nearly tight upper bound of $\tilde{O}(\log^ |X|)$, which matches a lower bound by Alon et al (that applies even with improper learning) and improves over a prior upper bound of $\tilde{O}((\log^ |X|)^{1.5})$ by Kaplan et al. We also provide matching upper and lower bounds of $\tilde{\Theta}(2^{\log^*|X|})$ for the additive error of private quasi-concave optimization (a related and more general problem). Our improvement is achieved via the novel Reorder-Slice-Compute paradigm for private data analysis which we believe will have further applications.
Keyword: scaling
Efficient and Accurate Quantized Image Super-Resolution on Mobile NPUs, Mobile AI & AIM 2022 challenge: Report
Authors: Andrey Ignatov, Radu Timofte, Maurizio Denna, Abdel Younes, Ganzorig Gankhuyag, Jingang Huh, Myeong Kyun Kim, Kihwan Yoon, Hyeon-Cheol Moon, Seungho Lee, Yoonsik Choe, Jinwoo Jeong, Sungjei Kim, Maciej Smyl, Tomasz Latkowski, Pawel Kubik, Michal Sokolski, Yujie Ma, Jiahao Chao, Zhou Zhou, Hongfan Gao, Zhengfeng Yang, Zhenbing Zeng, Zhengyang Zhuge, Chenghua Li, Dan Zhu, Mengdi Sun, Ran Duan, Yan Gao, Lingshun Kong, Long Sun, Xiang Li, Xingdong Zhang, Jiawei Zhang, Yaqi Wu, Jinshan Pan, Gaocheng Yu, Jin Zhang, Feng Zhang, Zhe Ma, Hongbin Wang, Hojin Cho, Steve Kim, Huaen Li, Yanbo Ma, Ziwei Luo, Youwei Li, Lei Yu, Zhihong Wen, Qi Wu, Haoqiang Fan, Shuaicheng Liu, Lize Zhang, Zhikai Zong, Jeremy Kwon, Junxi Zhang, Mengyuan Li, Nianxiang Fu, Guanchen Ding, Han Zhu, Zhenzhong Chen, Gen Li, Yuanfan Zhang, et al. (33 additional authors not shown)
Subjects: Computer Vision and Pattern Recognition (cs.CV)
Abstract
Image super-resolution is a common task on mobile and IoT devices, where one often needs to upscale and enhance low-resolution images and video frames. While numerous solutions have been proposed for this problem in the past, they are usually not compatible with low-power mobile NPUs having many computational and memory constraints. In this Mobile AI challenge, we address this problem and propose the participants to design an efficient quantized image super-resolution solution that can demonstrate a real-time performance on mobile NPUs. The participants were provided with the DIV2K dataset and trained INT8 models to do a high-quality 3X image upscaling. The runtime of all models was evaluated on the Synaptics VS680 Smart Home board with a dedicated edge NPU capable of accelerating quantized neural networks. All proposed solutions are fully compatible with the above NPU, demonstrating an up to 60 FPS rate when reconstructing Full HD resolution images. A detailed description of all models developed in the challenge is provided in this paper.
Structured Singular Value of a Repeated Complex Full Block Uncertainty
Authors: Talha Mushtaq, Diganta Bhattacharjee, Peter Seiler, Maziar S. Hemati
Subjects: Systems and Control (eess.SY); Optimization and Control (math.OC)
Abstract
The structured singular value (SSV), or mu, is used to assess the robust stability and performance of an uncertain linear time-invariant system. Existing algorithms compute upper and lower bounds on the SSV for structured uncertainties that contain repeated (real or complex) scalars and/or non-repeated complex full blocks. This paper presents algorithms to compute bounds on the SSV for the case of repeated complex full blocks. This specific class of uncertainty is relevant for the input output analysis of many convective systems, such as fluid flows. Specifically, we present a power iteration to compute a lower bound on SSV for the case of repeated complex full blocks. This generalizes existing power iterations for repeated complex scalar and non-repeated complex full blocks. The upper bound can be formulated as a semi-definite program (SDP), which we solve using a standard interior-point method to compute optimal scaling matrices associated with the repeated full blocks. Our implementation of the method only requires gradient information, which improves the computational efficiency of the method. Finally, we test our proposed algorithms on an example model of incompressible fluid flow. The proposed methods provide less conservative bounds as compared to prior results, which ignore the repeated full block structure.
pyRDDLGym: From RDDL to Gym Environments
Authors: Ayal Taitler, Michael Gimelfarb, Sriram Gopalakrishnan, Martin Mladenov, Xiaotian Liu, Scott Sanner
Abstract
We present pyRDDLGym, a Python framework for auto-generation of OpenAI Gym environments from RDDL declerative description. The discrete time step evolution of variables in RDDL is described by conditional probability functions, which fits naturally into the Gym step scheme. Furthermore, since RDDL is a lifted description, the modification and scaling up of environments to support multiple entities and different configurations becomes trivial rather than a tedious process prone to errors. We hope that pyRDDLGym will serve as a new wind in the reinforcement learning community by enabling easy and rapid development of benchmarks due to the unique expressive power of RDDL. By providing explicit access to the model in the RDDL description, pyRDDLGym can also facilitate research on hybrid approaches for learning from interaction while leveraging model knowledge. We present the design and built-in examples of pyRDDLGym, and the additions made to the RDDL language that were incorporated into the framework.
Coarse-graining of CFD-DEM for simulation of sand production in the modified cohesive contact model
Abstract
Sand production is an important issue for many hydrocarbon recovery applications in unconsolidated reservoirs. The model using the Computational Fluid Dynamics coupled with Discrete Element Method (CFD-DEM) can capture micro-scale features of sand transport problems. In this study, a coarse-graining approach of 3D CFD-DEM model is developed for the sand production phenomenon using the sample based on the particle size distribution (PSD) from the Kazakhstan reservoir field. The derivation of scaling from a fine to a coarse model is presented rigorously. The original (fine scale) model is validated to the laboratory results including the sand production rate. The results of the original model is compared to the SSW and SSP coarse-gained models. The SSW model results agrees with the result of the sand production rate for the original system. We also observe a good match of the fluid velocity streamline and the produced particle distribution between the original and the coarse models. The speedup of the coarse model is up to 9.4 in the parallelized coarse-grained model.
Theoretical error estimates for computing the matrix logarithm by Padé-type approximants
Abstract
In this article, we focus on the error that is committed when computing the matrix logarithm using the Gauss--Legendre quadrature rules. These formulas can be interpreted as Pad\'e approximants of a suitable Gauss hypergeometric function. Empirical observation tells us that the convergence of these quadratures becomes slow when the matrix is not close to the identity matrix, thus suggesting the usage of an inverse scaling and squaring approach for obtaining a matrix with this property. The novelty of this work is the introduction of error estimates that can be used to select a priori both the number of Legendre points needed to obtain a given accuracy and the number of inverse scaling and squaring to be performed. We include some numerical experiments to show the reliability of the estimates introduced.
DistGNN-MB: Distributed Large-Scale Graph Neural Network Training on x86 via Minibatch Sampling
Abstract
Training Graph Neural Networks, on graphs containing billions of vertices and edges, at scale using minibatch sampling poses a key challenge: strong-scaling graphs and training examples results in lower compute and higher communication volume and potential performance loss. DistGNN-MB employs a novel Historical Embedding Cache combined with compute-communication overlap to address this challenge. On a 32-node (64-socket) cluster of $3^{rd}$ generation Intel Xeon Scalable Processors with 36 cores per socket, DistGNN-MB trains 3-layer GraphSAGE and GAT models on OGBN-Papers100M to convergence with epoch times of 2 seconds and 4.9 seconds, respectively, on 32 compute nodes. At this scale, DistGNN-MB trains GraphSAGE 5.2x faster than the widely-used DistDGL. DistGNN-MB trains GraphSAGE and GAT 10x and 17.2x faster, respectively, as compute nodes scale from 2 to 32.
Keyword: calibration
Rethinking Log Odds: Linear Probability Modelling and Expert Advice in Interpretable Machine Learning
Authors: Danial Dervovic, Nicolas Marchesotti, Freddy Lecue, Daniele Magazzeni
Abstract
We introduce a family of interpretable machine learning models, with two broad additions: Linearised Additive Models (LAMs) which replace the ubiquitous logistic link function in General Additive Models (GAMs); and SubscaleHedge, an expert advice algorithm for combining base models trained on subsets of features called subscales. LAMs can augment any additive binary classification model equipped with a sigmoid link function. Moreover, they afford direct global and local attributions of additive components to the model output in probability space. We argue that LAMs and SubscaleHedge improve the interpretability of their base algorithms. Using rigorous null-hypothesis significance testing on a broad suite of financial modelling data, we show that our algorithms do not suffer from large performance penalties in terms of ROC-AUC and calibration.
Keyword: out of distribution detection
There is no result
Keyword: out-of-distribution detection
There is no result
Keyword: expected calibration error
There is no result
Keyword: overconfident
There is no result
Keyword: overconfidence
Comparison of Uncertainty Quantification with Deep Learning in Time Series Regression
Keyword: confidence
Test-time adversarial detection and robustness for localizing humans using ultra wide band channel impulse responses
MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge Distillation
Weighing votes in human-machine collaboration for hazard recognition: Inferring hazard perceptual threshold and decision confidence from electroencephalogram wavelets
Comparison of Uncertainty Quantification with Deep Learning in Time Series Regression
Leveraging the Hints: Adaptive Bidding in Repeated First-Price Auctions
Õptimal Differentially Private Learning of Thresholds and Quasi-Concave Optimization
Keyword: scaling
Efficient and Accurate Quantized Image Super-Resolution on Mobile NPUs, Mobile AI & AIM 2022 challenge: Report
Structured Singular Value of a Repeated Complex Full Block Uncertainty
pyRDDLGym: From RDDL to Gym Environments
Coarse-graining of CFD-DEM for simulation of sand production in the modified cohesive contact model
Theoretical error estimates for computing the matrix logarithm by Padé-type approximants
DistGNN-MB: Distributed Large-Scale Graph Neural Network Training on x86 via Minibatch Sampling
Keyword: calibration
Rethinking Log Odds: Linear Probability Modelling and Expert Advice in Interpretable Machine Learning