Abstract
Dense prediction tasks are common for 3D point clouds, but the inherent uncertainties in massive points and their embeddings have long been ignored. In this work, we present CUE, a novel uncertainty estimation method for dense prediction tasks of 3D point clouds. Inspired by metric learning, the key idea of CUE is to explore cross-point embeddings upon a conventional dense prediction pipeline. Specifically, CUE involves building a probabilistic embedding model and then enforcing metric alignments of massive points in the embedding space. We demonstrate that CUE is a generic and effective tool for dense uncertainty estimation of 3D point clouds in two different tasks: (1) in 3D geometric feature learning we for the first time obtain well-calibrated dense uncertainty, and (2) in semantic segmentation we reduce uncertainty`s Expected Calibration Error of the state-of-the-arts by 43.8%. All uncertainties are estimated without compromising predictive performance.
Keyword: overconfident
There is no result
Keyword: overconfidence
There is no result
Keyword: confidence
Out-of-Distribution Detection for LiDAR-based 3D Object Detection
Authors: Chengjie Huang, Van Duong Nguyen, Vahdat Abdelzad, Christopher Gus Mannes, Luke Rowe, Benjamin Therien, Rick Salay, Krzysztof Czarnecki
Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Robotics (cs.RO); Image and Video Processing (eess.IV)
Abstract
3D object detection is an essential part of automated driving, and deep neural networks (DNNs) have achieved state-of-the-art performance for this task. However, deep models are notorious for assigning high confidence scores to out-of-distribution (OOD) inputs, that is, inputs that are not drawn from the training distribution. Detecting OOD inputs is challenging and essential for the safe deployment of models. OOD detection has been studied extensively for the classification task, but it has not received enough attention for the object detection task, specifically LiDAR-based 3D object detection. In this paper, we focus on the detection of OOD inputs for LiDAR-based 3D object detection. We formulate what OOD inputs mean for object detection and propose to adapt several OOD detection methods for object detection. We accomplish this by our proposed feature extraction method. To evaluate OOD detection methods, we develop a simple but effective technique of generating OOD objects for a given object detection model. Our evaluation based on the KITTI dataset shows that different OOD detection methods have biases toward detecting specific OOD objects. It emphasizes the importance of combined OOD detection methods and more research in this direction.
Keyword: scaling
Breaking Time Invariance: Assorted-Time Normalization for RNNs
Authors: Cole Pospisil, Vasily Zadorozhnyy, Qiang Ye
Abstract
Methods such as Layer Normalization (LN) and Batch Normalization (BN) have proven to be effective in improving the training of Recurrent Neural Networks (RNNs). However, existing methods normalize using only the instantaneous information at one particular time step, and the result of the normalization is a preactivation state with a time-independent distribution. This implementation fails to account for certain temporal differences inherent in the inputs and the architecture of RNNs. Since these networks share weights across time steps, it may also be desirable to account for the connections between time steps in the normalization scheme. In this paper, we propose a normalization method called Assorted-Time Normalization (ATN), which preserves information from multiple consecutive time steps and normalizes using them. This setup allows us to introduce longer time dependencies into the traditional normalization methods without introducing any new trainable parameters. We present theoretical derivations for the gradient propagation and prove the weight scaling invariance property. Our experiments applying ATN to LN demonstrate consistent improvement on various tasks, such as Adding, Copying, and Denoise Problems and Language Modeling Problems.
Bayesian Neural Network Versus Ex-Post Calibration For Prediction Uncertainty
Authors: Satya Borgohain, Klaus Ackermann, Ruben Loaiza-Maya
Abstract
Probabilistic predictions from neural networks which account for predictive uncertainty during classification is crucial in many real-world and high-impact decision making settings. However, in practice most datasets are trained on non-probabilistic neural networks which by default do not capture this inherent uncertainty. This well-known problem has led to the development of post-hoc calibration procedures, such as Platt scaling (logistic), isotonic and beta calibration, which transforms the scores into well calibrated empirical probabilities. A plausible alternative to the calibration approach is to use Bayesian neural networks, which directly models a predictive distribution. Although they have been applied to images and text datasets, they have seen limited adoption in the tabular and small data regime. In this paper, we demonstrate that Bayesian neural networks yields competitive performance when compared to calibrated neural networks and conduct experiments across a wide array of datasets.
Scaling transformation of the multimode nonlinear Schrödinger equation for physics-informed neural networks
Authors: Ivan Chuprov, Dmitry Efremenko, Jiexing Gao, Pavel Anisimov, Viacheslav Zemlyakov
Subjects: Neural and Evolutionary Computing (cs.NE); Optics (physics.optics)
Abstract
Single-mode optical fibers (SMFs) have become the backbone of modern communication systems. However, their throughput is expected to reach its theoretical limit in the nearest future. Utilization of multimode fibers (MMFs) is considered as one of the most promising solutions rectifying this capacity crunch. Nevertheless, differential equations describing light propagation in MMFs are a way more sophisticated than those for SMFs, which makes numerical modelling of MMF-based systems computationally demanding and impractical for the most part of realistic scenarios. Physics-informed neural networks (PINNs) are known to outperform conventional numerical approaches in various domains and have been successfully applied to the nonlinear Schr\"odinger equation (NLSE) describing light propagation in SMFs. A comprehensive study on application of PINN to the multimode NLSE (MMNLSE) is still lacking though. To the best of our knowledge, this paper is the first to deploy the paradigm of PINN for MMNLSE and to demonstrate that a straightforward implementation of PINNs by analogy with NLSE does not work out. We pinpoint all issues hindering PINN convergence and introduce a novel scaling transformation for the zero-order dispersion coefficient that makes PINN capture all relevant physical effects. Our simulations reveal good agreement with the split-step Fourier (SSF) method and extend numerically attainable propagation lengths up to several hundred meters. All major limitations are also highlighted.
Keyword: calibration
Bayesian Neural Network Versus Ex-Post Calibration For Prediction Uncertainty
Authors: Satya Borgohain, Klaus Ackermann, Ruben Loaiza-Maya
Abstract
Probabilistic predictions from neural networks which account for predictive uncertainty during classification is crucial in many real-world and high-impact decision making settings. However, in practice most datasets are trained on non-probabilistic neural networks which by default do not capture this inherent uncertainty. This well-known problem has led to the development of post-hoc calibration procedures, such as Platt scaling (logistic), isotonic and beta calibration, which transforms the scores into well calibrated empirical probabilities. A plausible alternative to the calibration approach is to use Bayesian neural networks, which directly models a predictive distribution. Although they have been applied to images and text datasets, they have seen limited adoption in the tabular and small data regime. In this paper, we demonstrate that Bayesian neural networks yields competitive performance when compared to calibrated neural networks and conduct experiments across a wide array of datasets.
Exploring Cross-Point Embeddings for 3D Dense Uncertainty Estimation
Authors: Kaiwen Cai, Chris Xiaoxuan Lu, Xiaowei Huang
Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
Abstract
Dense prediction tasks are common for 3D point clouds, but the inherent uncertainties in massive points and their embeddings have long been ignored. In this work, we present CUE, a novel uncertainty estimation method for dense prediction tasks of 3D point clouds. Inspired by metric learning, the key idea of CUE is to explore cross-point embeddings upon a conventional dense prediction pipeline. Specifically, CUE involves building a probabilistic embedding model and then enforcing metric alignments of massive points in the embedding space. We demonstrate that CUE is a generic and effective tool for dense uncertainty estimation of 3D point clouds in two different tasks: (1) in 3D geometric feature learning we for the first time obtain well-calibrated dense uncertainty, and (2) in semantic segmentation we reduce uncertainty`s Expected Calibration Error of the state-of-the-arts by 43.8%. All uncertainties are estimated without compromising predictive performance.
Proportional Multicalibration
Authors: William La Cava, Elle Lett, Guangya Wan
Subjects: Machine Learning (cs.LG); Computers and Society (cs.CY)
Abstract
Multicalibration is a desirable fairness criteria that constrains calibration error among flexibly-defined groups in the data while maintaining overall calibration. However, when outcome probabilities are correlated with group membership, multicalibrated models can exhibit a higher percent calibration error among groups with lower base rates than groups with higher base rates. As a result, it remains possible for a decision-maker to learn to trust or distrust model predictions for specific groups. To alleviate this, we propose proportional multicalibration, a criteria that constrains the percent calibration error among groups and within prediction bins. We prove that satisfying proportional multicalibration bounds a model's multicalibration as well its differential calibration, a stronger fairness criteria inspired by the fairness notion of sufficiency. We provide an efficient algorithm for post-processing risk prediction models for proportional multicalibration and evaluate it empirically. We conduct simulation studies and investigate a real-world application of PMC-postprocessing to prediction of emergency department patient admissions. We observe that proportional multicalibration is a promising criteria for controlling simultenous measures of calibration fairness of a model over intersectional groups with virtually no cost in terms of classification performance.
Transfer Learning with Pretrained Remote Sensing Transformers
Authors: Anthony Fuller, Koreen Millard, James R. Green
Subjects: Computer Vision and Pattern Recognition (cs.CV)
Abstract
Although the remote sensing (RS) community has begun to pretrain transformers (intended to be fine-tuned on RS tasks), it is unclear how these models perform under distribution shifts. Here, we pretrain a new RS transformer--called SatViT-V2--on 1.3 million satellite-derived RS images, then fine-tune it (along with five other models) to investigate how it performs on distributions not seen during training. We split an expertly labeled land cover dataset into 14 datasets based on source biome. We train each model on each biome separately and test them on all other biomes. In all, this amounts to 1638 biome transfer experiments. After fine-tuning, we find that SatViT-V2 outperforms SatViT-V1 by 3.1% on in-distribution (matching biomes) and 2.8% on out-of-distribution (mismatching biomes) data. Additionally, we find that initializing fine-tuning from the linear probed solution (i.e., leveraging LPFT [1]) improves SatViT-V2's performance by another 1.2% on in-distribution and 2.4% on out-of-distribution data. Next, we find that pretrained RS transformers are better calibrated under distribution shifts than non-pretrained models and leveraging LPFT results in further improvements in model calibration. Lastly, we find that five measures of distribution shift are moderately correlated with biome transfer performance. We share code and pretrained model weights. (https://github.com/antofuller/SatViT)
Keyword: out of distribution detection
There is no result
Keyword: out-of-distribution detection
There is no result
Keyword: expected calibration error
Exploring Cross-Point Embeddings for 3D Dense Uncertainty Estimation
Keyword: overconfident
There is no result
Keyword: overconfidence
There is no result
Keyword: confidence
Out-of-Distribution Detection for LiDAR-based 3D Object Detection
Keyword: scaling
Breaking Time Invariance: Assorted-Time Normalization for RNNs
Bayesian Neural Network Versus Ex-Post Calibration For Prediction Uncertainty
Scaling transformation of the multimode nonlinear Schrödinger equation for physics-informed neural networks
Keyword: calibration
Bayesian Neural Network Versus Ex-Post Calibration For Prediction Uncertainty
Exploring Cross-Point Embeddings for 3D Dense Uncertainty Estimation
Proportional Multicalibration
Transfer Learning with Pretrained Remote Sensing Transformers