New submissions for Wed, 29 Jun 22

[ericbeyer]

Keyword: out of distribution detection

There is no result

Keyword: out-of-distribution detection

There is no result

Keyword: expected calibration error

There is no result

Keyword: overconfident

There is no result

Keyword: overconfidence

SLOVA: Uncertainty Estimation Using Single Label One-Vs-All Classifier

• Authors: Bartosz Wójcik, Jacek Grela, Marek Śmieja, Krzysztof Misztal, Jacek Tabor
• Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2206.13923
• Pdf link: https://arxiv.org/pdf/2206.13923
• Abstract Deep neural networks present impressive performance, yet they cannot reliably estimate their predictive confidence, limiting their applicability in high-risk domains. We show that applying a multi-label one-vs-all loss reveals classification ambiguity and reduces model overconfidence. The introduced SLOVA (Single Label One-Vs-All) model redefines typical one-vs-all predictive probabilities to a single label situation, where only one class is the correct answer. The proposed classifier is confident only if a single class has a high probability and other probabilities are negligible. Unlike the typical softmax function, SLOVA naturally detects out-of-distribution samples if the probabilities of all other classes are small. The model is additionally fine-tuned with exponential calibration, which allows us to precisely align the confidence score with model accuracy. We verify our approach on three tasks. First, we demonstrate that SLOVA is competitive with the state-of-the-art on in-distribution calibration. Second, the performance of SLOVA is robust under dataset shifts. Finally, our approach performs extremely well in the detection of out-of-distribution samples. Consequently, SLOVA is a tool that can be used in various applications where uncertainty modeling is required.

  Keyword: confidence

  Kwame for Science: An AI Teaching Assistant for Science Education in West Africa

• Authors: George Boateng, Samuel John, Andrew Glago, Samuel Boateng, Victor Kumbol
• Subjects: Computation and Language (cs.CL); Computers and Society (cs.CY); Human-Computer Interaction (cs.HC)
• Arxiv link: https://arxiv.org/abs/2206.13703
• Pdf link: https://arxiv.org/pdf/2206.13703
• Abstract Africa has a high student-to-teacher ratio which limits students' access to teachers. Consequently, students struggle to get answers to their questions. In this work, we extended Kwame, our previous AI teaching assistant, adapted it for science education, and deployed it as a web app. Kwame for Science answers questions of students based on the Integrated Science subject of the West African Senior Secondary Certificate Examination (WASSCE). Kwame for Science is a Sentence-BERT-based question-answering web app that displays 3 paragraphs as answers along with a confidence score in response to science questions. Additionally, it displays the top 5 related past exam questions and their answers in addition to the 3 paragraphs. Our preliminary evaluation of the Kwame for Science with a 2.5-week real-world deployment showed a top 3 accuracy of 87.5% (n=56) with 190 users across 11 countries. Kwame for Science will enable the delivery of scalable, cost-effective, and quality remote education to millions of people across Africa.

  SLOVA: Uncertainty Estimation Using Single Label One-Vs-All Classifier

• Authors: Bartosz Wójcik, Jacek Grela, Marek Śmieja, Krzysztof Misztal, Jacek Tabor
• Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2206.13923
• Pdf link: https://arxiv.org/pdf/2206.13923
• Abstract Deep neural networks present impressive performance, yet they cannot reliably estimate their predictive confidence, limiting their applicability in high-risk domains. We show that applying a multi-label one-vs-all loss reveals classification ambiguity and reduces model overconfidence. The introduced SLOVA (Single Label One-Vs-All) model redefines typical one-vs-all predictive probabilities to a single label situation, where only one class is the correct answer. The proposed classifier is confident only if a single class has a high probability and other probabilities are negligible. Unlike the typical softmax function, SLOVA naturally detects out-of-distribution samples if the probabilities of all other classes are small. The model is additionally fine-tuned with exponential calibration, which allows us to precisely align the confidence score with model accuracy. We verify our approach on three tasks. First, we demonstrate that SLOVA is competitive with the state-of-the-art on in-distribution calibration. Second, the performance of SLOVA is robust under dataset shifts. Finally, our approach performs extremely well in the detection of out-of-distribution samples. Consequently, SLOVA is a tool that can be used in various applications where uncertainty modeling is required.

  A Proposed Bi-LSTM Method to Fake News Detection

• Authors: Taminul Islam, MD Alamin Hosen, Akhi Mony, MD Touhid Hasan, Israt Jahan, Arindom Kundu
• Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2206.13982
• Pdf link: https://arxiv.org/pdf/2206.13982
• Abstract Recent years have seen an explosion in social media usage, allowing people to connect with others. Since the appearance of platforms such as Facebook and Twitter, such platforms influence how we speak, think, and behave. This problem negatively undermines confidence in content because of the existence of fake news. For instance, false news was a determining factor in influencing the outcome of the U.S. presidential election and other sites. Because this information is so harmful, it is essential to make sure we have the necessary tools to detect and resist it. We applied Bidirectional Long Short-Term Memory (Bi-LSTM) to determine if the news is false or real in order to showcase this study. A number of foreign websites and newspapers were used for data collection. After creating & running the model, the work achieved 84% model accuracy and 62.0 F1-macro scores with training data.

  Increasing Confidence in Adversarial Robustness Evaluations

• Authors: Roland S. Zimmermann, Wieland Brendel, Florian Tramer, Nicholas Carlini
• Subjects: Machine Learning (cs.LG); Cryptography and Security (cs.CR); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2206.13991
• Pdf link: https://arxiv.org/pdf/2206.13991
• Abstract Hundreds of defenses have been proposed to make deep neural networks robust against minimal (adversarial) input perturbations. However, only a handful of these defenses held up their claims because correctly evaluating robustness is extremely challenging: Weak attacks often fail to find adversarial examples even if they unknowingly exist, thereby making a vulnerable network look robust. In this paper, we propose a test to identify weak attacks, and thus weak defense evaluations. Our test slightly modifies a neural network to guarantee the existence of an adversarial example for every sample. Consequentially, any correct attack must succeed in breaking this modified network. For eleven out of thirteen previously-published defenses, the original evaluation of the defense fails our test, while stronger attacks that break these defenses pass it. We hope that attack unit tests - such as ours - will be a major component in future robustness evaluations and increase confidence in an empirical field that is currently riddled with skepticism.

  Towards Explainable Metaheuristic: Mining Surrogate Fitness Models for Importance of Variables

• Authors: Manjinder Singh, Alexander E.I. Brownlee, David Cairns
• Subjects: Neural and Evolutionary Computing (cs.NE); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2206.14135
• Pdf link: https://arxiv.org/pdf/2206.14135
• Abstract Metaheuristic search algorithms look for solutions that either maximise or minimise a set of objectives, such as cost or performance. However most real-world optimisation problems consist of nonlinear problems with complex constraints and conflicting objectives. The process by which a GA arrives at a solution remains largely unexplained to the end-user. A poorly understood solution will dent the confidence a user has in the arrived at solution. We propose that investigation of the variables that strongly influence solution quality and their relationship would be a step toward providing an explanation of the near-optimal solution presented by a metaheuristic. Through the use of four benchmark problems we use the population data generated by a Genetic Algorithm (GA) to train a surrogate model, and investigate the learning of the search space by the surrogate model. We compare what the surrogate has learned after being trained on population data generated after the first generation and contrast this with a surrogate model trained on the population data from all generations. We show that the surrogate model picks out key characteristics of the problem as it is trained on population data from each generation. Through mining the surrogate model we can build a picture of the learning process of a GA, and thus an explanation of the solution presented by the GA. The aim being to build trust and confidence in the end-user about the solution presented by the GA, and encourage adoption of the model.

  Keyword: scaling

  Heterogeneous mixtures of dictionary functions to approximate subspace invariance in Koopman operators

• Authors: Charles A. Johnson, Shara Balakrishnan, Enoch Yeung
• Subjects: Systems and Control (eess.SY); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2206.13585
• Pdf link: https://arxiv.org/pdf/2206.13585
• Abstract Koopman operators model nonlinear dynamics as a linear dynamic system acting on a nonlinear function as the state. This nonstandard state is often called a Koopman observable and is usually approximated numerically by a superposition of functions drawn from a \textit{dictionary}. A widely used algorithm, is \textit{Extended Dynamic Mode Decomposition}, where the dictionary functions are drawn from a fixed, homogeneous class of functions. Recently, deep learning combined with EDMD has been used to learn novel dictionary functions in an algorithm called deep dynamic mode decomposition (deepDMD). The learned representation both (1) accurately models and (2) scales well with the dimension of the original nonlinear system. In this paper we analyze the learned dictionaries from deepDMD and explore the theoretical basis for their strong performance. We discover a novel class of dictionary functions to approximate Koopman observables. Error analysis of these dictionary functions show they satisfy a property of subspace approximation, which we define as uniform finite approximate closure. We discover that structured mixing of heterogeneous dictionary functions drawn from different classes of nonlinear functions achieve the same accuracy and dimensional scaling as deepDMD. This mixed dictionary does so with an order of magnitude reduction in parameters, while maintaining geometric interpretability. Our results provide a hypothesis to explain the success of deep neural networks in learning numerical approximations to Koopman operators.

  Feature Refinement to Improve High Resolution Image Inpainting

• Authors: Prakhar Kulshreshtha, Brian Pugh, Salma Jiddi
• Subjects: Computer Vision and Pattern Recognition (cs.CV); Image and Video Processing (eess.IV)
• Arxiv link: https://arxiv.org/abs/2206.13644
• Pdf link: https://arxiv.org/pdf/2206.13644
• Abstract In this paper, we address the problem of degradation in inpainting quality of neural networks operating at high resolutions. Inpainting networks are often unable to generate globally coherent structures at resolutions higher than their training set. This is partially attributed to the receptive field remaining static, despite an increase in image resolution. Although downscaling the image prior to inpainting produces coherent structure, it inherently lacks detail present at higher resolutions. To get the best of both worlds, we optimize the intermediate featuremaps of a network by minimizing a multiscale consistency loss at inference. This runtime optimization improves the inpainting results and establishes a new state-of-the-art for high resolution inpainting. Code is available at: https://github.com/geomagical/lama-with-refiner/tree/refinement.

  DF-SCA: Dynamic Frequency Side Channel Attacks are Practical

• Authors: Debopriya Roy Dipta, Berk Gulmezoglu
• Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2206.13660
• Pdf link: https://arxiv.org/pdf/2206.13660
• Abstract The arm race between hardware security engineers and side-channel researchers has become more competitive with more sophisticated attacks and defenses in the last decade. While modern hardware features improve the system performance significantly, they may create new attack surfaces for malicious people to extract sensitive information about users without physical access to the victim device. Although many previously exploited hardware and OS features were patched by OS developers and chip vendors, any feature that is accessible from userspace applications can be exploited to perform software-based side-channel attacks. In this paper, we present DF-SCA, which is a software-based dynamic frequency side-channel attack on Linux and Android OS devices. We exploit unprivileged access to cpufreq interface that exposes real-time CPU core frequency values directly correlated with the system utilization, creating a reliable side-channel for attackers. We show that Dynamic Voltage and Frequency Scaling (DVFS) feature in modern systems can be utilized to perform website fingerprinting attacks for Google Chrome and Tor browsers on modern Intel, AMD, and ARM architectures. We further extend our analysis to a wide selection of scaling governors on Intel and AMD CPUs, verifying that all scaling governors provide enough information on the visited web page. Moreover, we extract properties of keystroke patterns on frequency readings, that leads to 95% accuracy to distinguish the keystrokes from other activities on Android phones. We leverage inter-keystroke timings of a user by training a k-th nearest neighbor model, which achieves 88% password recovery rate in the first guess on Bank of America application. Finally, we propose several countermeasures to mask the user activity to mitigate DF-SCA on Linux-based systems.

  Keyword: calibration

  SLOVA: Uncertainty Estimation Using Single Label One-Vs-All Classifier

• Authors: Bartosz Wójcik, Jacek Grela, Marek Śmieja, Krzysztof Misztal, Jacek Tabor
• Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2206.13923
• Pdf link: https://arxiv.org/pdf/2206.13923
• Abstract Deep neural networks present impressive performance, yet they cannot reliably estimate their predictive confidence, limiting their applicability in high-risk domains. We show that applying a multi-label one-vs-all loss reveals classification ambiguity and reduces model overconfidence. The introduced SLOVA (Single Label One-Vs-All) model redefines typical one-vs-all predictive probabilities to a single label situation, where only one class is the correct answer. The proposed classifier is confident only if a single class has a high probability and other probabilities are negligible. Unlike the typical softmax function, SLOVA naturally detects out-of-distribution samples if the probabilities of all other classes are small. The model is additionally fine-tuned with exponential calibration, which allows us to precisely align the confidence score with model accuracy. We verify our approach on three tasks. First, we demonstrate that SLOVA is competitive with the state-of-the-art on in-distribution calibration. Second, the performance of SLOVA is robust under dataset shifts. Finally, our approach performs extremely well in the detection of out-of-distribution samples. Consequently, SLOVA is a tool that can be used in various applications where uncertainty modeling is required.

  Visualizing and Alleviating the Effect of Radial Distortion on Camera Calibration Using Principal Lines

• Authors: Jen-Hui Chuang, Hsin-Yi Chen
• Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2206.14164
• Pdf link: https://arxiv.org/pdf/2206.14164
• Abstract Preparing appropriate images for camera calibration is crucial to obtain accurate results. In this paper, new suggestions for preparing such data to alleviate the adverse effect of radial distortion for a calibration procedure using principal lines are developed through the investigations of: (i) identifying directions of checkerboard movements in an image which will result in maximum (and minimum) influence on the calibration results, and (ii) inspecting symmetry and monotonicity of such effect in (i) using the above principal lines. Accordingly, it is suggested that the estimation of principal point should based on linearly independent pairs of nearly parallel principal lines, with a member in each pair corresponds to a near 180-degree rotation (in the image plane) of the other. Experimental results show that more robust and consistent calibration results for the foregoing estimation can actually be obtained, compared with the renowned algebraic methods which estimate distortion parameters explicitly.