Week 1. Mar. 22: Deep Learning? - Possibilities

[JunsolKim]

Pose a question about one of the following possibility readings:

“The unreasonable effectiveness of deep learning in artificial intelligence”. (2020). “Reducing the Dimensionality of Data with Neural Networks”, Hinton, G. E., Salakhutdinov, R. R., 2006. Science 313(5786):504-507. “A Unified Approach to Interpreting Model Predictions”. (2017). “Explainable artificial intelligence: a comprehensive review”. (2021). “Machine learning for pattern discovery in management research”. (2020).

[hantaoxiao]

In the study "Machine Learning for Pattern Discovery in Management Research," the author investigates a common classification task to predict employee turnover. My confusion arises during the interpretation phase, where the author presents feature importance from a tree-based model but fails to provide directional information. For example, it remains unclear whether longer tenure decreases or increases the likelihood of an employee leaving the company, as the model does not furnish this specific detail.

[anzhichen1999]

A response to “The unreasonable effectiveness of deep learning in artificial intelligence”. (2020).: Based on the textbook's emphasis on the role of deep learning in bridging supervised, unsupervised, and reinforcement learning, in what ways might the 'blessings of dimensionality' and the explorations of network architectures and learning algorithms contribute to advancing RLHF towards more efficient human-AI interaction and learning from limited or complex feedback scenarios?

[maddiehealy]

In Lundberg and Lee's work on A Unified Approach to Interpreting Model Predictions, Deep SHAP was mentioned briefly as a combination of DeepLIFT and SHAP values. I know that DeepLIFT can only be used when your input features are independent on one another, but I was wondering if this condition carries over into the Deep SHAP model? Essentially, can you use DeepSHAP models when your inputs are dependent upon one another? I anticipate the model may not work as well with dependent inputs, particularly when discerning the cause of certain outputs in interpretation stages. But is this even possible to conduct with Deep SHAP in this first place?

[Pei0504]

In the research“The unreasonable effectiveness of deep learning in artificial intelligence”, given the apparent paradox of deep learning networks functioning effectively despite theoretical expectations to the contrary, what novel insights might the geometry of high-dimensional spaces provide into the inner workings of these networks?

Also, we’re bridging the digital and real-world divide with deep learning, but the real world is messy. As deep learning bridges the gap between digital computing and analog real-world phenomena, how might future deep learning architectures evolve to better capture the analog, noisy, and uncertain nature of real-world data?

[HongzhangXie]

In the study "Machine Learning for Pattern Discovery in Management Research," I was impressed by the authors' comparison showing significant differences in performance between random forests and logistic regression when analyzing employee turnover rates. Traditional regression analysis relies on strong independence assumptions, yet in reality, the connections between data points are often complex. Designing an appropriate linear model is challenging without knowing the correct relationships between the data. Machine learning methods help us better discover the complex interconnections among data. The authors discussed that ML is less likely to be vulnerable to p-hacking due to the use of training and testing sets. However, in linear regression models, we can still use training and testing sets to check for overfitting. Beyond this, are there other ways ML methods can avoid p-hacking?

[guanhongliu2000]

“The unreasonable effectiveness of deep learning in artificial intelligence". (2020).

Given the success of stochastic gradient descent (SGD) in finding good solutions in nonconvex optimization problems encountered in deep learning, what theoretical explanations exist for the rare occurrence of local minima and the predominance of saddle points in high-dimensional spaces?

[jescib]

"The Unreasonable effectiveness of deep learning in artificial intelligence"

As we begin to implement these complex human brain practices into deep learning models, will we continue to increase the uncertainty and lack of interpretability within these models or will they make them easier to understand?

[La5zY]

after reading “A Unified Approach to Interpreting Model Predictions” “Considering the unified approach to interpreting model predictions presented by the SHAP framework, how might integrating or contrasting SHAP value attributions with other interpretation techniques like LIME or DeepLIFT influence the pursuit of more transparent and interpretable AI models? Specifically, could the unique properties of the SHAP method help address current limitations in interpretability for complex models such as deep neural networks?

[icarlous]

In "Reducing the Dimensionality of Data with Neural Network," I am interested in aligning the process of initialization and approximation with social science interpretation. For example, many social theories provide a simplified "ideal type" of sample distribution, like normative distribution or uniform distribution. This distribution can then be twisted and modified by all kinds of factors to adapt to "less idealized" social realities.

As more and more research in computational social science engages visualization of high dimensional data and variables and these dimension reduction methods (like tsne or pca or spectrum reduction) don't necessarily perform well, is it possible for us to "align" the progress of dimension reduction with the target we want to represent? For example, first scatter samples in the 2D space following uniform or gaussian distribution, and then "distort" this scattering with the result we have.

[Anmin-Yang]

"The Unreasonable effectiveness of deep learning in artificial intelligence"

The example of "bird, feather, and aerodynamics" is frequently used in both biological and artificial intelligence. It was the understanding of aerodynamics that eventually helps us build intercontinental planes. Is it the same for deep learning that to eventually achieve AGI, we need to know the principles of intelligence? However, it seems to me that there is little hope at least from the neuroscience side.

[shaangao]

"The Unreasonable effectiveness of deep learning in artificial intelligence"

Do deep learning models really work like human brains? There are for sure some similarities, like "neurons" (but the neurons in deep learning models are not the neurons in human brains) and layered structures. But the mainstream opinion on this questions right now seems to be "no". Deep learning models are more about finding statistical patterns in a large amount of data (which makes them super-human in this regard), but there are tasks that neural nets struggle at, especially those concerning structural generalization and reasoning. If neural nets and human brain rely on different mechanisms, is "artificial general intelligence" really possible?

[Yuxin-Ji]

In “A Unified Approach to Interpreting Model Predictions (2017)”, the authors proposed a novel method to interpret model predictions that advanced upon additive feature-based and Shapley-based methods. Knowing the tension between accuracy and interpretability, what are some pros and cons of the two sides? Is it possible to ever fully interpret a NN model? Sometimes, there could be different explanations for getting from the question to an answer, it could be that the machine figured out a smarter one that human could not yet understand (and for sure we need to study interpretability for ethical reasons as well). I found it interesting that one of the evaluation metrics is the method's consistency with human intuition. Many of evaluation tasks are comparing model performance with human performance. Sometimes, even human themselves could not fully understand themselves, and human brains are not fully interpretable yet. So I wonder what is the ultimate goal of the research on interpretability, if it might not be possible to achieve 100% interpretability? The same applies to accuracy, as many SOTA model already surpass human-level performance, yet for many complex tasks in real life, there is no hard boundary of what is "accurate" or "correct".

[atani21]

In "A Unified Approach to Model Predictions", the authors propose the SHAP to interpret predictions of complex models. It's not necessarily related to the article, but besides making predictions with new data input, what else can a complex model tell us? Also, rather than accuracy (from fitting the training and testing data), shouldn't we care more about the inferences we can make from the existent data?

[uc-diamon]

After reading “The unreasonable effectiveness of deep learning in artificial intelligence", is this new era of exploring high-dimensional spaces for deep learning feasible when it requires a ton of resources and computational power?

[joevilcai666]

Research Paper: "The unreasonable effectiveness of deep learning in artificial intelligence"

From the research, it mentions that"The complexity of learning and inference with fully parallel hardware is O(1). This means that the time it takes to process an input is independent of the size of the network. This is a rare conjunction of favorable computational properties." I'm still quite not get this part. How does the computational property of a parallel hardware impact the scalability and efficiency of a deep learning model?

[HamsterradYC]

“Can x2vec save lives? Integrating graph and language embeddings for automatic mental health classification”

This study conclusion mention that the modest performance gain of the ensemble model, despite its increased computational complexity and resource consumption, raises questions. When simply concatenating embeddings from different models might yield suboptimal outcomes, how could one effectively explore and construct ensemble deep learning models to discover more optimal solutions, balancing model performance with computational efficiency?

[kceeyang]

In "Reducing the Dimensionality of Data with Neural Networks," the author introduces a "pretraining" procedure that would use one single layer of learned binary features as data for learning a second layer of features to build an autoencoder. This method was used to solve the difficulty of initializing the weight of autoencoders. Since the paper was written in 2006, is it still a problem nowadays?

[risakogit]

The paper "The Unreasonable Effectiveness of Deep Learning in Artificial Intelligence" discusses self-supervised learning and imitation learning as promising techniques for handling large sets of unlabeled data. Since the paper was published in 2020, I am wondering if these methods are still considered state-of-the-art. Have other methods emerged, perhaps inspired by the way the human brain processes information and makes predictions?

[Marugannwg]

I find the first paper ("unreasonable effectiveness of deep learning") made a great connection between neurobiology and the structure of deep learning. With a psychology background, I found the storytelling very inspiring. (I wonder if it is a little hard to grasp without a neuroscience background?) Looks like the author is comparing the path of neural network development to a more and more comprehensive understanding of the human neural system --- The "unreasonable" in the title lies in the fact that neural networks performed so well despite the fact it hasn't covered many facets of the human mind. This reminds me of our professor James' idea in the textbook preface -- we probably should appreciate the inhuman aspect of the neural network (considering a more alien intelligence supplementing the human mind). I'm curious about the interaction here --- to what extent can AI become human-like? Or is it a sensible mindset to consider LLM (or other AI tool) a part of human capability?

[beilrz]

[The unreasonable effectiveness of deep learning in artificial intelligence (https://www.pnas.org/doi/pdf/10.1073/pnas.1907373117)

One question I have is to what extent we should model our artificial system, such as an artificial neural network, to emulate natural evolution. Because evolution happens through natural selection, it suggests that at any given stage of evolution, the organism must maintain an advantage to succeed in the selection, which prevents the organism from developing a complex, Interlocked system that may lack an advantageous middle stage.

[Xtzj2333]

"A Unified Approach to Interpreting Model Predictions" The 'additive feature' approach is fascinating. However, I wonder if simpler models could also sometimes give up some accuracy in order to gain more interpretability? This might introduce unwanted errors and biases - but just like most models in psychology and economics which also trade off accuracy for interpretability & practicality.

[erikaz1]

From my understanding, Hinton and Salakhutdinov (2006) compare autoencoders to standard and logistic PCA specifically but not TNSE or kernel PCA (non-linear dimension reduction techniques) because autoencoders are uniquely able to reconstruct high-dimensional input after dimension reduction. Are there techniques other than RBM that have been explored since for pretraining autoencoders?

[CYL24]

“The unreasonable effectiveness of deep learning in artificial intelligence,”

Chapter 2 also mentions that self-supervised learning provides a new perspective on unsupervised learning when large labeled datasets are unavailable. How effective can self-supervised learning and imitation learning mentioned in the article perform? And compared to traditional supervised learning methods?

[kangyic]

Machine learning for pattern discovery in management research.

The pipeline in the paper is very thorough and clear, and I especially like the partial dependence analysis. I wonder whether we can use the features that discovered to be useful to build a economics meaningful model for turnover rate evaluation.

[XueweiLi1027]

"The Unreasonable effectiveness of deep learning in artificial intelligence" If humans cannot make meaningful interpretation on the predictions generated by AI, is it really a good idea to make decisions based on these predictions?

[La5zY]

The paper talks about how deep learning models can have way more settings (parameters) than the examples they learn from, which normally would make them mess up by memorizing rather than actually learning. But somehow, they still manage to work really well and make accurate predictions on new stuff they haven't seen. How does that even happen without them just repeating what they've seen? What does this mean for how we think about making these models and teaching them with less info?

[Carolineyx]

Hinton and Salakhutdinov's work demonstrates the power of deep autoencoders for nonlinear dimensionality reduction, surpassing traditional methods like PCA. Given the complexity of training deep networks, what are the specific challenges associated with initializing the weights in deep autoencoders to avoid poor local minima? Additionally, how can these pretraining methods be adapted or improved to handle even larger and more diverse datasets while maintaining computational efficiency?

[00ikaros]

How do dopamine neurons contribute to reinforcement learning, and what impact does this have on the development of autonomous AI systems, as exemplified by AlphaGo's victory over the world champion Go player in 2017?

[icarlous]

In “A Unified Approach to Model Predictions,” the authors introduce SHAP for interpreting complex model predictions. Beyond making predictions with new data, what insights can complex models provide? Shouldn’t we prioritize the inferences drawn from existing data over accuracy from fitting the training and testing data?

[Brian-W00]

Despite the impressive performance of deep learning networks in solving real-world problems, why is our understanding of the reasons behind their effectiveness still limited? How does this phenomenon challenge traditional sample complexity and non-convex optimization theories?