digantamisra98
commented
5 years ago Hi. Thanks for this comment and I would be happy to put forth my view in regards to the same. Firstly, I don' tag myself a researcher neither do I represent any big player in the research scene especially in the domain of Machine/ Deep Learning and I mostly indulge in experimenting and exploring Mathematics. I strongly agree that collaborative research where using all the best available parameters will result in the epitome of SOTA but you see, this domain has prompted big players to self induct themselves into a sort of competition where everyone is after getting max credits and to use a research outcome of another major competitor doesn't sometimes go well (This is what I think happens). I had a meeting with my colleagues couple of days ago and we are competing for the world leaderboards for the CIFAR 10 and CIFAR 100 classification as mentioned in Paperswithcode.com and we are planning to combine the best of everything (Ranger Optimizer(by Geoffrey Hinton) + Efficient Net(by Google) + CoordConv Layers(By Uber) + Mish activation). I believe collaborative research is the way forward. Indeed I agree sometimes even seemingly small factors like an activation function is overlooked because of the notion of knowledge sharing. Mostly all courses and resources talk about ReLU, Leaky ReLU and Tanh. Hardly anyone knows of activation functions like Swish, Mish and SQNL existing. There are many factors involved here but I'm glad someone could voice that and I'm trying my best to build something using the best of all worlds but then there are many limitations poised against me as I'm an individual with limited resources. But thank you for this. I hope my reply made sense.
Best Diganta
LifeIsStrange
glenn-jocher
Hi! This is not an issue per se and can be closed.
First of all, thank you for advancing progress in deep learning.
I'm just a random guy that want to implement an AGI (lol) and like many Nlp engeeners, I need HIGHLY accurate neural networks for fundamental NLP tasks (e.g POS tag, NER, dep parsing, Coref resolution, WSD, etc) They are all not very accurate (often sub 95% F1 score) and their errors add up.
Such limitations make Nlp not yet suitable for many things. This is why improving the state of the art (which can be observed on paperswithcode.com) is a crucial priority from academicians.
Effectively, many researchers have smart ideas to improve the state of the art and often slightly improve it by: Having a "standard neural network" for the task and mix with it their new fancy idea.
I talk from knowledge, I've read most papers from state of the art leaderboards from most fundamental NLP tasks. Almost always they have this common baseline + one idea, theirs. The common baseline sometimes slowly evolve (e.g now it's often a pre trained model (say BERT) + fine tuning + their idea.
Sorry to say, but "this" is to me retarded Where "this" mean the fact that by far, most researchers work in isolation, not integrating others ideas (or with such a slow inertia). I would have wished that state of the art in one Nlp task would be a combination of e.g 50 innovative and complementary ideas from researchers. You are researchers, do you have an idea why that is the case? If someone actually tried to merge all good complementary and compatible ideas, would they have the best, unmatchable state of the art? Why facebookresearch, Microsoft, Google don't try the low hanging fruit in addition to producing X new shiny ideas per month, actually try to merge them in a coherent, synergetic manner?? I would like you to tell me what you think of this major issue that slow AI progress.
As an example of such inertia let's talk about Swish, Mish or RAdam : Those things are incredibly easy to try and see "hey does it give to my neural network free accuracy gains?" Yet not any paper on state of the art leaderboards has tried Swish, Mish or RAdam despite being soo simple to try (you don't need to change the neural network) Not even pre trained models where so many papers depend on them (I opened issues for each of them).
Thank you for reading.