lunarflu
commented
9 months ago Great job @mmhamdy ! 🤗 It's very thorugh, nice strong draft!🔥
Some small comments:
I think it'd be better to call it alternative rather than new.
IMO it's good to keep "New Architectures", assuming the focus is on the latest improvements, releases and ideas. For me, "Alternative" would be better if comparing older architectures among themselves.
On innovation: An introduction
I would do "On Innovation: An Introduction" (maybe good to emphasize capitals especially for titles)
Case study: ViT vs Image Transformer
I would do "Case Study: ViT vs. Image Transformer
Image Transformer is an early attempt by the authors of attention is all you need to introduce
I would do '...authors of "Attention Is All You Need" to introduce...' (we all probably have heard of the paper, but good to be consistent, maybe even link to it to keep things super clear especially if beginners are following along. In a way, maybe we could think of this as "an optional side exploration", people can click and supplement their knowledge very easily if they feel like it, condenses a lot more info into the course if we do more of this and if you think it sounds reasonable)
Why alternative Architectures
I would do "Why Alternative Architectures ❓ " (optional whether your team prefers emoji / normal question mark, emojis can stand out and inject little bits of fun here and there, while also conveying ideas fast and simply)
A hierarchical vision transformer that was introduced by Meta's FAIR.
Is there some blogpost by Meta on this maybe? If so, could be cool to link, same idea as above where people can do extra exploration if they want to, and a blogpost might have different details or explanations to help understanding.
The first AI model based on Yann LeCun’s vision for more human-like AI.
Could be cool to link to places where Yann lays out his vision, or if it exists, one really good / main one (same concept)
Retentive Network (RetNet) is a new architecture
I would link to RetNet (same concept)
that ambitiously aims to replace Transformer and
I would do "Transformers"
Overall up to you which changes to keep 🤗 TLDR; Since you mentioned you want to keep the chapter short, links might be a cool way to reference a lot of related content without taking up more screen space
mmhamdy
youssefadr
lulmer
farrosalferro
Hello👋 ,
This is a draft outline for the Trends and New Architectures chapter. I think it'd be better to call it alternative rather than new. Below I'll give a brief overview of the chapter content.
🔹 On Innovation: An Introduction
🔹 Case Study: ViT vs. Image Transformer
Image Transformer was an early attempt by the authors of Attention Is All You Need to introduce transformers into computer vision. It would be interesting to compare it to the now-established ViT (both models come from Google Brain).
🔹 Why Alternative Architectures?
🔹 Hiera
Paper: Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles A hierarchical vision transformer that was introduced by Meta's FAIR. It improves modern hierarchical vision transformers not by adding new components, but by questioning the several vision-specific components in these architectures. Meta wrote a twitter thread about it.
🔹 Hyena
Paper: Multi-Dimensional Hyena for Spatial Inductive Bias Initially introduced in NLP, the hyena layer offers a replacement for the transformer's self-attention with subquadratic complexity. A variant of it is the Multi-Dimensional Hyena, Hyena N-D layer, boosts the performance of various Vision Transformer architectures, such as ViT, Swin, and DeiT.
🔹 I-JEPA
Paper: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture The first AI model based on Yann LeCun’s vision for more human-like AI. Image-based Joint-Embedding Predictive Architecture (I-JEPA), is a non-generative approach for self-supervised learning from images. It delivers strong performance on multiple computer vision tasks, and it’s much more computationally efficient than other widely used computer vision models.
🔹 RMT
Paper: RMT: Retentive Networks Meet Vision Transformers Retentive Network (RetNet) is a new architecture that ambitiously aims to replace Transformers and to be the new foundation architecture for large language models. Inspired by RetNet, RMT is an attempt to further improve the retention mechanism into a 2D form and introduce it to visual tasks.
🔹 Trends And Research Directions
[ ] Open Vocabulary Learning: aims to enable models to recognize and classify objects in images and videos without being explicitly trained on those categories. This is in contrast to traditional computer vision approaches, which require a large dataset of labeled images for each category.
[ ] General foundation models for computer vision: In NLP, language models predicting the next token have proven to be a good foundational model that can be fine-tuned for various tasks. In computer vision, what model and loss objective has the potential to serve as the foundational model for different computer vision tasks tasks.
[ ] Domain-specific models: CV models that have been trained on a domain-specific dataset or task achieve higher accuracy and performance on that task than general computer vision models. This has many applications, for example, in medicine and healthcare.
🔹 Summary
This part offers a summary of the chapter and refers to other research work that was not mentioned in the chapter.
Notes:
Other Resources