Video & Video Processing

wonhyeongseo commented 9 months ago

Hello everyone,

Together with my collaborators, we've drafted a curriculum for our section. We're keen on getting feedback from the HF team during our writing. Here's what we've got so far:

Proposed Curriculum

INTRODUCTION (`introduction-to-video.mdx`)

[ ] What is a Video? – Delving into the Basic Concept (Assigned to @DiwakarBasnet, Estimated: by Nov 25th)
[ ] Role of Video Processing in AI and CV (Assigned to @wonhyeongseo, Estimated: by Nov 25th)

VIDEO PROCESSING BASICS (`video-processing-basics.mdx`)

[ ] Video vs. Image – Exploring Key Processing Differences (Assigned to @DiwakarBasnet, Estimated: by Nov 25th)
[ ] Temporal Continuity & Motion Estimation (Assigned to @DiwakarBasnet, Estimated: by Nov 25th)
[ ] Key Techniques: Stabilization, Background Subtraction, and Object Tracking

DEEP LEARNING FOR VIDEO (`deep-learning-for-video.mdx`)

[ ] Overview of Previous SOTA Models – Including 3D CNNs like (2+1)D Resnets and Two Stream Nets (Assigned to @sazio , Time: To be specified)
[ ] Transformers in Video Processing – Highlighting differences in ViT for Image vs. Video (Assigned to @DiwakarBasnet and @sazio , Time: To be specified)
[ ] Key Applications: Action Recognition and Video Captioning

PRACTICAL APPLICATIONS & CHALLENGES (`applications-and-challenges.mdx`)

[ ] Practical Applications: Surveillance, Streaming, Autonomous Driving, AR/VR, Healthcare, and Sports Analytics (Note: Might be too much; consider narrowing down for depth, Assigned to @sanae-a11y, Estimated: To be specified)
[ ] Challenges: Real-time Constraints & Privacy Concerns (Assigned to @wonhyeongseo, Estimated: by Nov 25th)

HANDS-ON DEMO (Jupyter Notebooks)

[ ] Simple Video Stabilization
[ ] Object Tracking
[ ] Action Recognition
[ ] Video Summarization

cc. @DiwakarBasnet @sazio @sanae-a11y @jungnerd bcc. @johko @lunarflu @merveenoyan

With our curriculum now finalized and our initial meeting behind us, we're eager to write the contents. Here are some things to keep in mind while writing:

Currently, we're leaning towards PyTorch as our primary library for the demos and internal scripts. Discussion on Github and related issue.

jungnerd commented 9 months ago

Thank you for drafting the curriculum! It's great 👍 Is Fundamentals of Video a necessary part of the curriculum for our course? Why not include a brief section in the Video vs. Image part?

DiwakarBasnet commented 9 months ago

Hey, I was thinking maybe we could start with what a video is rather than video vs image that way we could add the architectural differences between Vision transformers for images and videos in the intro to transformers for video processing section.

sazio commented 9 months ago

Thanks for this great outline ! I might have only one addition in the DL for videos, should we include also 3D CNN architectures ? Like (2+1)D Resnets or two streams nets? Just to have a reference with previous SOTA. Besides that I think that, with the previous comments, it's rather complete !

jungnerd commented 9 months ago

Oops, I accidentally unassigned myself, please reassign me. 🥲

wonhyeongseo commented 9 months ago

Hi everyone,

Firstly, I want to extend my sincere gratitude for the thoughtful feedback and insights shared on the draft curriculum. Each of your points has contributed significantly to refining the structure and content.

In response to the feedback:

Fundamentals of Video: @jungnerd The consensus seems to lean towards making the curriculum more streamlined. As suggested, I've integrated the fundamental aspects into the "Video vs. Image" section, focusing on the processing differences.
Introduction & Transformers: @DiwakarBasnet I've restructured the introduction to start with the basic concept of what a video is. This will allow a smooth transition into the differences between videos and images, and how they are processed. Additionally, the distinction in architectures for ViTs for images and videos has been incorporated in the transformers section.
Deep Learning Architectures: @sazio Including previous SOTA models like 3D CNNs is an excellent suggestion, and I've added that to provide a more comprehensive overview.

Next Steps:

Further Feedback: If there are any more insights or suggestions on this revised curriculum, please share them. @sanae-a11y
Content Creation: Once we finalize the curriculum, we can begin creating the content. It would be great to assign sections/topics based on expertise and interest.
Hands-on Demos: We'll need to gather datasets and tools for the demos. Any recommendations or resources in this area would be beneficial.
Timeline: It would be helpful to set up a tentative timeline for the completion of each section, ensuring we remain on track.

Again, thank you all for the valuable contributions. Together, we're shaping this chapter to be insightful and comprehensive for our audience.

Best, Won

sanae-a11y commented 9 months ago

Hi @wonhyeongseo, I wanted to express my gratitude for creating the curriculum! 👍 I'd like to take on the section about "PRACTICAL APPLICATIONS & CHALLENGES" in the Jupyter Notebook. I'm highly interested and eager to invest my time in developing a real-time computer vision application. Thank you!

sanae-a11y commented 9 months ago

Oops @wonhyeongseo , I accidentally removed the assignment you gave me . My bad! 😅 Could you please hook me up with that assignment again?

Sookeyy-12 commented 9 months ago

@sanae-a11y assigned you back.

sanae-a11y commented 9 months ago

thank you !

merveenoyan commented 9 months ago

Hello @wonhyeongseo I really liked how it goes from very basic concepts to specific. Another thing, we will use .mdx format with executable cells that you can open in Colab with one click, so instead of putting together a notebook, maybe let's stick with .mdx?

johko commented 9 months ago

hey @wonhyeongseo, I thought I already commented here, but apparently not :sweat_smile:

As someone who does actually not know that much about video I really like the outline and topics you want to cover and as Merve said, with our new separate folders for .mdx files and notebooks you might want to split things up a little differently.

But apart from that I'm really looking forward to your work :muscle:

wonhyeongseo commented 9 months ago

Hello @merveenoyan and @johko, I agree completely! Using mdx is much more natural.

Really love HF and its community 💯 This is my first time writing for a course, so kinda nervous but as a collective effort will try our best while respecting each other's time and contributions.

Thanks again for pushing this idea forward. Cheers to a comprehensive and successful course! 🥂

jungnerd commented 9 months ago

Though writing a course is no easy task, let's work together to make it work! 🔥

lunarflu commented 9 months ago

@wonhyeongseo @DiwakarBasnet @sazio I think I see the reasoning for this and it makes sense. What I would suggest is maybe to have some main core concepts that are most important (something like a fast-track through the course for the main ideas), and then have additional content that is still useful, but maybe labeled as "optional" or "bonus". Especially as we will probably be improving (fine-tuning 😉 ) the course over the long-term, my guess is everything probaby won't fit into a few days, but with this sort of "main content" "bonus content" distinction, maybe we can achieve both A) an optional short, condensed course and B) additional optional content to achieve "100%", or something like that.

What do you think?

jungnerd commented 9 months ago

Thanks for the suggestion @lunarflu 😍 I think it's a great idea! If we divide the course into "main content" and "bonus content", I think we can use our resources by focusing on the main content first and then writing the bonus content when we're done.

lunarflu commented 7 months ago

@wonhyeongseo @jungnerd @sazio @DiwakarBasnet @sanae-a11y Hey folks! How is this going?

I would aim to get something small and concrete merged by the end of the week if you think that's possible - It would be cool to have a v1 of the course by the end of the year (doesn't have to be perfect). WDYT? 🚀

jungnerd commented 6 months ago

@wonhyeongseo @sazio @DiwakarBasnet @sanae-a11y Hi folks! How's everyone doing?

I left a post on the discord thread and I don't think you guys saw it, so I'm reposting it here! I've been prepared for graduate school, so progress has been slow. How is everyone else's progress going? I think it would be great to have an online meeting with people who are available.

DiwakarBasnet commented 6 months ago

@wonhyeongseo @sazio @DiwakarBasnet @sanae-a11y Hi folks! How's everyone doing?

I left a post on the discord thread and I don't think you guys saw it, so I'm reposting it here! I've been prepared for graduate school, so progress has been slow. How is everyone else's progress going? I think it would be great to have an online meeting with people who are available.

Hey 👋, I am also currently busy because of my final exams. But I can attend meeting and get on work soon.

jungnerd commented 6 months ago

@wonhyeongseo @sazio @DiwakarBasnet @sanae-a11y Hi folks! How's everyone doing? I left a post on the discord thread and I don't think you guys saw it, so I'm reposting it here! I've been prepared for graduate school, so progress has been slow. How is everyone else's progress going? I think it would be great to have an online meeting with people who are available.

Hey 👋, I am also currently busy because of my final exams. But I can attend meeting and get on work soon.

Great! If the others don't reply, let's have a meeting between the two of us.

jungnerd commented 5 months ago

@DiwakarBasnet It looks like everyone else is busy. Maybe the two of us should have a meeting. When are you available for meetings?

DiwakarBasnet commented 5 months ago

@DiwakarBasnet It looks like everyone else is busy. Maybe the two of us should have a meeting. When are you available for meetings?

Today or tomorrow whenever you want

johko / computer-vision-course