storyboard-ai
AI Generated: The Story of Shepherd Boy And The Wolf
I am still under the process of cleaning and re-structuring the codebase
https://www.youtube.com/watch?v=iSb1HJXRO04
![https://www.youtube.com/watch?v=iSb1HJXRO04](https://img.youtube.com/vi/iSb1HJXRO04/0.jpg)
HOW THE VIDEO WAS MADE ?
- The story was taken from internet.
- A 4 line summary of the story was generated using ChatGPT
- For each summary line, a corresponding image was generated using Stable Diffusion
- In each image (4 in our case), the important object were masked using MetaAI SAM
- I have developed a custom image to white-board animation code which converted the images to videos (This was the most time-consuming part of the development process).
- The audio was generated using gTTS (Google Text-to-Speech)
- The sub-titles were generated with the help of OpenAI whisper
- All the different audios, videos and subtitles were somehow synchronized using FFMPEG and OpenCV (This part gave a lot of pain🥲... ALL HAIL FFMPEG🙌)
Code Structure:
- text-processing:
- story.txt: The story has been taken from internet
- summary.txt: The 4 line summary was generated using chatGPT (for the story present in story.txt)
- generate-whiteboard-animated-videos:
- Convert any image to white-board animations video
- If object masks are provided in JSON formats, they will be automatically considered while drawing
- Code File: draw-whiteboard-animations.py
- tts (text to speech):
- The story is devided into 4 parts in correspondance to the summary. For each story part, google's text-2-speec used to generate 4 mp3 files.
Sam assisted mask generation:
- The whiteboard animation is refined by drawing one object at a time.
- These object masks are drawn/created using Meta-AI SAM assisted tagging tool named: Anylabeling
- Anylabeling Tool: https://github.com/vietanhdev/anylabeling
- Post tagging, the tool returns a JSON output which can be used for sketching video generation
Please NOTE:
- The videos can also be generated using without the JSON masks