Repository for sign language transcription related models.
Ideally pose based models should use a shared large-pose-language-model, able to encode arbitrary pose sequence lengths, and pre-trained on non-autoregressive reconstruction.
pip install git+https://github.com/sign-language-processing/transcription
# Update conda
# Create environment
conda create -y --name sign python=3.10
conda activate sign
# Install all dependencies, may cause a segmentation fault
pip install .[dev]
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
Let's start with having a video file of a sign language sentence, word, or conversation.
curl https://media.spreadthesign.com/video/mp4/13/93875.mp4 --output sign.mp4
Next, we'll use video_to_pose
to extract the human pose from the video.
pip install mediapipe # depends on mediapipe
video_to_pose -i sign.mp4 --format mediapipe -o sign.pose
Now let's create an ELAN file with sign and sentence segments: (To demo this on a longer file, you can download a large pose file from here)
pip install pympi-ling # depends on pympi to create elan files
pose_to_segments -i sign.pose -o sign.eaf --video sign.mp4
Let's start with having a spoken language word, or sentence - "Hello World".
Finally, we'll animate the pose sequence into a video:
pip install git+https://github.com/sign-language-processing/pose-to-video
# Using Pix2Pix
wget -O pix2pix.h5 "https://firebasestorage.googleapis.com/v0/b/sign-mt-assets/o/models%2Fgenerator%2Fmodel.h5?alt=media"
pose_to_video --type=pix2pix --model=pix2pix.h5 --pose=sign.pose --video=sign.mp4 --upscale