search

lucidrains / imagen-pytorch

Implementation of Imagen, Google's Text-to-Image Neural Network, in Pytorch
MIT License
8.11k stars 768 forks source link

Inpainting with Videos (UNET3D) #338

Closed matvogel closed 1 year ago

matvogel commented 1 year ago

Is it possible to inpaint in videos? I tried to pass it just the inpaint images but let them be a video batch the same size as during training, however then the assertion also reminds me that

'video_frames must be passed in on sample time if training on video' AssertionError: video_frames must be passed in on sample time if training on video

lucidrains commented 1 year ago

@MrBirrd ah yea, that was on my todo

i hacked in support for single mask across frame video inpainting, but will eventually support an inpaint mask with a frame dimension greater than 1. the issue at the moment is with the temporal down and upsampling

i'll also have to rename the inpaint_images to be modality agnostic at some point

lucidrains commented 1 year ago

@MrBirrd should be all good now https://github.com/lucidrains/imagen-pytorch/commit/8be7a30760a37b090e85783ff4442db8b7582500