Closed wr0124 closed 1 month ago
[ ] step0 Get a similar architecture of UNet(ResBlock+AttentionBlock) in joliGEN comparable as in AnimateDiff(ResBlock+TransformerBlock+MotionModule)
[ ] step1 Modify ResBlock to process 5D tensor image for input and output
[ ] step1.1 test ResBlock to process 5D tensor for input and output
[ ] step1.2 ResBlock embedding ?
[ ] step2 Replace AttentionBlock by MotionModule
[ ] step2.1 Use QKVAttention class for attention score in MotionModule
[ ] step2.2 Use PositionalEncoding1D for position encoding ?
[ ] step2.3 test AttentionBlock by MotionModule for 5D tensor for input and output
[ ] step2.4 MotionModule embedding ?
[ ] step3 Merge MotionModule in the Video_generator_attn file
[ ] step3.1 Aligned attention head, input/output channels and maybe other parameters.
[ ] step3.2 test MotionModule in the Video_generator_attn file for 5D tensor input/output
[ ] step4 Test UNet for 5D tensor input/output ?
[ ] step5 Create Dataloader
[ ] step5 Test UNet with Dataloader
[ ] step0 Get a similar architecture of UNet(ResBlock+AttentionBlock) in joliGEN comparable as in AnimateDiff(ResBlock+TransformerBlock+MotionModule)
[ ] step1 Modify ResBlock to process 5D tensor image for input and output
[ ] step1.1 test ResBlock to process 5D tensor for input and output
[ ] step1.2 ResBlock embedding ?
[ ] step2 Replace AttentionBlock by MotionModule
[ ] step2.1 Use QKVAttention class for attention score in MotionModule
[ ] step2.2 Use PositionalEncoding1D for position encoding ?
[ ] step2.3 test AttentionBlock by MotionModule for 5D tensor for input and output
[ ] step2.4 MotionModule embedding ?
[ ] step3 Merge MotionModule in the Video_generator_attn file
[ ] step3.1 Aligned attention head, input/output channels and maybe other parameters.
[ ] step3.2 test MotionModule in the Video_generator_attn file for 5D tensor input/output
[ ] step4 Test UNet for 5D tensor input/output ?
[ ] step5 Create Dataloader
[ ] step5 Test UNet with Dataloader