search

Rudrabha / LipGAN

This repository contains the codes for LipGAN. LipGAN was published as a part of the paper titled "Towards Automatic Face-to-Face Translation".
http://cvit.iiit.ac.in/research/projects/cvit-projects/facetoface-translation
MIT License
578 stars 122 forks source link

Weird output for some videos #27

Closed aretius closed 4 years ago

aretius commented 4 years ago

Hey @prajwalkr , I have attached the video used for inferencing the LipGan models. I ran the following command

!python batch_inference.py --checkpoint_path logs/lipgan_residual_mel.h5 --model residual --pad 0 0 0 0 --face "/content/male.mp4" --fps 29.970 --audio "/content/male.wav" --results_dir "/content/"

Here are the videos used. I don't think FPS is the problem here, as the model seems to work for variable FPS as well. videos.zip

/content/LipGAN
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
Number of frames available for inference: 747
(80, 1874)
Length of mel chunks: 693
  0% 0/3 [00:00<?, ?it/s]
  0% 0/12 [00:00<?, ?it/s]
  8% 1/12 [00:02<00:25,  2.28s/it]
 17% 2/12 [00:02<00:13,  1.36s/it]
 25% 3/12 [00:03<00:09,  1.04s/it]
 33% 4/12 [00:03<00:07,  1.13it/s]
 42% 5/12 [00:03<00:05,  1.27it/s]
 50% 6/12 [00:04<00:04,  1.37it/s]
 58% 7/12 [00:04<00:03,  1.46it/s]
 67% 8/12 [00:05<00:02,  1.54it/s]
 75% 9/12 [00:05<00:01,  1.61it/s]
 83% 10/12 [00:06<00:01,  1.66it/s]
 92% 11/12 [00:06<00:00,  1.70it/s]
100% 12/12 [00:06<00:00,  1.78it/s]
Skipping 0
Skipping 1
Skipping 2
Skipping 3
Skipping 4
Skipping 5
Skipping 6
Skipping 7
Skipping 8
Skipping 9
Skipping 10
Skipping 11
Skipping 12
Skipping 13
Skipping 14
Skipping 15
Skipping 16
Skipping 17
Skipping 18
Skipping 19
Skipping 20
Skipping 21
Skipping 22
Skipping 23
Skipping 24
Skipping 25
Skipping 26
Skipping 27
Skipping 28
Skipping 29
Skipping 30
Skipping 31
Skipping 32
Skipping 33
Skipping 34
Skipping 35
Skipping 36
Skipping 37
Skipping 38
Skipping 39
Skipping 40
Skipping 41
Skipping 42
Skipping 43
Skipping 44
Skipping 45
Skipping 46
Skipping 47
Skipping 48
Skipping 49
Skipping 50
Skipping 51
Skipping 52
Skipping 53
Skipping 54
Skipping 55
Skipping 56
Skipping 57
Skipping 58
Skipping 59
Skipping 60
Skipping 61
Skipping 62
Skipping 63
Skipping 64
Skipping 65
Skipping 66
Skipping 67
Skipping 68
Skipping 69
Skipping 70
Skipping 71
Skipping 72
Skipping 73
Skipping 74
Skipping 75
Skipping 76
Skipping 77
Skipping 78
Skipping 79
Skipping 80
Skipping 81
Skipping 82
Skipping 83
Skipping 84
Skipping 85
Skipping 86
Skipping 87
Skipping 88
Skipping 89
Skipping 90
Skipping 91
Skipping 92
Skipping 93
Skipping 94
Skipping 95
Skipping 96
Skipping 97
Skipping 98
Skipping 99
Skipping 100
Skipping 101
Skipping 102
Skipping 103
Skipping 104
Skipping 105
Skipping 106
Skipping 107
Skipping 108
Skipping 109
Skipping 110
Skipping 111
Skipping 112
Skipping 113
Skipping 114
Skipping 115
Skipping 116
Skipping 117
Skipping 118
Skipping 119
Skipping 120
Skipping 121
Skipping 122
Skipping 123
Skipping 124
Skipping 125
Skipping 126
Skipping 127
Skipping 128
Skipping 129
Skipping 130
Skipping 131
Skipping 132
Skipping 133
Skipping 134
Skipping 135
Skipping 136
Skipping 137
Skipping 138
Skipping 139
Skipping 140
Skipping 141
Skipping 142
Skipping 143
Skipping 144
Skipping 145
Skipping 146
Skipping 147
Skipping 148
Skipping 149
Skipping 150
Skipping 151
Skipping 152
Skipping 153
Skipping 154
Skipping 155
Skipping 156
Skipping 157
Skipping 158
Skipping 159
Skipping 160
Skipping 161
Skipping 162
Skipping 163
Skipping 164
Skipping 165
Skipping 166
Skipping 167
Skipping 168
Skipping 169
Skipping 170
Skipping 171
Skipping 172
Skipping 173
Skipping 174
Skipping 175
Skipping 176
Skipping 177
Skipping 178
Skipping 179
Skipping 180
Skipping 181
Skipping 182
Skipping 183
Skipping 184
Skipping 185
Skipping 186
Skipping 187
Skipping 188
Skipping 189
Skipping 190
Skipping 191
Skipping 192
Skipping 193
Skipping 194
Skipping 195
Skipping 196
Skipping 197
Skipping 198
Skipping 199
Skipping 200
Skipping 201
Skipping 202
Skipping 203
Skipping 204
Skipping 205
Skipping 206
Skipping 207
Skipping 208
Skipping 209
Skipping 210
Skipping 211
Skipping 212
Skipping 213
Skipping 214
Skipping 215
Skipping 216
Skipping 217
Skipping 218
Skipping 219
Skipping 220
Skipping 221
Skipping 222
Skipping 223
Skipping 224
Skipping 225
Skipping 226
Skipping 227
Skipping 228
Skipping 229
Skipping 230
Skipping 231
Skipping 232
Skipping 233
Skipping 234
Skipping 235
Skipping 236
Skipping 237
Skipping 238
Skipping 239
Skipping 240
Skipping 241
Skipping 242
Skipping 243
Skipping 244
Skipping 245
Skipping 246
Skipping 247
Skipping 248
Skipping 249
Skipping 250
Skipping 251
Skipping 252
Skipping 253
Skipping 254
Skipping 255
Skipping 256
Skipping 257
Skipping 258
Skipping 259
Skipping 260
Skipping 261
Skipping 262
Skipping 263
Skipping 264
Skipping 265
Skipping 266
Skipping 267
Skipping 268
Skipping 269
Skipping 270
Skipping 271
Skipping 272
Skipping 273
Skipping 274
Skipping 275
Skipping 276
Skipping 277
Skipping 278
Skipping 279
Skipping 280
Skipping 281
Skipping 282
Skipping 283
Skipping 284
Skipping 285
Skipping 286
Skipping 287
Skipping 288
Skipping 289
Skipping 290
Skipping 291
Skipping 292
Skipping 293
Skipping 294
Skipping 295
Skipping 296
Skipping 297
Skipping 298
Skipping 299
Skipping 300
Skipping 301
Skipping 302
Skipping 303
Skipping 304
Skipping 305
Skipping 306
Skipping 307
Skipping 308
Skipping 309
Skipping 310
Skipping 311
Skipping 312
Skipping 313
Skipping 314
Skipping 315
Skipping 316
Skipping 317
Skipping 318
Skipping 319
Skipping 320
Skipping 321
Skipping 322
Skipping 323
Skipping 324
Skipping 325
Skipping 326
Skipping 327
Skipping 328
Skipping 329
Skipping 330
Skipping 331
Skipping 332
Skipping 333
Skipping 334
Skipping 335
Skipping 336
Skipping 337
Skipping 338
Skipping 339
Skipping 340
Skipping 341
Skipping 342
Skipping 343
Skipping 344
Skipping 345
Skipping 346
Skipping 347
Skipping 348
Skipping 349
Skipping 350
Skipping 351
Skipping 352
Skipping 353
Skipping 354
Skipping 355
Skipping 356
Skipping 357
Skipping 358
Skipping 359
Skipping 360
Skipping 361
Skipping 362
Skipping 363
Skipping 364
Skipping 365
Skipping 366
Skipping 367
Skipping 368
Skipping 369
Skipping 370
Skipping 371
Skipping 372
Skipping 373
Skipping 374
Skipping 375
Skipping 376
Skipping 377
Skipping 378
Skipping 379
Skipping 380
Skipping 381
Skipping 382
Skipping 383
Skipping 384
Skipping 385
Skipping 386
Skipping 387
Skipping 388
Skipping 389
Skipping 390
Skipping 391
Skipping 392
Skipping 393
Skipping 394
Skipping 395
Skipping 396
Skipping 397
Skipping 398
Skipping 399
Skipping 400
Skipping 401
Skipping 402
Skipping 403
Skipping 404
Skipping 405
Skipping 406
Skipping 407
Skipping 408
Skipping 409
Skipping 410
Skipping 411
Skipping 412
Skipping 413
Skipping 414
Skipping 415
Skipping 416
Skipping 417
Skipping 418
Skipping 419
Skipping 420
Skipping 421
Skipping 422
Skipping 423
Skipping 424
Skipping 425
Skipping 426
Skipping 427
Skipping 428
Skipping 429
Skipping 430
Skipping 431
Skipping 432
Skipping 433
Skipping 434
Skipping 435
Skipping 436
Skipping 437
Skipping 438
Skipping 439
Skipping 440
Skipping 441
Skipping 442
Skipping 443
Skipping 444
Skipping 445
Skipping 446
Skipping 447
Skipping 448
Skipping 449
Skipping 450
Skipping 451
Skipping 452
Skipping 453
Skipping 454
Skipping 455
Skipping 456
Skipping 457
Skipping 458
Skipping 459
Skipping 460
Skipping 461
Skipping 462
Skipping 463
Skipping 464
Skipping 465
Skipping 466
Skipping 467
Skipping 468
Skipping 469
Skipping 470
Skipping 471
Skipping 472
Skipping 473
Skipping 474
Skipping 475
Skipping 476
Skipping 477
Skipping 478
Skipping 479
Skipping 480
Skipping 481
Skipping 482
Skipping 483
Skipping 484
Skipping 485
Skipping 486
Skipping 487
Skipping 488
Skipping 489
Skipping 490
Skipping 491
Skipping 492
Skipping 493
Skipping 494
Skipping 495
Skipping 496
Skipping 497
Skipping 498
Skipping 499
Skipping 500
Skipping 501
Skipping 502
Skipping 503
Skipping 504
Skipping 505
Skipping 506
Skipping 507
Skipping 508
Skipping 509
Skipping 510
Skipping 511
Skipping 512
Skipping 513
Skipping 514
Skipping 515
Skipping 516
Skipping 517
Skipping 518
Skipping 519
Skipping 520
Skipping 521
Skipping 522
Skipping 523
Skipping 524
Skipping 525
Skipping 526
Skipping 527
Skipping 528
Skipping 529
Skipping 530
Skipping 531
Skipping 532
Skipping 533
Skipping 534
Skipping 535
Skipping 536
Skipping 537
Skipping 538
Skipping 539
Skipping 540
Skipping 541
Skipping 542
Skipping 543
Skipping 544
Skipping 545
Skipping 546
Skipping 547
Skipping 548
Skipping 549
Skipping 550
Skipping 551
Skipping 552
Skipping 553
Skipping 554
Skipping 555
Skipping 556
Skipping 557
Skipping 558
Skipping 559
Skipping 560
Skipping 561
Skipping 562
Skipping 563
Skipping 564
Skipping 565
Skipping 566
Skipping 567
Skipping 568
Skipping 569
Skipping 570
Skipping 571
Skipping 572
Skipping 573
Skipping 574
Skipping 575
Skipping 576
Skipping 577
Skipping 578
Skipping 579
Skipping 580
Skipping 581
Skipping 582
Skipping 583
Skipping 584
Skipping 585
Skipping 586
Skipping 587
Skipping 588
Skipping 589
Skipping 590
Skipping 591
Skipping 592
Skipping 593
Skipping 594
Skipping 595
Skipping 596
Skipping 597
Skipping 598
Skipping 599
Skipping 600
Skipping 601
Skipping 602
Skipping 603
Skipping 604
Skipping 605
Skipping 606
Skipping 607
Skipping 608
Skipping 609
Skipping 610
Skipping 611
Skipping 612
Skipping 613
Skipping 614
Skipping 615
Skipping 616
Skipping 617
Skipping 618
Skipping 619
Skipping 620
Skipping 621
Skipping 622
Skipping 623
Skipping 624
Skipping 625
Skipping 626
Skipping 627
Skipping 628
Skipping 629
Skipping 630
Skipping 631
Skipping 632
Skipping 633
Skipping 634
Skipping 635
Skipping 636
Skipping 637
Skipping 638
Skipping 639
Skipping 640
Skipping 641
Skipping 642
Skipping 643
Skipping 644
Skipping 645
Skipping 646
Skipping 647
Skipping 648
Skipping 649
Skipping 650
Skipping 651
Skipping 652
Skipping 653
Skipping 654
Skipping 655
Skipping 656
Skipping 657
Skipping 658
Skipping 659
Skipping 660
Skipping 661
Skipping 662
Skipping 663
Skipping 664
Skipping 665
Skipping 666
Skipping 667
Skipping 668
Skipping 669
Skipping 670
Skipping 671
Skipping 672
Skipping 673
Skipping 674
Skipping 675
Skipping 676
Skipping 677
Skipping 678
Skipping 679
Skipping 680
Skipping 681
Skipping 682
Skipping 683
Skipping 684
Skipping 685
Skipping 686
Skipping 687
Skipping 688
Skipping 689
Skipping 690
Skipping 691
Skipping 692

Traceback (most recent call last):
  File "batch_inference.py", line 217, in <module>
    main()
  File "batch_inference.py", line 210, in main
    out.release()
UnboundLocalError: local variable 'out' referenced before assignment
Rudrabha commented 4 years ago

This is because the face detector fails to detect any face in these frames. We have updated the message to be more clearer in both branches of the repo.