How large GPU memory can hold the entire network

[DavieHR]

I tried to train your complete network under the nvidia GTX 1080Ti with 12G GPU memory. But pytorch throw out a "out of cuda memory" error. I just wonder how large gpu memory i have i can train the so big network?

[zhangluustb]

I tried to train your complete network under the nvidia GTX 1080Ti with 12G GPU memory. But pytorch throw out a "out of cuda memory" error. I just wonder how large gpu memory i have i can train the so big network?

Default setting just need a little more than 11G, you can reduce the image_size to 224 or 128.

[DavieHR]

＠zhangluustb. hi. if i want to train the complete network to reproduce the paper result, what can i do? Split the network into several parts? Or buy a larger memory gpu : )

[zhangluustb]

＠zhangluustb. hi. if i want to train the complete network to reproduce the paper result, what can i do? Split the network into several parts? Or buy a larger memory gpu : )

In UGATIT.py:

1.
# Update D
set_requires_grad([self.genA2B, self.genB2A], False)
self.D_optim.zero_grad()
2.
fake_GA_logit, fake_GA_cam_logit, _ = self.disGA(fake_B2A.detach())
fake_LA_logit, fake_LA_cam_logit, _ = self.disLA(fake_B2A.detach())
fake_GB_logit, fake_GB_cam_logit, _ = self.disGB(fake_A2B.detach())
fake_LB_logit, fake_LB_cam_logit, _ = self.disLB(fake_A2B.detach())
# fake_GA_logit, fake_GA_cam_logit, _ = self.disGA(fake_B2A)
# fake_LA_logit, fake_LA_cam_logit, _ = self.disLA(fake_B2A)
# fake_GB_logit, fake_GB_cam_logit, _ = self.disGB(fake_A2B)
# fake_LB_logit, fake_LB_cam_logit, _ = self.disLB(fake_A2B)
3.
self.D_optim.step()
set_requires_grad([self.genA2B, self.genB2A], True)
# Update G
4.
self.G_optim.zero_grad()

# fake_A2B, fake_A2B_cam_logit, _ = self.genA2B(real_A)
# fake_B2A, fake_B2A_cam_logit, _ = self.genB2A(real_B)

And
def set_requires_grad(nets, requires_grad=False):
    """Set requies_grad=Fasle for all the networks to avoid unnecessary computations
    Parameters:
        nets (network list)   -- a list of networks
        requires_grad (bool)  -- whether the networks require gradients or not
    """
    if not isinstance(nets, list):
        nets = [nets]
    for net in nets:
        if net is not None:
            for param in net.parameters():
                param.requires_grad = requires_grad

So,u can use this to run 256*256 pixel model with a 1080ti. If I am wrong, please point out my mistake.Thanks.

[woshituobaye]

＠zhangluustb. hi. if i want to train the complete network to reproduce the paper result, what can i do? Split the network into several parts? Or buy a larger memory gpu : )

In UGATIT.py: 1.

Update D

set_requires_grad([self.genA2B, self.genB2A], False) self.D_optim.zero_grad() 2. fake_GA_logit, fake_GA_camlogit, = self.disGA(fake_B2A.detach()) fake_LA_logit, fake_LA_camlogit, = self.disLA(fake_B2A.detach()) fake_GB_logit, fake_GB_camlogit, = self.disGB(fake_A2B.detach()) fake_LB_logit, fake_LB_camlogit, = self.disLB(fake_A2B.detach())

fake_GA_logit, fake_GA_camlogit, = self.disGA(fake_B2A)

fake_LA_logit, fake_LA_camlogit, = self.disLA(fake_B2A)

fake_GB_logit, fake_GB_camlogit, = self.disGB(fake_A2B)

fake_LB_logit, fake_LB_camlogit, = self.disLB(fake_A2B)

3. self.D_optim.step() set_requires_grad([self.genA2B, self.genB2A], True)

Update G

4. self.G_optim.zero_grad()

fake_A2B, fake_A2B_camlogit, = self.genA2B(real_A)

fake_B2A, fake_B2A_camlogit, = self.genB2A(real_B)

And def set_requires_grad(nets, requires_grad=False): """Set requies_grad=Fasle for all the networks to avoid unnecessary computations Parameters: nets (network list) -- a list of networks requires_grad (bool) -- whether the networks require gradients or not """ if not isinstance(nets, list): nets = [nets] for net in nets: if net is not None: for param in net.parameters(): param.requires_grad = requires_grad

So,u can use this to run 256*256 pixel model with a 1080ti. If I am wrong, please point out my mistake.Thanks.

hi i use your code . The g_loss does not down , but up. so annotation it

set_requires_grad([self.genA2B, self.genB2A], False)

set_requires_grad([self.genA2B, self.genB2A], True)

again out of memory

[zhangluustb]

＠zhangluustb. hi. if i want to train the complete network to reproduce the paper result, what can i do? Split the network into several parts? Or buy a larger memory gpu : ) In UGATIT.py: 1.

Update D

set_requires_grad([self.genA2B, self.genB2A], False) self.D_optim.zero_grad() 2. fake_GA_logit, fake_GA_camlogit, = self.disGA(fake_B2A.detach()) fake_LA_logit, fake_LA_camlogit, = self.disLA(fake_B2A.detach()) fake_GB_logit, fake_GB_camlogit, = self.disGB(fake_A2B.detach()) fake_LB_logit, fake_LB_camlogit, = self.disLB(fake_A2B.detach())

fake_GA_logit, fake_GA_camlogit, = self.disGA(fake_B2A)

fake_LA_logit, fake_LA_camlogit, = self.disLA(fake_B2A)

fake_GB_logit, fake_GB_camlogit, = self.disGB(fake_A2B)

fake_LB_logit, fake_LB_camlogit, = self.disLB(fake_A2B)

self.D_optim.step() set_requires_grad([self.genA2B, self.genB2A], True)

Update G

self.G_optim.zero_grad()

fake_A2B, fake_A2B_camlogit, = self.genA2B(real_A)

fake_B2A, fake_B2A_camlogit, = self.genB2A(real_B)

And def set_requires_grad(nets, requires_grad=False): """Set requies_grad=Fasle for all the networks to avoid unnecessary computations Parameters: nets (network list) -- a list of networks requires_grad (bool) -- whether the networks require gradients or not """ if not isinstance(nets, list): nets = [nets] for net in nets: if net is not None: for param in net.parameters(): param.requires_grad = requires_grad So,u can use this to run 256*256 pixel model with a 1080ti. If I am wrong, please point out my mistake.Thanks.

hi i use your code . The g_loss does not down , but up. so annotation it

set_requires_grad([self.genA2B, self.genB2A], False)

set_requires_grad([self.genA2B, self.genB2A], True)

again out of memory

I also met this problem, cannot find a solution.

[scutlrr]

21

in UGATIT.py train_transform = transforms.Compose([ transforms.RandomHorizontalFlip(), transforms.Resize((self.img_size + 30, self.img_size+30)), transforms.RandomCrop(self.img_size), transforms.ToTensor(), transforms.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5)) ]) 1. Update D set_requires_grad([self.genA2B, self.genB2A], False) self.D_optim.zero_grad() # fake_A2B, _, _ = self.genA2B(real_A) # fake_B2A, _, _ = self.genB2A(real_B) fake_A2B, fake_A2B_cam_logit, _ = self.genA2B(real_A) fake_B2A, fake_B2A_cam_logit, _ = self.genB2A(real_B) 2. fake_GA_logit, fake_GA_cam_logit, _ = self.disGA(fake_B2A.detach()) fake_LA_logit, fake_LA_cam_logit, _ = self.disLA(fake_B2A.detach()) fake_GB_logit, fake_GB_cam_logit, _ = self.disGB(fake_A2B.detach()) fake_LB_logit, fake_LB_cam_logit, _ = self.disLB(fake_A2B.detach()) # fake_GA_logit, fake_GA_cam_logit, _ = self.disGA(fake_B2A) # fake_LA_logit, fake_LA_cam_logit, _ = self.disLA(fake_B2A) # fake_GB_logit, fake_GB_cam_logit, _ = self.disGB(fake_A2B) # fake_LB_logit, fake_LB_cam_logit, _ = self.disLB(fake_A2B) 3. self.D_optim.step() set_requires_grad([self.genA2B, self.genB2A], True) 4. Update G self.G_optim.zero_grad()

# fake_A2B, fake_A2B_cam_logit, _ = self.genA2B(real_A) # fake_B2A, fake_B2A_cam_logit, _ = self.genB2A(real_B)

And def set_requires_grad(nets, requires_grad=False): """Set requies_grad=Fasle for all the networks to avoid unnecessary computations Parameters: nets (network list) -- a list of networks requires_grad (bool) -- whether the networks require gradients or not """ if not isinstance(nets, list): nets = [nets] for net in nets: if net is not None: for param in net.parameters(): param.requires_grad = requires_grad

in main.py '--light', type=str2bool, default=True '--ch', type=int, default=32 '--img_size', type=int, default=128 in this way, you can run the model in your computer with gpu memory more than 3G

[Lornatang]