Consider conditional GANs

[schrum2]

Read here: https://machinelearningmastery.com/how-to-develop-a-conditional-generative-adversarial-network-from-scratch/

The idea is that each training example is associated with a class label so that during generation you can input the class label and get an output of the desired type. This could help with the UP, DOWN, HORIZONTAL issue in Mega Man #537 or in generating rooms of a specific type in Zelda.

However, this issue would require a deep understanding of how PyTorch works, and would require editing the Python code. I'm pretty sure we would want to keep the code we have, and make some brand new code that is similar but different.

[schrum2]

We may really need this in Mega Man to allow for distinct classes: Up, Horizontal, Down, and 4 corner cases

[schrum2]

This image seems to be key: This indicates that the discriminator and the generator have a one-hot encoded version of the class label as an input.

However, a later paragraph indicates that things are not actually as simple as this:

There are many ways to encode and incorporate the class labels into the discriminator and generator models. A best practice involves using an embedding layer followed by a fully connected layer with a linear activation that scales the embedding to the size of the image before concatenating it in the model as an additional channel or feature map.

[schrum2]

Would be good to look at examples in PyTorch. I've found some, but haven't looked at the code yet: https://github.com/malzantot/Pytorch-conditional-GANs https://github.com/eriklindernoren/PyTorch-GAN#conditional-gan https://github.com/znxlwm/pytorch-MNIST-CelebA-cGAN-cDCGAN

[schrum2]

I changed main.py in small ways to make using cdcgan possible. I was able to successfully train a test model, but I don't want it to be in the repo ... I'll paste it to this issue thread.

Modify generator_ws so that is can also use cdcgan if a parameter is set. Note that generator_ws does not use a fancy argument parser ... it just uses positional command line parameters. You should be able to load the model I trained, attached in a zip file here:

netG_epoch_21500_0_5.zip

[schrum2]

Here are some helpful notes for making sense of the code we already have and how we have to change it.

The models are trained in main.py. At line 232 the following happens:

errD_real = netD(inputv)
errD_real.backward(one)

Note that netD is a variable containing an instance of a discriminator network (CDCGAN_D is we use the conditional GAN). It may seem confusing that this class variable is being treated like a method call in netD(inputv), but note that this is an implicit call to the forward method of the discriminator: netD.forward(inputv). Afterward is an explicit call to backward, which is where learning occurs.

We need to change this to: netD.forward(inputv,labels). However, this isn't as simple as inputting the labels that we load from json. I've been comparing with code in this other repo: https://github.com/malzantot/Pytorch-conditional-GANs/blob/master/conditional_dcgan.py

There are some extra tricky steps for reorganizing the data to make sure the models are trained with the right combination of regular inputs and labels. One aspect of the problem is we are training in batches, so that is something to keep in mind. We can compare sizes of inputs with the shape property to see if the number of inputs and labels are the same. Anyway, in the code we are comparing with, this is the line that corresponds to our forward pass: output = model_d(inputv, Variable(one_hot_labels))

There is still more work to do to make sense of this though.

[schrum2]

I've been comparing the other conditional GAN code to each other. Here are the steps that seem to be common in the training of each. The files I'm comparing are: src1: https://github.com/znxlwm/pytorch-MNIST-CelebA-cGAN-cDCGAN/blob/master/pytorch_MNIST_cDCGAN.py src2: https://github.com/malzantot/Pytorch-conditional-GANs/blob/master/conditional_dcgan.py

Inside of the main loop, the following things happen: Step 1: Send real data and real labels to D (discriminator) to get the error src1:

            D_result = D(x_, y_fill_).squeeze()
            D_real_loss = BCE_loss(D_result, y_real_)

src2:

            output = model_d(inputv, Variable(one_hot_labels))
            optim_d.zero_grad()
            errD_real = criterion(output, labelv)
            errD_real.backward()

Step 2: Generate random labels and random noise for input to G (generator) src1:

        z_ = torch.randn((mini_batch, 100)).view(-1, 100, 1, 1)
        y_ = (torch.rand(mini_batch, 1) * 10).type(torch.LongTensor).squeeze()
        y_label_ = onehot[y_]
        y_fill_ = fill[y_]

src2:

            rand_y = torch.from_numpy(
                np.random.randint(0, NUM_LABELS, size=(batch_size,1))).cuda()
            one_hot_labels.scatter_(1, rand_y.view(batch_size,1), 1)
            noise.resize_(batch_size, args.nz).normal_(0,1)
            label.resize_(batch_size).fill_(fake_label)

Step 3: Send random labels and noise to G to get output src1:

        G_result = G(z_, y_label_)

src2:

            g_out = model_g(noisev, onehotv)

Step 3: Send output from G with the same random labels to D. Get error src1:

        D_result = D(G_result, y_fill_).squeeze()

        D_fake_loss = BCE_loss(D_result, y_fake_)
        D_fake_score = D_result.data.mean()

src2:

            output = model_d(g_out, onehotv)
            errD_fake = criterion(output, labelv)
            fakeD_mean = output.data.cpu().mean()

Step 4: Combine real and fake error to train D src1:

        D_train_loss = D_real_loss + D_fake_loss

        D_train_loss.backward()
        D_optimizer.step()

src2:

            errD = errD_real + errD_fake
            errD_fake.backward()
            optim_d.step()

Step 5: Generate new random noise and random labels and send to G src1:

        z_ = torch.randn((mini_batch, 100)).view(-1, 100, 1, 1)
        y_ = (torch.rand(mini_batch, 1) * 10).type(torch.LongTensor).squeeze()
        y_label_ = onehot[y_]
        y_fill_ = fill[y_]
        z_, y_label_, y_fill_ = Variable(z_.cuda()), Variable(y_label_.cuda()), Variable(y_fill_.cuda())

        G_result = G(z_, y_label_)

src2:

            noise.normal_(0,1)
            one_hot_labels.zero_()
            rand_y = torch.from_numpy(
                np.random.randint(0, NUM_LABELS, size=(batch_size,1))).cuda()
            one_hot_labels.scatter_(1, rand_y.view(batch_size,1), 1)
            label.resize_(batch_size).fill_(real_label)
            onehotv = Variable(one_hot_labels)
            noisev = Variable(noise)
            labelv = Variable(label)
            g_out = model_g(noisev, onehotv)

Step 6: Send output from G to D and compute error for G src1:

        D_result = D(G_result, y_fill_).squeeze()

        G_train_loss = BCE_loss(D_result, y_real_)

src2:

            output = model_d(g_out, onehotv)
            errG = criterion(output, labelv)

Step 7: Train G src1:

        G_train_loss.backward()
        G_optimizer.step()

src2:

            optim_g.zero_grad()
            errG.backward()
            optim_g.step()

The trick now is to figure out how this code corresponds to the code we already have, so we can modify what we have slightly, in order to accommodate this extra information. I wonder if our main.py should have a completely separate training loop though ... or maybe we even make an entirely different file for training the CDCGAN ... need to discuss more.

[schrum2]

I think the GAN models are ready for Conditional GAN now. This can be verified with the following test code:

import cdcgan as cg
import torch
gen = cg.CDCGAN_G(32,5,30,64,1,7)
classLabel = torch.FloatTensor(1,7,1,1).normal_(0,1)
input = torch.FloatTensor(1,5,1,1).normal_(0,1)
fake = gen(input,classLabel)
dis = cg.CDCGAN_D(32,5,30,64,1,7)
dis(fake,classLabel)

Basically, this code confirms two things. One is that the generator accepts and processes inputs of the correct sizes ... a latent vector (assuming a size of 5 in this example) and a vector the size of a one-hot encoded class label (assuming 7 classes in this example). The other thing the code confirms is that you can take the generator output and send it to the discriminator, along with the class label, and it will be fully processed as well.

The next thing to figure out is the changes required to the training procedure. That will be rough.

[schrum2]

It is possible to launch a previously saved CDCGAN model from generator_ws now. Use the following command on the pth model in the attached zip file. python generator_ws.py GENTEST.pth 5 30 16 14 7 Once this is running, first enter an int from 0 to 6 and press enter, then provide an array of 5 values from -1 to 1. Example:

READY
0
[0,0,0,0,0]
generator_ws.py:143: UserWarning: volatile was removed and now has no effect. Use `with torch.no_grad():` instead.
  levels = generator(Variable(latent_vector, volatile=True),Variable(classOneHot, volatile=True))
[[2, 16, 24, 16, 4, 29, 24, 16, 13, 16, 2, 1, 19, 17, 13, 28], [12, 14, 16, 1, 16, 28, 0, 20, 7, 14, 16, 26, 0, 1, 27, 14], [13, 12, 21, 17, 21, 22, 0, 10, 21, 9, 4, 16, 0, 17, 4, 1], [27, 20, 16, 1, 16, 1, 24, 26, 7, 14, 14, 1, 16, 20, 0, 9], [15, 1, 2, 17, 10, 13, 9, 23, 13, 1, 19, 17, 13, 23, 29, 8], [12, 26, 24, 14, 16, 1, 16, 26, 16, 8, 27, 20, 13, 16, 10, 20], [22, 17, 13, 23, 21, 8, 2, 23, 4, 23, 28, 16, 24, 1, 8, 13], [13, 14, 10, 20, 13, 1, 25, 20, 29, 23, 16, 20, 4, 14, 9, 9], [25, 10, 2, 13, 13, 13, 17, 3, 2, 8, 9, 28, 22, 13, 2, 8], [6, 1, 0, 14, 16, 1, 19, 20, 24, 14, 7, 14, 16, 18, 3, 9], [23, 25, 28, 17, 4, 9, 28, 1, 4, 1, 28, 10, 4, 1, 0, 27], [13, 28, 16, 1, 7, 1, 26, 26, 19, 1, 7, 26, 25, 14, 9, 20], [25, 8, 2, 9, 13, 1, 18, 16, 17, 1, 15, 9, 4, 4, 0, 17], [7, 22, 15, 14, 3, 1, 20, 14, 25, 3, 27, 20, 1, 22, 12, 20]]
2
[0,0,0,0,0]
[[4, 16, 27, 1, 18, 16, 5, 1, 2, 23, 22, 1, 10, 16, 19, 1], [14, 14, 16, 23, 1, 1, 12, 21, 0, 8, 13, 20, 16, 1, 0, 14], [6, 28, 7, 10, 6, 1, 10, 17, 21, 9, 25, 23, 28, 23, 4, 16], [12, 1, 16, 28, 12, 1, 0, 26, 7, 1, 13, 9, 7, 28, 0, 26], [6, 8, 13, 9, 10, 17, 5, 6, 20, 1, 28, 4, 10, 23, 0, 13], [9, 8, 13, 26, 12, 14, 19, 20, 16, 1, 24, 26, 1, 10, 19, 20], [13, 4, 11, 10, 4, 1, 28, 0, 4, 26, 22, 28, 4, 1, 28, 28], [13, 1, 14, 20, 7, 13, 27, 26, 2, 23, 9, 20, 8, 14, 6, 26], [13, 27, 17, 17, 21, 17, 4, 8, 13, 23, 17, 3, 21, 13, 2, 13], [16, 1, 0, 14, 16, 1, 13, 20, 23, 14, 12, 26, 1, 1, 29, 28], [14, 12, 28, 17, 19, 9, 3, 23, 4, 14, 25, 13, 4, 18, 25, 23], [29, 21, 16, 1, 4, 1, 23, 26, 19, 1, 24, 20, 24, 1, 26, 26], [1, 8, 28, 4, 13, 13, 0, 8, 17, 1, 9, 26, 8, 13, 13, 6], [1, 14, 24, 14, 16, 1, 19, 14, 16, 26, 13, 14, 19, 1, 19, 14]]
2
[0.1,0.1,0.1,0.1,0.1]
[[4, 16, 27, 17, 8, 23, 13, 6, 28, 17, 22, 1, 10, 17, 19, 1], [14, 14, 16, 23, 13, 26, 12, 20, 0, 8, 13, 20, 0, 1, 13, 14], [6, 28, 7, 10, 6, 1, 10, 17, 9, 9, 25, 23, 28, 26, 4, 16], [27, 20, 1, 28, 12, 1, 0, 26, 7, 1, 13, 9, 24, 14, 0, 26], [22, 8, 15, 14, 21, 17, 5, 6, 4, 1, 17, 4, 10, 16, 0, 13], [9, 22, 13, 26, 1, 1, 19, 20, 16, 10, 24, 20, 16, 1, 19, 20], [13, 23, 2, 28, 4, 17, 3, 0, 4, 25, 22, 23, 4, 28, 2, 25], [13, 20, 3, 20, 7, 13, 12, 26, 4, 1, 1, 20, 8, 14, 13, 26], [13, 4, 17, 23, 21, 9, 13, 8, 17, 24, 28, 1, 21, 26, 28, 3], [16, 1, 10, 14, 16, 13, 13, 26, 23, 14, 12, 8, 16, 7, 29, 20], [13, 8, 28, 17, 4, 14, 3, 23, 4, 18, 25, 23, 4, 9, 22, 17], [1, 1, 16, 16, 4, 1, 23, 26, 19, 8, 19, 26, 24, 14, 14, 23], [1, 23, 3, 23, 13, 13, 5, 4, 13, 9, 9, 23, 28, 13, 0, 6], [12, 9, 24, 14, 16, 18, 10, 20, 16, 26, 10, 20, 29, 1, 19, 14]]

GENTEST.zip

[schrum2]

Ugh, after putting in lots of work on this I finally got the conditional GAN to train, but the results are garbage. This is the command I used: python main.py --niter 5000 --nz 5 --json MegaManConditionalGANTrainingData.json --experiment CGANTests --tiles 12 --cuda --jsonID MegaManConditionalGANTrainingID.json --num_classes 7 But here is what the fake samples look like:

There are many possible causes for these bad results. It could be that the Conditional GAN simply takes longer to train (I guess that would be easy to test). It could be that our data is too imbalanced (percentage of data in each of the 7 categories is not evenly split). Or there simply might be an error in the code I wrote.

I suppose the next step is to try training for a much longer time, I also need to look back at earlier stages in training and see if it got better before getting worse. We could also try an easier training split first ... just 2 or 3 classes with an even data split instead of 7 uneven classes.

In any case, I don't think this will be resolved before the end of SCOPE.

[cappsb]

Horizontal: 1,462 screens Up: 518 screens Down: 364 screens Lower Left Corner: 7 screens Upper Left Corner: 10 screens Lower Right Corner: 9 screens Upper Right Corner: 8 screens

[schrum2]

The following training sets would be useful for troubleshooting:

• One training set that ONLY has the four corner classes
• One training set with ONLY two classes that are evenly split: 500 of the horizontal samples (doesn't matter which) and 500 of the Up samples

Each of these training sets needs both the level data and the class onehot vectors

[schrum2]

@cappsb Some fixes are needed for the json files that contain the class labels. In the case where there are only two cases (Up and Horizontal) the class label vectors should only have a length of 2: [1,0] for horizontal, [0,1] for up (or vice-versa .. it doesn't matter). The corner file needs to be similarly restricted. Basically, the length of each one-hot vector needs to equal the total number of classes (not exceed it)

[cappsb]

I fixed them so that the corner cases have a length of four and the up/horizontal a length of two (it's [1,0] for up, [0,1] for horizontal)

[schrum2]

Currently running the following command as a sanity check:

python main.py --niter 5000 --nz 5 --json cdcgan500UpAnd500HorizontalWith12Tiles.json --experiment CGANTests --tiles 12 --cuda --jsonID cdcgan500UpAnd500HorizontalWith12TileID.json --num_classes 2

We'll see if training succeeds with two balanced classes

[schrum2]

I think the test run above may have worked. Here are the fake samples, which seem reasonable: Testing more extensively would require some coding though. Also, in the long run, we're not interested in having a GAN with just these 2 classes ... we want 7.

Still, I guess the next step is to make some code on the Java side that can interact with a conditional GAN.

[schrum2]

FINAL.zip The model trained on two classes is attached. It can be launched with the command

python generator_ws.py FINAL.pth 5 12 16 14 2

After the console prints READY it expects two lines of input. The first line should be either 0 or 1. Then next is a list of 5 numbers, such as [0,0,0,0,0].

This produces outputs, but we need to hook this up to the Java side visualization code to see what it really looks like.

[schrum2]

Something that @cappsb and I looked at a bit was the idea of conditional GANs (CGANs). A CGAN allows you to split the data up into several classes, and tell it which class each training example belongs to, so that later when using the generator, you can provide a desired class along with your latent vector to produce a fake sample that looks like it comes from the desired class.

The work above pertains mainly to Mega Man, where we have 7 classes: horizontal, up, down, lower-left, lower-right, upper-left, upper-right. However, when I tried training a CGAN with these 7 classes, the corner sets proved too small to allow for effective training (at least, I believe that was the problem). Looking through the thread, I apparently trained a CGAN on just horizontal and vertical test classes, and that worked, but would be insufficient for our purposes.

So what do I want you to do @MBatt1 ? Try applying the conditional GAN to Mario. Split the data into overworld and underground levels (ignore the "athletic" levels) so that the GAN can be trained to serve an overworld or underground segment on demand.

However, play with the Mega Man stuff a bit before you dive into that. I seem to have provided reasonably good notes regarding commands you can run using the included files, so make sure that still works before trying to do something different. If we can get the CGAN to behave properly from the command prompt, then we can worry about interfacing with the Java code.