assignment1-4

[JinwoongKim]

My first try but it doesn't show correct numbers

    ### Compute the forward pass
    H1 = X@W1+b1
    H2 = H1@W2+b2
    print(H2)

then I read this architecture LOL

    ### Compute the forward pass
    H1 = np.maximum(0,X@W1+b1)
    H2 = H1@W2+b2
    print(H2)

Now it seems right :)

[JinwoongKim]

Also, store it into scores !

    H1 = np.maximum(0,X@W1+b1)
    H2 = H1@W2+b2
    scores = H2

[JinwoongKim]

just copying softmax codes give me a lot of diff.

    loss = 0.0
    for i in xrange(N):
        scores -= np.max(scores)
        sum_esj = np.sum(np.exp(scores))
        loss += np.log(np.exp(scores[y[i]])/sum_esj)*-1
    print('loss : ',np.sum(loss))

[JinwoongKim]

Again, I forget to put reg T-T

and I divided scores int sub-score to fit my codes

   loss = 0.0
    for i in xrange(N):
        _scores = scores[i]
        _scores -= np.max(_scores)
        sum_esj = np.sum(np.exp(_scores))
        loss += np.log(np.exp(_scores[y[i]])/sum_esj)*-1
    loss /= N
    loss += reg * (np.sum(W1*W1)+np.sum(W2*W2))

[JinwoongKim]

room for bp

[JinwoongKim]

train,

      choices = np.random.choice(X.shape[0], batch_size)
      X_batch = X[choices]
      y_batch = y[choices]

      self.params['W2'] += -(learning_rate *  grads['W2'])
      self.params['b2'] += -(learning_rate *  grads['b2'])

      self.params['W1'] += -(learning_rate *  grads['W1'])
      self.params['b1'] += -(learning_rate *  grads['b1'])

predict

    pred_scores_1 = X@self.params['W1']
    pred_scores_2 = pred_scores_1@self.params['W2']
    y_pred = np.argmax(pred_scores_2, axis=1)

[JinwoongKim]

[JinwoongKim]

First shot iteration 0 / 1000: loss 2.302893 iteration 100 / 1000: loss 2.302890 iteration 200 / 1000: loss 2.302865 iteration 300 / 1000: loss 2.302876 iteration 400 / 1000: loss 2.302891 iteration 500 / 1000: loss 2.302882 iteration 600 / 1000: loss 2.302904 iteration 700 / 1000: loss 2.302893 iteration 800 / 1000: loss 2.302904 iteration 900 / 1000: loss 2.302883 Validation accuracy: 0.105 best_val : 0.105 iteration 0 / 1000: loss 2.302976 iteration 100 / 1000: loss 2.302967 iteration 200 / 1000: loss 2.302975 iteration 300 / 1000: loss 2.302978 iteration 400 / 1000: loss 2.302983 iteration 500 / 1000: loss 2.302985 iteration 600 / 1000: loss 2.302977 iteration 700 / 1000: loss 2.302962 iteration 800 / 1000: loss 2.302964 iteration 900 / 1000: loss 2.302994 Validation accuracy: 0.106 best_val : 0.106 iteration 0 / 1000: loss 2.303053 iteration 100 / 1000: loss 2.303053 iteration 200 / 1000: loss 2.303054 iteration 300 / 1000: loss 2.303059 iteration 400 / 1000: loss 2.303057 iteration 500 / 1000: loss 2.303057 iteration 600 / 1000: loss 2.303038 iteration 700 / 1000: loss 2.303065 iteration 800 / 1000: loss 2.303060 iteration 900 / 1000: loss 2.303059 Validation accuracy: 0.075 iteration 0 / 1000: loss 2.302880 iteration 100 / 1000: loss 2.302897 iteration 200 / 1000: loss 2.302881 iteration 300 / 1000: loss 2.302864 iteration 400 / 1000: loss 2.302880 iteration 500 / 1000: loss 2.302865 iteration 600 / 1000: loss 2.302879 iteration 700 / 1000: loss 2.302887 iteration 800 / 1000: loss 2.302879 iteration 900 / 1000: loss 2.302866 Validation accuracy: 0.113 best_val : 0.113 iteration 0 / 1000: loss 2.302964 iteration 100 / 1000: loss 2.302981 iteration 200 / 1000: loss 2.302961 iteration 300 / 1000: loss 2.302966 iteration 400 / 1000: loss 2.302963 iteration 500 / 1000: loss 2.302972 iteration 600 / 1000: loss 2.302952 iteration 700 / 1000: loss 2.302975 iteration 800 / 1000: loss 2.302957 iteration 900 / 1000: loss 2.302973 Validation accuracy: 0.097 iteration 0 / 1000: loss 2.303053 iteration 100 / 1000: loss 2.303031 iteration 200 / 1000: loss 2.303049 iteration 300 / 1000: loss 2.303057 iteration 400 / 1000: loss 2.303038 iteration 500 / 1000: loss 2.303055 iteration 600 / 1000: loss 2.303049 iteration 700 / 1000: loss 2.303050 iteration 800 / 1000: loss 2.303036 iteration 900 / 1000: loss 2.303032 Validation accuracy: 0.096 iteration 0 / 1000: loss 2.302878 iteration 100 / 1000: loss 2.302884 iteration 200 / 1000: loss 2.302865 iteration 300 / 1000: loss 2.302887 iteration 400 / 1000: loss 2.302867 iteration 500 / 1000: loss 2.302881 iteration 600 / 1000: loss 2.302873 iteration 700 / 1000: loss 2.302871 iteration 800 / 1000: loss 2.302871 iteration 900 / 1000: loss 2.302888 Validation accuracy: 0.126 best_val : 0.126 iteration 0 / 1000: loss 2.302966 iteration 100 / 1000: loss 2.302976 iteration 200 / 1000: loss 2.302974 iteration 300 / 1000: loss 2.302963 iteration 400 / 1000: loss 2.302966 iteration 500 / 1000: loss 2.302974 iteration 600 / 1000: loss 2.302972 iteration 700 / 1000: loss 2.302946 iteration 800 / 1000: loss 2.302970 iteration 900 / 1000: loss 2.302968 Validation accuracy: 0.112 iteration 0 / 1000: loss 2.303108 iteration 100 / 1000: loss 2.303049 iteration 200 / 1000: loss 2.303100 iteration 300 / 1000: loss 2.303071 iteration 400 / 1000: loss 2.303063 iteration 500 / 1000: loss 2.303084 iteration 600 / 1000: loss 2.303078 iteration 700 / 1000: loss 2.303065 iteration 800 / 1000: loss 2.303082 iteration 900 / 1000: loss 2.303077 Validation accuracy: 0.064 iteration 0 / 1000: loss 2.302870 iteration 100 / 1000: loss 2.302884 iteration 200 / 1000: loss 2.302895 iteration 300 / 1000: loss 2.302871 iteration 400 / 1000: loss 2.302914 iteration 500 / 1000: loss 2.302878 iteration 600 / 1000: loss 2.302892 iteration 700 / 1000: loss 2.302889 iteration 800 / 1000: loss 2.302896 iteration 900 / 1000: loss 2.302896

[JinwoongKim]

Validation accuracy: 0.106 best_val : 0.106 lr lrd reg : 1e-07 0.8 0.2 Validation accuracy: 0.075 Validation accuracy: 0.112 best_val : 0.112 lr lrd reg : 1e-07 0.8 0.3 Validation accuracy: 0.097 Validation accuracy: 0.096 Validation accuracy: 0.123 best_val : 0.123 lr lrd reg : 1e-07 0.85 0.3 Validation accuracy: 0.112 Validation accuracy: 0.064 Validation accuracy: 0.076 Validation accuracy: 0.117 Validation accuracy: 0.106 Validation accuracy: 0.073 Validation accuracy: 0.145 best_val : 0.145 lr lrd reg : 1e-06 0.8 0.2 Validation accuracy: 0.135 Validation accuracy: 0.11 Validation accuracy: 0.121 Validation accuracy: 0.092 Validation accuracy: 0.11 Validation accuracy: 0.131 Validation accuracy: 0.126 Validation accuracy: 0.116 Validation accuracy: 0.108 Validation accuracy: 0.085 Validation accuracy: 0.134 Validation accuracy: 0.211 best_val : 0.211 lr lrd reg : 1e-05 0.8 0.2 Validation accuracy: 0.231 best_val : 0.231 lr lrd reg : 1e-05 0.8 0.25 Validation accuracy: 0.18 Validation accuracy: 0.222 Validation accuracy: 0.19 Validation accuracy: 0.218 Validation accuracy: 0.2 Validation accuracy: 0.212 Validation accuracy: 0.23 Validation accuracy: 0.244 best_val : 0.244 lr lrd reg : 1e-05 0.95 0.2 Validation accuracy: 0.226 Validation accuracy: 0.195 Validation accuracy: 0.245 best_val : 0.245 lr lrd reg : 0.0001 0.8 0.2 Validation accuracy: 0.244 Validation accuracy: 0.247 best_val : 0.247 lr lrd reg : 0.0001 0.8 0.3 Validation accuracy: 0.259 best_val : 0.259 lr lrd reg : 0.0001 0.85 0.2 Validation accuracy: 0.254 Validation accuracy: 0.255 Validation accuracy: 0.269 best_val : 0.269 lr lrd reg : 0.0001 0.9 0.2 Validation accuracy: 0.266 Validation accuracy: 0.269 Validation accuracy: 0.278 best_val : 0.278 lr lrd reg : 0.0001 0.95 0.2 Validation accuracy: 0.275 Validation accuracy: 0.272 Validation accuracy: 0.36 best_val : 0.36 lr lrd reg : 0.001 0.8 0.2 Validation accuracy: 0.361 best_val : 0.361 lr lrd reg : 0.001 0.8 0.25 Validation accuracy: 0.371 best_val : 0.371 lr lrd reg : 0.001 0.8 0.3 Validation accuracy: 0.37 Validation accuracy: 0.361 Validation accuracy: 0.364 Validation accuracy: 0.36 Validation accuracy: 0.35 Validation accuracy: 0.36 Validation accuracy: 0.37 Validation accuracy: 0.367 Validation accuracy: /Users/jwkim/GitHub/cs231n/assignment1/cs231n/classifiers/neural_net.py:111: RuntimeWarning: divide by zero encountered in log loss = -np.sum(np.log(softmax[range(N),y])) /Users/jwkim/GitHub/cs231n/assignment1/cs231n/classifiers/neural_net.py:108: RuntimeWarning: overflow encountered in subtract scores -= stab.reshape(N,1) 0.378 best_val : 0.378 lr lrd reg : 0.001 0.95 0.3 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: /Users/jwkim/GitHub/cs231n/assignment1/cs231n/classifiers/neural_net.py:108: RuntimeWarning: invalid value encountered in subtract scores -= stab.reshape(N,1) /Users/jwkim/GitHub/cs231n/assignment1/cs231n/classifiers/neural_net.py:139: RuntimeWarning: invalid value encountered in less_equal dhidden[H1 <= 0] = 0 /Users/jwkim/GitHub/cs231n/assignment1/cs231n/classifiers/neural_net.py:115: RuntimeWarning: overflow encountered in multiply loss += regnp.sum(W2W2) 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.087 Validation accuracy: 0.176 Validation accuracy: 0.199 Validation accuracy: 0.183 Validation accuracy: 0.183 Validation accuracy: 0.196 Validation accuracy: 0.2 Validation accuracy: 0.209 Validation accuracy: 0.209 Validation accuracy: 0.215 Validation accuracy: 0.22 Validation accuracy: 0.216 Validation accuracy: 0.217 Validation accuracy: 0.131 Validation accuracy: 0.102 Validation accuracy: 0.074 Validation accuracy: 0.117 Validation accuracy: 0.14 Validation accuracy: 0.115 Validation accuracy: 0.099 Validation accuracy: 0.106 Validation accuracy: 0.109 Validation accuracy: 0.1 Validation accuracy: 0.101 Validation accuracy: 0.086

[JinwoongKim]

[JinwoongKim]