X_train_256_v2.npy

[vamsi369]

Hi Greg, The X_train_256_v2.npy was implemented in all three classification files. But it was never created in the code that you provided. I only created X_train.npy file while preprocessing. By using that in cnn.py, the model is over fitting to the training data. The different implementation like categorical cnn_class.py and cnn_multi.py, but we never separated the images into 5 classes. We only did it for two classes. I am so confused Greg. Can you please help me. Thanks, Vamsi.

[gregwchase]

@vamsi369 X_train.npy will work; you'll just have to change the code.

Regarding the scripts: you'll want to use cnn_multi.py for all 5 classes, and cnn.py for binary classification.

[vamsi369]

Hi Greg, Thank you for replying. I tried to run X_train.npy with cnn.py for binary, the model is overfitting the training data(99% accuracy on train and 68% accuracy on test set). Even in the code you are using X_train_256_v2.npy, i thought it was by mistake and replace it by X_train.npy but i get overfitting model. And my concern regarding the categorical classification is, do we have a diffrent script which does the categorical classification? The eda.py scripts has function which assigns 0 for no DR and 1 for all other. In that(correct me if i am wrong Greg) case how can we use it for categorical classification? Is there any different script to get X_train_256_v2.npy? If yes, would you mind sharing that script. Thanks, Vamsi.

[gregwchase]

@vamsi369 Yes, the model is currently overfitting. I'm working to introduce regularization to prevent this.

For full categorical classification, you should omit the list comprehension in cnn.py, and replace it with the following:

y = np.array(labels.level, dtype=int)

[vamsi369]

oh is it. Thank you for confirming Greg. I am searching through the entire code and been tuning the hyper parameters and adding layers and doing more dropouts and thinking where did i go wrong. Are you trying to introduce L2 regularization in Conv2d layers Greg?

Thanks, Vamsi.

[vamsi369]

And one more thing Greg, so if we use y = np.array(labels.level, dtype=int) instead of the previous one, i think we should also change the labels file from trainLabels_master_256_v2.csv to trainLabels_master.csv. Is that correct Greg?

Thanks, Vamsi.

[gregwchase]

I might add them within the Dense layers, but only after adding more neurons, which may assist with overfitting before regularization is introduced.

The images and labels should go together (v2 images, v2 labels).

[vamsi369]

Oh Yes. I am sorry Greg. My bad, you are right. Alright, ill try to add some L2 regularization and also tinker with the number of neurons, and i will let you know if i get any better results. Thank you for posting this code Greg. It really helped me a lot. Thanks, Vamsi.

[vamsi369]

But i think there is no v2 images built in the code Greg. Did you? I only see X_train.npy in the entire code. No X_train_v2.npy is present.

On Sat, May 26, 2018 at 3:50 PM Greg Chase notifications@github.com wrote:

Closed #7 https://github.com/gregwchase/dsi-capstone/issues/7.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gregwchase/dsi-capstone/issues/7#event-1647990658, or mute the thread https://github.com/notifications/unsubscribe-auth/AT2sKyW-93dq-4THu5Xrwe2jQVZTcwIJks5t2bIdgaJpZM4UMsnm .

[gregwchase]

@vamsi369 Yes, you're correct. If you pass in X_train.npy with the original train labels, it should work.

[vamsi369]

Oh is it. If we pass in the original train labels instead of v2, is the model working greg?

Thanks, Vamsi.

On Sun, May 27, 2018 at 1:20 PM Greg Chase notifications@github.com wrote:

@vamsi369 https://github.com/vamsi369 Yes, you're correct. If you pass in X_train.npy with the original train labels, it should work.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gregwchase/dsi-capstone/issues/7#issuecomment-392348806, or mute the thread https://github.com/notifications/unsubscribe-auth/AT2sKwsbgVYNS8GLGr8OY2uN1v2be7Ktks5t2uBHgaJpZM4UMsnm .

[gregwchase]

Correction to my previous statement: Use X_train.py and trainLabels_master_256_v2.csv together. That should be the starting point for training the model. The way to verify this is looking at the shape of the array and training labels.

[vamsi369]

Yes, i have been trying that for the past four days Greg. I added more layers, introduced l2 regularizarion, changed the optimization method, decreased the soze of filters, everytime the test validation will ve at 0.734 like some kind of a migic number. I have even tried the aame with cnnmulti.npy and cnn class.npy but its doing the same Greg.

Thanks, Vamsi.

On Sun, May 27, 2018 at 1:44 PM Greg Chase notifications@github.com wrote:

Correction to my previous statement: Use X_train.py and trainLabels_master_256_v2.csv together. That should be the starting point for training the model. The way to verify this is looking at the shape of the array and training labels.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gregwchase/dsi-capstone/issues/7#issuecomment-392350314, or mute the thread https://github.com/notifications/unsubscribe-auth/AT2sK6f0UTaW8I1Py-OKNAEoaXoYUnfjks5t2uXxgaJpZM4UMsnm .

[gregwchase]

Something else you could try: apply transfer learning with a pre-trained network. The Keras documentation has tutorials on how to do this.

[vamsi369]

Yes, i will try that and let you know if i get any better results Greg.

Thanks, Vamsi.

On Sun, May 27, 2018 at 2:34 PM Greg Chase notifications@github.com wrote:

Something else you could try: apply transfer learning with a pre-trained network. The Keras documentation has tutorials on how to do this.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gregwchase/dsi-capstone/issues/7#issuecomment-392353339, or mute the thread https://github.com/notifications/unsubscribe-auth/AT2sK0venSllpHAk_IfhXew6DT8naMv5ks5t2vHSgaJpZM4UMsnm .

[echoaj]

Hi Greg. For cnn.py & cnn_multi.py is it suppose to be: X = np.load("../data/X_train.npy") on line 145 & 173?

Because I still get an error: File "/Users/alexjoslin/PycharmProjects/eye/src/cnn.py", line 146, in X = np.load("../data/X_train.npy") File "/Users/alexjoslin/venv/eye/lib/python3.7/site-packages/numpy/lib/npyio.py", line 422, in load fid = open(os_fspath(file), "rb") FileNotFoundError: [Errno 2] No such file or directory: '../data/X_train.npy'

[gregwchase]

@echoaj Yes, I believe it should be X_train.npy. The discrepancy is due to array versioning I was dealing with at the time.