Ag-Net: building a customized deep neural network for recognizing crop categories based on spectral characteristics

[ZihengSun]

ESIP Member Organization

CSISS/LAITS, George Mason University Alaska Ocean Observing System (AOOS) and Axiom Data Science

Mentors

Ziheng Sun Jesse Lopez

Project Ideas

Ag-Net: building a customized deep neural network for recognizing crop categories based on spectral characteristics

Information for students

See ESIP general guidelines

Abstract

How many kinds of crops can you recognize? It is hard to say many. In most time of growing season, they are all green plants. Dent corn and sweet corn, black bean and red bean, barley and wheat, grass and weeds, etc. Distinguishing them takes a ton of knowledge and experiences. Agriculture scientists have struggled for years to figure out an automated way to recognize them. Deep learning is a powerful tool for non-linear classification problems. The critical part for deep learning is training dataset, which can be extracted from the reports and map products of U.S. department of agriculture. However, the existing deep neural networks are not performing as well as expected on crop classification because of their learned representation features in the back propagation are not common enough to tell the small differences among crops with similar external look. A customized network with special filters may help tell those minor differences in high spectral characteristics for more accurate recognition results.

Technical Details

Python; Keras; Geoweaver; numpy; scikit-learn; matplotlib; GDAL.

Helpful Experience

Machine learning knowledge; satellite image manipulation; python programming.

First steps

Start to get familiar with DeeplabV3, U-net or any other state-of-art deep neural network and test them on a sample training dataset.

[1998at]

@ZihengSun I tried combining the 2,3 and 4th band to form RGB image and i will be posting soon,it doesnt seem like an image from satellite.Yes I understand that RGB wont be enough and I would be leveraging all the channels.Could you also clarify my doubts regarding The Proposal as i mentioned above?

[ZihengSun]

You can include any idea that might improve the model in the proposal and state the reasons aside.

[1998at]

@ZihengSun This is what I am getting on stacking RGB channels Together. Here is the code to combine the channels which has been borrowed from https://gist.github.com/ckrehbiel/f5760a3d07aed91edc14a5e4ec03472f

def visualise3chnnels(file_name):
  file_name=file_name.split("_")
  file_name=file_name[0]+"_"+file_name[1]+"_"+file_name[2]+"_"+file_name[3]
  path=f'{im_input}/'+file_name
  b2_link = gdal.Open(path + '_2.tif') # blue band
  b3_link = gdal.Open(path + '_3.tif') # green band
  b4_link = gdal.Open(path + '_4.tif') # red band
  b2 = norm(b2_link.ReadAsArray().astype(np.float))
  b3 = norm(b3_link.ReadAsArray().astype(np.float))
  b4 = norm(b4_link.ReadAsArray().astype(np.float))
  rgb = np.dstack((b4,b3,b2))
  plt.imshow(rgb)
  del b2, b3, b4
  sm.toimage(rgb,cmin=np.percentile(rgb,2),
               cmax=np.percentile(rgb,98)).save("RGB/"+file_name+'.png')

The image doesn't seem right

[sankalpmittal1911-BitSian]

@ZihengSun I tried debugging my code. Softmax is performed correctly. All pixels over 255 classes sum to one. Data is being loaded correctly using the custom loader batchwise. I tried using DeepLab and even this model gets stuck on about 53% with validation accuracy poorer than U-Net. Now I am truly lost. Tried increasing the layers as well but the accuracy does not seem to improve. Changed metric to categorical_accuracy but again no improvement.

[sankalpmittal1911-BitSian]

@ZihengSun Finally some improvement. Took only training this time.

->Increased number of layers. ->Trained for less data first, saved the model and gradually increased the data to 1593. ->Keras has issues with softmax. That is why I created a custom layer using backend.

Epoch 69/200
375/375 [==============================] - 1214s 3s/step - loss: 0.6522 - categorical_accuracy: 0.7840
Epoch 70/200
375/375 [==============================] - 1324s 3s/step - loss: 0.6465 - categorical_accuracy: 0.7900
Epoch 71/200
375/375 [==============================] - 1152s 3s/step - loss: 0.6459 - categorical_accuracy: 0.7921
Epoch 72/200
375/375 [==============================] - 1223s 3s/step - loss: 0.6426 - categorical_accuracy: 0.7944
Epoch 73/200
375/375 [==============================] - 1198s 3s/step - loss: 0.6457 - categorical_accuracy: 0.7921
Epoch 74/200
375/375 [==============================] - 1201s 3s/step - loss: 0.6468 - categorical_accuracy: 0.7909

[ZihengSun]

The Landsat band value ranges are not 0 ~ 255. They are 16-bit numbers. Valid range is 0 ~ 10000. The creation of true-color image is not straightforward. Here is a guide how to create true-color Landsat image. My suggestion is rendering your results using the CDL color table first. That is easier.

[ZihengSun]

The training looks promising. You need check the validation accuracy at the same time to avoid overfitting. Early stop might also be required, e.g., stop if no improvement after 10 consecutive epochs.

[Swati707]

I am Swati Kanchan, a final year B. Tech undergraduate from NIT Durgapur, India. I have been working with Deep Learning in biomedical imaging for last one year and also published a work on Brain tumor classification using CNN, achieving the best accuracy, in proceedings of ICACCP'19. Recently, I along with my team won in Smart India Hackathon'19 where I developed two deep learning models and a user-friendly GUI application for Brain image segmentation and visualization.

I have started training on the sample training dataset without any kind of preprocessing provided by you on a slightly modified version of LinkNet for 13 epochs which is giving an accuracy of around 55%. Below is graph for accuracy and Loss.

I have also started training this dataset using the FCN8 model and it will be complete 40 epochs by night. I will keep you informed about the accuracy and send the graphs for the same as soon as possible.

[1998at]

@ZihengSun
Here is what I have been able to go to in the training after training for about 150 epochs.The validation accuracy is still around 70% So I am going to work on that.Increase Dropout and try various methods.It seems to be stuck in this region.In the first 50 epochs it was able to go from 53-80% but then it too about 50 more epochs to get above 85%.As you can see above 85 it struggles ,sometimes it goes up to 87 but then again the accuracy goes back to 85ish.Same goes for the loss.

[sankalpmittal1911-BitSian]

@ZihengSun I have kept dropout as 0.15 and I am getting 73% validation categorical accuracy. By the way the training log I have been posting here consisted of 1500 images out of 1593. Can you also please comment on the proposal (draft) I sent you on the email?

@at1998 Are you using colab? You are using model.fit or model.fit_generator ?

[1998at]

@sankalpmittal1911-BitSian I am not using Keras

[ZihengSun]

LinkNet seems interesting. Looking forward to see if it can produce the similar results as the other nets used here.

[ZihengSun]

@sankalpmittal1911-BitSian I will read it and get back to you as soon as I can.

Do you mean the left 93 images are used as validation datasets?

[ZihengSun]

@at1998 I suggest you could print the validation accuracy as well in the log. It seems the training accuracy is still rising but very slow. Maybe it is already overfitting.

[1998at]

@ZihengSun OKay I will do that and share it.I will also try adding some weight decay to try to stabilise it more and get back to you.

[sankalpmittal1911-BitSian]

@ZihengSun Yes, those 93 images are in validation set.

[1998at]

@ZihengSun Is there any kind Of Limit on How Lengthy the proposal Can be.I have almost finished my first draft of the proposal and its almost 8 pages Long.Do I need to Reduce it?

[ZihengSun]

There is no length limit for the whole proposal. However, the project abstract should be no more than ten sentences. Precisely describe your plan. It must be doable within the project time window.

[sankalpmittal1911-BitSian]

My draft is 13 pages. What should I modify in that? Can you please comment?

Thank you.

[ZihengSun]

The length is fine.

I have read your draft. It has all the required sections. That is very good. You might need more emphasize your idea and work plan. The background and challenge introduction should be very brief in abstract. The abstract must have no more than 10 sentences.

Customizing deep learning model is not equal to switching among deep learning models. Trying different models and comparing their results is very important. But more importantly, we need know why. Why does one model perform better than others? If the reasons are figured, we will know how to improve it to next level. You need be more specific about how you are going to customize models and why it will improve the accuracy, what is the expected ideal outcome? You need figure out which part of the model is hindering the learning and proposing a solution like adding a customized layer, apply customized loss function, change the input features, etc.

The deliverable part is too brief. Not sure what results will be generated and how they can be used by the community. Please be more specific and rational.

[sankalpmittal1911-BitSian]

Okay, I will get back with the updated draft as soon as I can. Thank you for the feedback.

[1998at]

@ZihengSun I have mailed you my draft for the proposal at zsun@gmu.edu.Can you please go through it and provide me the feedback.Thanks

[1998at]

@ZihengSun Can you please give me suggestions to improve the draft.Thanks

[ZihengSun]

@at1998 It reads really good. The plan is clear and doable. The network configuration is promising. My advice is adding some core equations to help explain how the model will converge well on training datasets with highly complex semantics. It is also better if the deliverables are explicitly itemized with expected outcome (e.g., source code, software release, web installation package, documentation, etc).

[1998at]

@ZihengSun Thank You for the Feedback. I can build up the equations and can explain why the model will converge faster however I didnt quite get what you meant in the latter part.Can you please give me an example.

[ZihengSun]

Like where the code will reside, how to manage the version of models, what software will be used to create the documentation, what license will be used for the source code and the generated data results respectively, etc. These information is basic but essential in a proposal.

[1998at]

@ZihengSun Okay I will do that.One more thing that i needed to discuss was that I have only recently begun with Open Source Projects I have done a lot of My own personal Projects and Have worked at companies as an Intern and took part in Various Deep Learning Competitions.But I havent contributed much to the the open source community.Will that pose a problem ?This is my first time with GSoC so I am having these doubts

[1998at]

@ZihengSun I will Edit the Proposal And get back to you by tomorrow

[ZihengSun]

Open source contributors' experiences will add credit to your proposal. No or less experiences are also fine, as long as you are willing to contribute to open source project in this summer.

[1998at]

@ZihengSun Thank You for the clarification

[1998at]

@ZihengSun I tried various methods for a UNet and after 85% it starts to overfit.If i train it for more the training accuracy goes up by 1-2% but then validation accuracy starts jumping between 76-78%.I am going to try FCN and observe if similar things happen.

[1998at]

@ZihengSun I am also working on the equations and updating my proposal and will be sending the next version in a day or two.Any Suggestions on the above problem ?

[1998at]

@ZihengSun I was thinking about writing a custom loss function and then multiplying it by some factor to penalise it even more. lets say loss=1.5 and no.of inaccurate predictions were 34.We can get 34 between a fixed range say[1,4] and multiply the loss by that.The more inaccurate predictions the more heavily the model is penalised and the more accurate predictions it makes the less.Would that work out

[sankalpmittal1911-BitSian]

@ZihengSun I noticed that the dataset is imbalanced, so I guess we could replace the loss with the weighted loss function. I am trying the same right now. I will also send updated proposal in 1-2 days.

They have derived it for MNIST dataset: https://github.com/keras-team/keras/issues/2115

I can try to derive it for this dataset. Also, I think we already have predefined function in tf.

[sankalpmittal1911-BitSian]

I am also going to try PSPNet or LinkNet because for UNet I cannot go beyond 80% training and 73% validation.

[ZihengSun]

If the network starts to overfit when validation accuracy is ~80%, I would say its capability cap is already reached. Customization is the only way to go beyond that line.

[ZihengSun]

The dataset is imbalanced because the land cover is naturally biased. The pixels of roads are much fewer than the pixels of crop fields. You could try class weight.

[esip-lab]

Hi all - a friendly reminder that there is ONE WEEK LEFT to submit your proposals for this project! Best of luck and we're excited to see what is submitted!

[sankalpmittal1911-BitSian]

@ZihengSun I know that very less time is left. I will send you updated proposal by today's end. Shall I also share that on the portal or wait for it?

[ZihengSun]

@abburgess Hi Annie, where should the proposals be submitted to?

[CaptainDredge]

@ZihengSun please check your email. I've sent something regarding proposal :smile:

[esip-lab]

Draft proposals should be submitted through the GSoC portal.

Annie Burgess, PhD

Lab Director | Earth Science Information Partners (ESIP)

esipfed.org/lab http://esipfed.org/lab | 585.738.7549

Sign up for the monthly ESIP Lab update here http://eepurl.com/dtKL8z.

On Thu, Apr 4, 2019 at 7:43 AM Ziheng Sun notifications@github.com wrote:

@abburgess https://github.com/abburgess Hi Annie, where should the proposals be submitted to?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESIPFed/gsoc/issues/13#issuecomment-479926811, or mute the thread https://github.com/notifications/unsubscribe-auth/AG_XwIfbcdCWA3cx0muteSOshLN2cM23ks5vdg-rgaJpZM4afBCr .

[sankalpmittal1911-BitSian]

@ZihengSun Please check your email for updated draft proposal. What can I do to improve it further?

[sankalpmittal1911-BitSian]

Also shall I share it with the organization?

[ghost]

Hi @ZihengSun ! I am very excited to contribute to this project. I know that it's way too late to get started, but I think that I can understand much in this one week. So, can you please provide me with the required details to get started. Thanks.

[sankalpmittal1911-BitSian]

@ZihengSun Also, I found a very interesting publication: https://www.mdpi.com/2072-4292/10/8/1217/pdf

It makes use of temporal dependencies, as in it uses GRUs to classify crops. Maybe I can make use of that. Right now I am trying to find segmentation models which can achieve accuracy greater than 80% and I am thinking of accomplishing this by the end of 6th May (Pre-GSoC).

[1998at]

@ZihengSun I have Sent you the Updated Proposal.Can you please go through it and please provide me with FeedBack before i upload it on the official GSOC Site.ThankYou

[1998at]

@ZihengSun I trained my model for about 100 epochs(takes about an hour ) and while my training loss started going down very slowly but my validation loss started Stabilising to 78%.I will add The Custom Layer and Try On some more Customisation Itself.Meanwhile Can you please help me on with the proposal .Thanks

[sankalpmittal1911-BitSian]

@ZihengSun I have already tried using custom lambda layer and it doesn't seem to make any difference. I have already used UNet. I will FPN/PSPNet/LinkNet and get back with the result as soon as I can.

Also please give feedback to the draft proposal.

[ZihengSun]

Hi @ZihengSun ! I am very excited to contribute to this project. I know that it's way too late to get started, but I think that I can understand much in this one week. So, can you please provide me with the required details to get started. Thanks.

Most information is available in this thread.