Error in reproducing results for CVPPP to Komatsuna Leaf Counting

[shreystechtips]

Hi! I was currently trying to reproduce the results for adapting the CVPPP dataset to Komatsuna using the provided weights file (weights/CVPPP2Komatsuna.pth). However, whenever I input an image from the Komatsuna dataset into the CountEstimate model, the output is quite different than what is provided in the paper. Is there possibly a bug in the code for evaluating the datasets? Thank you!

[tedywond]

Hello, Which evaluation code did you use is it the one on a notebook? One more thing did you resize the images?

This is how we got the predictions from the model

actual = []
predict = []

transf = transforms.Compose([transforms.Resize((256,256)),transforms.ToTensor()])

for i in test_imgs:  
    count = test_lbls[i]
    out,_ = model(transf(Image.fromarray((test_imgs[i]*255).astype(np.uint8))).unsqueeze(0).type(torch.FloatTensor).cuda())

    actual.append(count)
    predict.append(int(out.sum()))

[shreystechtips]

Hi, I used the UDA4POC evaluation CropQuant.ipynb notebook, and resized all the images to (256, 256).

What is the the purpose of multiplying the test image by 255? The image plant image I load has value ranges from 0 to 255 already (only a resized image from the Komatsuna Dataset). Upon running the above provided code on one of the Komatsuna input images, I received a value of 22 (but there were only 2 leaves in that image). Also, how do I generate heatmaps similar to the one presented on page 13 of the paper with the CVPPP to Komatsuna domain adaptation?

Thank you!

[tedywond]

Oh, yea. The CropQuant evaluation notebook had a bug. I'll update it.

If you load the images with PyTorch's data loader and transform them with the transformation I provided in my previous comment, you would need to scale the pixel values that's why I multiplied it by 255.

Here is the code to generate the heatmaps.

import matplotlib.pyplot as plt

pred,_= model(x)
plt.imshow(x[index,:,:].permute(1,2,0).cpu().detach().numpy())
plt.imshow(pred.cpu().detach().numpy()[index,0,:,:],alpha=0.5,cmap='jet')

[shreystechtips]

I followed the steps outlined and produced the following heatmap:

Here is the snippet of code used to generate this (I did not multiply by 255 since i am loading directly from opencv)

counter_model = model.CountEstimate()
device = torch.device('cuda' if  torch.cuda.is_available() else 'cpu')
counter_model.load_state_dict(torch.load('./weights/CVPPP2Komatsuna.pth'))
counter_model.to(device)

url = "./komatsuna/multi_plant/rgb_00_00_000_03.png"
ar = cv2.imread(url, cv2.IMREAD_COLOR)

transf = transforms.Compose([transforms.Resize((256,256)),transforms.ToTensor()])
x = transf(Image.fromarray((ar).astype(np.uint8))).unsqueeze(0).type(torch.FloatTensor).cuda()
out,_ = counter_model(x)

plt.figure(figsize=(10,10))
plt.imshow(x[0,:,:].permute(1,2,0).cpu().detach().numpy())
plt.imshow(out.cpu().detach().numpy()[0,0,:,:],alpha=0.5,cmap='jet')

[tedywond]

transfors.ToTensor() changes the pixel value range from 0 to 1. See the documentation here https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.ToTensor

So you have to multiply the array values by 255

[shreystechtips]

Ah ok, makes sense. Upon running that, the results are still not what is expected:

Thank you!

[tedywond]

From what I can see from your code, the only issue I noticed is the model being in training mode. The model should be in evaluation mode since the batch norm layer behaves differently during training and test.

counter_model.eval()

[shreystechtips]

Hi,

That seems to have done it! Now the output looks as expected. One other quick question: When setting up the training data for CVPPP and resizing the leaf center images to (256,256), do we have to do any additional processing on those images, or can we just pass them in raw?

Thank you so much, I really appreciate your help!

[tedywond]

For the dot annotations, you have to recompute the leaf center positions to the new size (256,256). If you resize the density maps in a similar fashion as the input images, the pixel values will get interpolated and the total number of leaves will change. So I recommend you take the coordinates of the leaves from the annotation and remap them (via rescaling) to 256x256

[shreystechtips]

I also tried running the model with Komatsuna images that were similar to the one presented in the paper results, but the output look quite different. Is there something else I may need to do?

Thank you!

[tedywond]

I am not sure what issues you are experiencing but here is a figure generated using the code below taking a random image from the Komatsuna dataset. I used the pillow library (https://pillow.readthedocs.io/en/stable/) to load the image.

plt.xticks([])
plt.yticks([])
test_img = Image.open(kom_random[0])
plt.imshow(test_img.resize((256,256)))
y_hat,_ = model(transf(test_img).type(torch.float32).unsqueeze(0).cuda())
plt.imshow(y_hat[0,0].cpu().detach().numpy(),alpha=0.3,cmap='jet') 

[shreystechtips]

Hmm, I am getting very different results than that. Would you mind uploading the CVPPP2Komatsuna weights file you are using? Is it the same one in the repo?

[tedywond]

Try removing the multiplication by 255 if it's creating a problem. Here is a link to the weights file https://drive.google.com/file/d/15y8EVJqLzj0raU3p7ZYkvktS6Dq3aYI0/view?usp=sharing

[shreystechtips]

Thank you for providing that. It seems that the results are still not lining up. I uploaded the complete iPython Notebook I created to evaluate the model. They have both the methods that you suggested. https://gist.github.com/shreystechtips/1638dd05e0c1838813ffb4cca28e3960

[tedywond]

I looked at your iPython Notebook and the results look weird. I wonder if it has something to do with OpenCV. I have pasted a link to a Notebook I just created on colab where I generated some plots using the example image you chose and random ones I took from the Komatsuna dataset.

https://colab.research.google.com/drive/18Aug-xVIOddCPgCdkLkRENfJShxm7UPY?usp=sharing

[shreystechtips]

Hi,

I ran this notebook by uploading the weights file and the Komatsuna images in the notebook to Colab and running straight from there. However, the results are still not correct (see image). The only discrepancy that I can think of now is a different model.py file. I have uploaded the file that I am using (from the git repo) here: https://gist.github.com/shreystechtips/8dd4f81e967f9973bb53f66f39f2e056

Would you mind sharing the model.py file you are using?

This is the output of the notebook (with the same exact notebook code, weights, and input image)

Thank you!

[tedywond]

Sure. Here is the model.py file I am using https://drive.google.com/file/d/1dfNCRaOcA-nB4XRyEHqNksH6kCfVWdXb/view?usp=sharing

[shreystechtips]

This is really weird... The heatmap just doesn't output anything (running on Colab). Its just like the image I posted in the last post. For the Colab notebook you sent before, did you run it locally and upload, or run it directly on Colab servers? The results really seem promising and I want to see if I can build upon the work for other datasets, too.

[tedywond]

I originally run it on my local machine and upload it to colab. However, I just run it on colab and the results are the same as what I got on my local machine. I've included a snip of the output at the bottom.

It would be great if the method was tried on different datasets. I wonder if you could pickle the output tensor you are getting from the model and share it so that I could try to see what's going on.

[shreystechtips]

Hey! I got it to work. Looks like there is some difference in the model.py file in the github repository and the model.py file you sent above. I was still using old one (the import name was wrong) and the moment I used the new model.py sent above, everything started working.

Thank you so much for all the guidance! Excited to adapt it to new datasets.

[shreystechtips]

Hey, I was trying to train the model using CVPPP and wanted to ensure that the center points were correct. I used the rescaling methodology mentioned above to perform that. However, the results look a bit weird. I've uploaded the pickle file with each key corresponding to a list with the CVPPP image (cut out from the mask) and the center point dots.

The link is here: https://drive.google.com/file/d/1Qc_Cdtk06VklZrQBBJ50e1QTuybTncng/view?usp=sharing

Thank you!