yil8
commented
6 years ago @yux94 Thanks for trying to reproduce my results. I actually feel what you got is already roughly the same as mine. First, I did not have any tricks for preprocessing/postprocessing. Everything is within the codebase. There are some important details when you try to sample the coordinates for training patches, but since I've already provided my sampled coordinates, it doesn't matter anyway. As for testing reproducibility, have you tried to use the ckpt I provided within the codebase to generate the probability map before you trained your own? If you use my ckpt, you should be able to get a probability map of Test_001.tif as this:
I would highly recommend using cmap=jet to plot the probability map as compared to black/white in your case, which does not quite differentiate numbers around 0.5
In [1]: import numpy as np
In [2]: from matplotlib import pyplot as plt
In [3]: probs_map = np.load('./Test_001.npy')
In [4]: plt.imshow(probs_map.transpose(), vmin=0, vmax=1, cmap='jet')
Out[4]: <matplotlib.image.AxesImage at 0x7f35e994ca58>
In [5]: plt.show()And I would subjectively argue this figure matches the ground truth annotation pretty well. If you could reproduce this probability map and the corresponding FROC score of ~0.8 (as already achieved by one user), then at least there should be no problems in the postprocessing steps.
yux94
And below is the prob map with my previous reproduced ckpt:
And this is your result:
That's very confusing since you have said that one user have already achieved good performance. And I am working on retraining the network again. Besides, it would be very nice if you could provide the detailed process of your sampling with hard mining. (
And this is my resampling training curve with 20 epoch.
Hukongtao
When I tried to reproduce your codes, there are less-than-perfect results. For example, below is the raw test_001 tiff
And after the whole training with the ResNet18-CRF, I got the test prob map result as:
while the ground-truth mask is something like:(since the camelyon16 organizers didn't provide the test GT with the format of tiff anymore, I transferred the raw tiff test file and xml file to the tiff mask with the ASAP software manually.)
And I have just followed your test steps and evaluated the average FROC score for the whole test set, and got this:
However, the result is not at all satisfying.
And is there any other trick in your preprocess, postprocess, or the training process?
Here is the prob map of test_026: