Training on a customized dataset

[alexlorenzo]

Hello,

We have strange results running MagicPoint on our own dataset (satellite imagery from SpaceNet).

The creation of keypoints using MagicPoint Synthetic datasets is working well:

However, when we run MagicPoint with our datasets and the previously created key points the model does not learn anything.

Any clue of what is happening ?

The same config was used for this training (batch size= 32, training iterations = 18 000, validation_size = 192)

Thank you in advance for your answer

[rpautrat]

Hi,

I have never trained with satellite images, but I think that several people already did it in the past with success. So it should be possible.

Could you provide more details about the training and validation losses/metrics during training?

[imcohen]

Hi,

The training(red)/valid(green) losses are well decreasing after 10 epochs of training. The training set is composed of around 8000 images (data augmentation is also enabled in the config), the valid set is around 2000 images. Images are of size 648 * 648.

However, we could not compute recall/precision since the running time of the following lines is very very long (few minutes) :

precision = tf.reduce_sum(pred * labels) / tf.reduce_sum(pred)
recall = tf.reduce_sum(pred * labels) / tf.reduce_sum(labels)

So, we just commented those lines and monitored only 'loss' (besides, is it a normal behaviour ? I find it very strange. We use Tensorflow 1.13)

[rpautrat]

2000 images for validation is a lot, this might explain why computing the metrics takes a long time. Even though these two operations should be very fast in theory, so this is a bit weird.

It would be interesting to re-enable these metrics (potentially at a reduced frequency) to see if it increased at least a bit during training.

Did you try training on another dataset like COCO, to check that you were doing everything correctly?

[triangleCZH]

Hi, I am also trying satellite images. I found that the SuperPoint COCO pretrain can already give strong results on point detection. Have you tried a similar experiment? Is there a reason why you want to start with MagicPoint again? If you wish, maybe we can discuss about this in the issue, or by email.

[imcohen]

Hi all,

First, thanks @rpautrat for your quick and helpful answers.

We have finally succeeded to train magic-point on our custom dataset, with relevant results (loss decreases and recall/precision increase). See an example of results on training set :

Two main problems have been identified :

• in the training pass, when we resize images, labels are not resized because they are supposed to have been generated with the same image resize. So, for labels generated without image resize, we've just modified the code like this (suppose images are squared, shape NxN pixels with N = original_size):

def _read_points(filename):
            return np.load(filename.decode('utf-8'))['points'].astype(np.float32) / original_size * new_size

@rpautrat : I think it should be good to mention this point somewhere in the doc.

• since images are TIFF format, we had to use Opencv or Scikit-image instead of Tensorflow to read them. Indeed, Tensorflow does not provide stable decoder for tiff (as for png or jpeg). Also, we had to normalize correctly uint8 and uint16 images. All these modifications in the code were not well implemented by our fault. A simple change in the order of operations finally improved the result.

@triangleCZH : yes, for the moment, we observed the same behaviour as you described. We try now to fine-tune magic-point on our own dataset, because we think intuitively that a custom training for the whole pipeline (magic-point and superpoint) on our dataset could improve performances.

[rpautrat]

Hi, great that you found these issues and managed to retrain MagicPoint!

Good point for the image resizing, I will update the doc accordingly.

[alexlorenzo]

@triangleCZH if you want to discuss please email me at alexandra.lorenzodebrionne at gmail.com

[triangleCZH]

@alexlorenzo Thank you for reaching out! I have sent an email to you. Looking forward to further discussion:)