Training on Custom Dataset

[MahdiKoochali]

Can you provide a guideline for training on a custom dataset?

[m43]

Thank you for your interest in training on a custom dataset.

We utilize pre-trained checkpoints provided by the respective authors for all point trackers and SAM variants. If you're keen on training or fine-tuning the models on a custom point tracking or segmentation dataset, I'd advise delving into PIPS and SAM/HQ-SAM.

To give you a brief overview:

• PIPS is exclusively trained on the synthetic FlyingThings++, derived from the FlyingThings optical flow dataset.
• SAM was developed using the expansive SA-1B dataset.
• HQ-SAM training relied on HQSeg-44K.

I hope this helps guide your efforts!

[MahdiKoochali]

Thanks for the quick response.

Regarding your suggestion, in order to fine-tune the sam-pt model, I have to fine-tune both PIPS and SAM(or HQ-SAM) and at last load the obtained checkpoints from both models to run sam-pt, did I get your point?

[m43]

Correct, to fine-tune SAM-PT, you'd finetune PIPS or SAM or both

[MahdiKoochali]

Thanks for your answer.

Can you also explain which repository did you use to fine-tune the SAM?

[m43]

Hej @MahdiKoochali, we rely on the original checkpoints provided by the authors and do not perform additional fine-tuning. For SAM, you can download these checkpoints using the instructions provided here. Let us know if you need further assistance!

[MahdiKoochali]

Thank you so much!

I am trying to use SAM-PT on a custom dataset (in the microscopic image domain). Unfortunately, the results were not satisfying in this domain. Regarding your answer earlier, about fine-tuning, I decided to fine-tune SAM. I used micro-sam and fine-tuned it, but I got the attached error when I wanted to run SAM-PT with my custom checkpoint.

My suspicion is that this error might stem from an incompatibility issue with the SAM models.

It would be perfect if you help me to solve this error or help me to fine-tune SAM.

[m43]

Hej, micro-sam uses the same SAM architecture as the original SAM as it uses SAM directly (see here and here). Thus, it must be possible to load the state directly into an instance of SAM with sam_model.load_state_dict(your_model_state).

However, the training checkpoints in micro-sam might be saved in a peculiar way so that they cannot be directly loaded into SAM. What code have you used to save the checkpoint? You might want to try this short snippet out that could convert the micro-sam training checkpoint to a normal SAM checkpoint. I didn't have a training checkpoint to test it with, perhaps you could test if it works well on your side. The cleaned model_state should look something like odict_keys(['image_encoder.pos_embed', 'image_encoder.patch_embed.proj.weight', ... ]).

[m43]

On a side note, I'd also suggest you try out different point trackers from the repository for your data. For example, you can try using CoTracker instead of PIPs by adding model/point_tracker=cotracker model.positive_points_per_mask=16 to the command line arguments.

I'd also suggest you try varying the number of iterative refinement iterations to better tune the parameter to your fine-tuned SAM checkpoint. You can do so by adding and varying the model.iterative_refinement_iterations=3 command line argument. Cheers!