No output from predict

[wboykinm]

Using this workflow - CPU-only, on a Macbook, I'm consistently producing no outputs here:

root@f5a70d76856f:/app# ./rs predict --batch_size 1 --checkpoint rutland/tmp/checkpoint-00001-of-00001.pth --tile_size 256 --model config/model-unet-rutland.toml --dataset config/dataset-building.toml rutland probs
Eval: 0batch [00:00, ?batch/s]

The 0batch eval bit seems like a pretty clear indicator that nothing's happening, but the question in my mind is "Just how fundamental of a config screwup is this on my part?" The train step produced some 436MB checkpoints, but the associated "history" chart looks . . . well . . . mostly dead:

Is this lack of output likely the result of

• [ ] How I configured train?
• [ ] How I configured predict?
• [ ] How I arranged the training/validation split?
• [ ] Something else altogether?

(🙏 for the awesome toolset and docs; mostly more helpful to the semiliterate like me than I have any right to expect)

[ocourtin]

Directly related to your question:

With predict command, the tile directory to pass is the one containing tiles images, so for instance rather than rutland you should use: rutland/validation/images
(or any other predict dedicated tiles directory)

Indirectly related to your question:

• One single epoch is not enough (at the very very least 10)
• You computed weight but don't update it's value in your dataset file
• You should give a try with Lovasz loss rather than CrossEntropy
• You should Increase the amount of data you provide to training (it's really a small dataset there)
• You should Increase the ration training / validation 70/30 will be more meaningful than 90/10

Enjoy :)

[wboykinm]

[wboykinm]

I've made the recommended adjustments and am running this again. As each training epoch takes ~1hr on this machine, it'll be a little while before I post results 😄

Thanks again!

[wboykinm]

ZING:

Thanks, @ocourtin!