sheridana
commented
2 years ago Computing can sometimes become slow depending on the resolution of the data. We generally downsample in the gp node by a factor of 2 as it doesn't seem to affect accuracy too much. Also it is important to ensure that the sigma is a reasonable size for the resolution of your data. The second to last cell of this tutorial gives an idea for how to choose the sigma.
Some sigmas we have found to work well and the corresponding resolution (zyx, nanometers):
- cremi
- sigma:
80 - resolution:
40,4,4
- sigma:
- zebrafinch
- sigma:
120 - resolution:
20,9,9
- sigma:
- fib25/hemi
- sigma:
80 - resolution:
8,8,8
- sigma:
I also believe that @bentaculum was testing the gaussian filter on gpu using this implementation, but not sure how much of a speedup it gave.
Generally speaking, I found that training was pretty reasonable for 3d data using rtx 2080s, e.g 2-3 days to 400k iterations for non-autocontext nets. But this was on a cluster, so I can imagine this is too slow if you are trying to test multiple setups on a single gpu. Computing the lsds and storing before training would be tricky since it needs to be computed locally. I guess it mostly depends on your data and goals. We've found that the accuracy isn't affected too much on downsampled data which also speeds all other nodes up, so maybe something to consider (e.g 40,8,8 for cremi).
Thank you for providing the good work!
I'm implementing lsd using the code in this repository in my pytorch training pipeline. I found that computing lsd on cpu takes much more time than training (~10 times when I train on a device with 1 Titan X and 8 dataloader workers).
I just want to make sure if this is common since I didn't find the training time in the paper. If it's true, do you think there are proper ways to alleviate it, such as computing lsd on a cropped label(smaller than the model input size but bigger than output size), or computing lsd and storing before training?
Thanks!