Training error in the conventional pipeline (not end2end)

aabramovrepo commented 5 years ago

Hi,

first of all, thanks for making this project public, some good work and ideas here! To my issue, I'm trying to run the model in the conventional mode (not end2end) with:

python3 -W ignore main.py --image_dir /mount/ssd_middle/TuSimple_subset/images/ --gt_dir /mount/ssd_middle/TuSimple_subset/ground_truth/ --end_to_end False --nclasses 4

but get the following error message:

0it [00:00, ?it/s]Batch with idx [2236 879 2282 2079 262 1271 2015 1018] skipped due to singular matrix The size of tensor a (4) must match the size of tensor b (8) at non-singleton dimension 0

commenting out the try / catch block I'm getting the following error:

RuntimeError: The size of tensor a (4) must match the size of tensor b (8) at non-singleton dimension 0

does anybody know what might be the reason for this? Is any quick fix possible? For the moment, I'm interested in getting segmentation maps for lanes without modeling.

Many thanks in advance!

Best, Alexey

wvangansbeke commented 5 years ago

Hey @aabramovrepo,

I appreciate your interest.

I see. Try not to use the fitting module in the beginning of training when using the cross-entropy loss. In the beginning, almost all pixels are assigned to the same class. This results in a singular matrix when solving the least squares problem. A quick fix would be to comment it out or try to use a flag for it.

Best, Wouter

aabramovrepo commented 5 years ago

Thanks for the quick response, very appreciated, I will try it out!

aabramovrepo commented 5 years ago

It seems to work, but I had to do the following fix in LSQ_layer.py forward():

if self.classification_branch:
    line = self.line_classification(shared_encoder)
    horizon = self.horizon_estimation(shared_encoder)

should appear in if not end_to_end mode as well.

Thanks again for helping me!

wvangansbeke / LaneDetection_End2End

Training error in the conventional pipeline (not end2end) #48