search

idsia-robotics / RandLA-Net-pytorch

Our PyTorch implementation of RandLA-Net (https://github.com/QingyongHu/RandLA-Net)
26 stars 5 forks source link

Some classes don't get trained at all for Hessigheim dataset #8

Open kuerbiskopf opened 1 year ago

kuerbiskopf commented 1 year ago

I'm adapting RandLA-Net for Hessigheim (https://ifpwww.ifp.uni-stuttgart.de/benchmark/hessigheim/default.aspx) and I am running into issues, where certain classes (roof, facade, soil, chimney) are not recognized at all and have an IoU of zero or NaN. Roof and facades are very large classes and it's not like they're difficult to recognize, so I don't really understand why this is. I tinkered with class weights in the data sampler as well as in the loss function to no avail. Changing learning rate also did not change much. Because Hessigheim is so large, I manually split the train and test dataset into small patches, each possibly not containing all eleven available classes. Could this be the issue? Does every patch need to contain every class?

Thank you very much.

Gabry993 commented 1 year ago

Hi, It's hard to give a precise answer, but here are my 2 cents: A NaN value for the IoU probably means that the class is not present in the patch considered. In general, I would avoid splitting the dataset into small patches, but rather into a few large ones. The random sampling proposed by randlanet's authors it's critical for its performance (in my experience) and works pretty well to ensure that all points/classes are taken into account in a balanced way. Having many small patches defeats its purpose in a way, since it's quite likely that the weight of a class in each small patch does not match the global weight across the entire dataset. Then, there are obvious aspects that get tricky when working with point clouds: for example, having 3 set (train, validation and test) which are representative of the data. In Hessigheim, simply splittining the dataset into 3 sets will give you 3 very different sets (e.g. many building vs rural areas)

kuerbiskopf commented 1 year ago

I tried running the testing script on the entire test area on Hessigheim at once - and it never finished, because it never achieved the "visited each point at least twice". I feared that during training the same might happen, that not all points are visited/trained. For that reason I split all the areas into smaller patches. But I will try again, thank you.