Inquiry of parameter settings on the MulRan datset (OS1-64)

[LimHyungTae]

Yo bro, I saw the updated README.md; LGTM!

Btw, I'm writing this issue to ask you how to set the parameters when I run your Segregator on the MulRan dataset.

In fact, I'm curious about the generalization capability of Segregator, so I extracted .label files and tested your algorithm on the MulRan dataset; unfortunately, it just breaks down.

Could you test it if I give you labels and corresponding bin files?

[LimHyungTae]

MulRan_DCC01_example.zip I already fetched the corresponding pairs.

Of course, an algorithm could fail, yet correspondence estimation itself does not work :( Maybe if we set appropriate parameters, your algorithm could run well.

[LimHyungTae]

MulRan_DCC01_more_overlapped.zip Just in case, I uploaded easier scenes, whose position diff. is around 9 m and angle diff is 26 deg.

[Pamphlett]

Hey HyungTae,

Thanks for reaching out!

1. I ran Segregator using the data you provided above and found that the problem resides in the .label files. For both examples, I failed with the labels you provided but succeeded (at least to run the program =) with labels I freshly generated by SPVNAS. After A closer inspection of the labels you have, I found it's a bit weird. See the statistics below. where class 18 represents the traffic-sign but dominates the whole label. Did you use Salsanext for the prediction?
2. For the semantic segmentation network part, another mate from our lab (credits to @AronCao49) tune a bit with SPVNAS and got satisfying segmentation results,s and finally got Segregator to work. For Segregator part, I set use_veg here as true. I uploaded them here for your reference Mulran_example_spvnas.zip.
3. For Segregator, the algorithm itself, it is good that there are some instance-level objects (e.g., cars, tree trunks) to set up correspondences, in addition to the geometric (e.g., fpfh) ones. If not, Segregator will downgrade to a point feature-only method (just like a 6-DoF Quatro) and also suffer from degeneration cases (as mentioned in Quatro :)

[LimHyungTae]

Hey Pengyu, such a great job! How did you do that?

1. As far as I understand, you save the raw labels, not the inversed mapped one. Right? I'll check the problem in my test thoroughly (in fact, my codes work well in KITTI, but in MulRan, it showed poor instance results.)
2. Could you share that tip for fine tunning? How did you tune SPVNAS? It's great if we can fine-tune semantic segmentation pipeline, then we can generalize Segregator.
3. Totally true. I had to meet you in person at ICRA, haha. I really like your research! XD Thanks again for your sharing and contribution

[LimHyungTae]

Here's my full code of evaluate.py in SPVNAS.

# import os
# os.environ["PL_TORCH_DISTRIBUTED_BACKEND"] = "gloo"
import os
import argparse
import sys

import numpy as np
import torch
import torch.backends.cudnn
import torch.cuda
import torch.nn
import torch.utils.data

import ssl
ssl._create_default_https_context = ssl._create_unverified_context

from torchpack import distributed as dist
from torchpack.callbacks import Callbacks, SaverRestore
from torchpack.environ import auto_set_run_dir, set_run_dir
from torchpack.utils.config import configs
from torchpack.utils.logging import logger
from tqdm import tqdm

from core import builder
from core.callbacks import MeanIoU
from core.trainers import SemanticKITTITrainer
from model_zoo import minkunet, spvcnn, spvnas_specialized

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument('config', metavar='FILE', help='config file')
    parser.add_argument('--run-dir', metavar='DIR', help='run directory')
    parser.add_argument('--name', type=str, help='model name')
    parser.add_argument('--save-dir', type=str, help='save directory of `.label` files')
    args, opts = parser.parse_known_args()

    # for i in range(11):
    #     directory_name = f"{i:02d}"  # format the number as a two-digit string
    for directory_name in ["DCC01", "DCC02", "DCC03", "KAIST02", "KAIST03", "Riverside01", "Riverside02", "Riverside03"]:
        if not os.path.exists(args.save_dir + "/" + directory_name):
            os.makedirs(args.save_dir + "/" + directory_name)

    configs.load(args.config, recursive=True)
    configs.update(opts)

    if configs.distributed:
       dist.init()

       torch.backends.cudnn.benchmark = True
       torch.cuda.set_device(dist.local_rank())

    if args.run_dir is None:
        args.run_dir = auto_set_run_dir()
    else:
        set_run_dir(args.run_dir)

    logger.info(' '.join([sys.executable] + sys.argv))
    logger.info(f'Experiment started: "{args.run_dir}".' + '\n' + f'{configs}')

    dataset = builder.make_dataset()
    dataflow = {}
    for split in dataset:
        sampler = torch.utils.data.distributed.DistributedSampler(
            dataset[split],
            num_replicas=dist.size(),
            rank=dist.rank(),
            shuffle=(split == 'train'))
        dataflow[split] = torch.utils.data.DataLoader(
            dataset[split],
            batch_size=configs.batch_size if split == 'train' else 1,
            sampler=sampler,
            num_workers=configs.workers_per_gpu,
            pin_memory=True,
            collate_fn=dataset[split].collate_fn)

    if 'spvnas' in args.name.lower():
        model = spvnas_specialized(args.name)
    elif 'spvcnn' in args.name.lower():
        model = spvcnn(args.name)
    elif 'mink' in args.name.lower():
        model = minkunet(args.name)
    else:
        raise NotImplementedError

    if configs.distributed:
        model = torch.nn.parallel.DistributedDataParallel(
            model.cuda(),
            device_ids=[dist.local_rank()],
            find_unused_parameters=True)
    model.eval()

    criterion = builder.make_criterion()
    optimizer = builder.make_optimizer(model)
    scheduler = builder.make_scheduler(optimizer)

    trainer = SemanticKITTITrainer(model=model,
                                   criterion=criterion,
                                   optimizer=optimizer,
                                   scheduler=scheduler,
                                   num_workers=configs.workers_per_gpu,
                                   seed=configs.train.seed)
    callbacks = Callbacks([
        SaverRestore(),
        MeanIoU(configs.data.num_classes, configs.data.ignore_label)
    ])
    callbacks._set_trainer(trainer)
    trainer.callbacks = callbacks
    trainer.dataflow = dataflow['test']

    trainer.before_train()
    trainer.before_epoch()

    model.eval()
    with torch.no_grad():
        for feed_dict in tqdm(dataflow['test'], desc='eval'):
            _inputs = {}
            for key, value in feed_dict.items():
                if 'name' not in key:
                    _inputs[key] = value.cuda()

            inputs = _inputs['lidar']

            targets = feed_dict['targets'].F.long().cuda(non_blocking=True)
            outputs = model(inputs)

            invs = feed_dict['inverse_map']
            all_labels = feed_dict['targets_mapped']
            _outputs = []
            _targets = []
            for idx in range(invs.C[:, -1].max() + 1):
                cur_scene_pts = (inputs.C[:, -1] == idx).cpu().numpy()
                cur_inv = invs.F[invs.C[:, -1] == idx].cpu().numpy()
                cur_label = (all_labels.C[:, -1] == idx).cpu().numpy()
                outputs_mapped = outputs[cur_scene_pts][cur_inv].argmax(1)
                targets_mapped = all_labels.F[cur_label]
                _outputs.append(outputs_mapped)
                _targets.append(targets_mapped)
            outputs = torch.cat(_outputs, 0)
            targets = torch.cat(_targets, 0)
            output_dict = {'outputs': outputs, 'targets': targets}
            trainer.after_step(output_dict)

            abs_input_path = feed_dict['file_name'][0].split("/")
            output_file_name = args.save_dir + "/" + abs_input_path[-3] + "/" + abs_input_path[-1].replace('.bin', '.label')
            sem = outputs.cpu().numpy().astype(np.uint32)
            ins = np.zeros_like(sem).astype(np.uint32)
            pred_eval = sem + (ins << 16)
            pred_eval.astype(np.uint32).tofile(output_file_name)
        trainer.after_epoch()

if __name__ == '__main__':
    main()

• Command: torchpack dist-run -np 1 python evaluate.py configs/mulran/default.yaml --name SemanticKITTI_val_SPVNAS@65GMACs --dataset.root /home/shapelim/git/spvnas/dataset/mulran --save-dir /media/shapelim/UX980/UX960NVMe/mulran_labels/spvnas

Not only the evaluate.py, other minor parts are revised to set appropriate variables/file paths.

[AronCao49]

Hi HyungTae,

Thanks for your interest in this work. Perhaps I can share my experience about how to get the semantic labels. Hope it would help :)

1. About the label mapping. We use the mapped label table of SPVNAS (i.e., a total of 19 semantic classes). You can find the corresponding label table in their original visualization script. In fact, our label generation is mostly based on this script: https://github.com/mit-han-lab/spvnas/blob/08570cf687622c4d5a5cfa2d7d5f3dcb629222d1/visualize.py#L107-L127

2. About the fine-tuning trick. Based on our observation, the raw intensity distribution of Ouster varies a lot compared to the one from SemanticKITTI, which is the main reason that causes the noisy prediction. A naive solution is to set the intensity of all points to 0, which leads to the segmentation result posted by @Pamphlett previously. Of course, there should be a better alternative but we found this simple try-out works in this case. I post the (x, y, z, intensity) range of your data and the one from SemanticKITTI for your reference.

SemanticKITTI:

Yours:

[Pamphlett]

Hi HyungTae,

Yes, actually I saw you in London weeks before haha but didn't have the chance to say hi. Segregator was mainly built upon Quatro and the paper itself also gave me a lot of insight. (Cannot wait to see Quatro++ haha)

[LimHyungTae]

It works! I really appreciate you guys' kind sharing :). This issue gonna be closed