More and more complete information

[BudBudding]

Hello, could you please share more complete code and usage methods, such as how to train models, obtain weights, and perform inspections?

[ChuanxinSong]

Hello, I am quite busy recently, the remaining code of the paper will be updated in the near future, please wait patiently.

If you are in a hurry, you can also refer to this source code: SAS-Net. Our code is implemented under this framework.

[BudBudding]

Thank you for your reply. I am still looking forward to your work. If you have updated the code, hope to receive your sharing.

[ChuanxinSong]

Thank you for your reply！

[BudBudding]

I'm sorry to bother you. Based on your suggestion, I have reviewed the above paper and code. That code did not provide a training py file. I wrote the training file myself, but the training effect was very different from what the author provided. May I ask if it's convenient for me to take a look and see if there are any issues with my writing? Or could you provide your training documents? Here is my training code for SemBranch: """ Training file Usage: --config [PATH to configuration file for desired dataset] """ from Libs.Utils.dfw import dfw from RGBBranch import RGBBranch from SemBranch import SemBranch from Libs.Datasets.ADE20KDataset import ADE20KDataset from Libs.Utils import utils

import torch.backends.cudnn as cudnn import numpy as np import argparse import yaml import torch import os import time

parser = argparse.ArgumentParser(description='Semantic-Aware Scene Recognition Evaluation') parser.add_argument('--ConfigPath', metavar='DIR', help='Configuration file path')

def evaluationDataLoader(dataloader, model, set): batch_time = utils.AverageMeter() losses = utils.AverageMeter() top1 = utils.AverageMeter() top2 = utils.AverageMeter() top5 = utils.AverageMeter()

Extract batch size

batch_size = CONFIG['VALIDATION']['BATCH_SIZE']['TEST']
# Start data time
data_time_start = time.time()
model.eval()
with torch.no_grad():
    for i, (mini_batch) in enumerate(dataloader):
        start_time = time.time()
        if USE_CUDA:
            RGB_image = mini_batch['Image'].cuda()
            semantic_mask = mini_batch['Semantic'].cuda()
            semantic_scores = mini_batch['Semantic Scores'].cuda()
            sceneLabelGT = mini_batch['Scene Index'].cuda()

        if set is 'Validation' and CONFIG['VALIDATION']['TEN_CROPS']:
            # Fuse batch size and ncrops to set the input for the network
            bs, ncrops, c_img, h, w = RGB_image.size()
            RGB_image = RGB_image.view(-1, c_img, h, w)

            bs, ncrops, c_sem, h, w = semantic_mask.size()
            semantic_mask = semantic_mask.view(-1, c_sem, h, w)

            bs, ncrops, c_sem, h, w = semantic_scores.size()
            semantic_scores = semantic_scores.view(-1, c_sem, h, w)

        semanticTensor = utils.make_one_hot(semantic_mask, semantic_scores, C=CONFIG['DATASET']['N_CLASSES_SEM'])
        # Model Forward
        outputSceneLabel, feature_conv, _, outputSceneLabelSEM = model(RGB_image, semanticTensor)

        if set is 'Validation' and CONFIG['VALIDATION']['TEN_CROPS']:
            # Average results over the 10 crops
            outputSceneLabelSEM = outputSceneLabelSEM.view(bs, ncrops, -1).mean(1)

        if batch_size is 1:
            if set is 'Validation' and CONFIG['VALIDATION']['TEN_CROPS']:
                RGB_image = torch.unsqueeze(RGB_image[4, :, :, :], 0)

        # Compute Loss
        loss = model.loss(outputSceneLabelSEM, sceneLabelGT)

        # Measure Top1, Top2 and Top5 accuracy
        prec1, prec2, prec5 = utils.accuracy(outputSceneLabelSEM.data, sceneLabelGT, topk=(1, 2, 5))

        # Update values
        losses.update(loss.item(), batch_size)
        top1.update(prec1.item(), batch_size)
        top2.update(prec2.item(), batch_size)
        top5.update(prec5.item(), batch_size)

        # Measure batch elapsed time
        batch_time.update(time.time() - start_time)

        # Print information
        if i % CONFIG['VALIDATION']['PRINT_FREQ'] == 0:
            print('Testing {} set batch: [{}/{}]\t'
                  'Batch Time {batch_time.val:.3f} (avg: {batch_time.avg:.3f})\t'
                  'Loss {loss.val:.3f} (avg: {loss.avg:.3f})\t'
                  'Prec@1 {top1.val:.3f} (avg: {top1.avg:.3f})\t'
                  'Prec@2 {top2.val:.3f} (avg: {top2.avg:.3f})\t'
                  'Prec@5 {top5.val:.3f} (avg: {top5.avg:.3f})'.
                  format(set, i, len(dataloader), set, batch_time=batch_time, loss=losses,
                         top1=top1, top2=top2, top5=top5))

    print('Elapsed time for {} set evaluation {time:.3f} seconds'.format(set, time=time.time() - data_time_start))
    print("")

    return top1.avg, top2.avg, top5.avg, losses.avg

def train_model(model, train_loader, val_loader, optimizer, scheduler=None): best_top1 = 0.0 best_epoch = 0

Extract batch size

batch_size = CONFIG['TRAINING']['BATCH_SIZE']['TRAIN']
# Start data time
for epoch in range(100):
    model.train()
    losses = utils.AverageMeter()
    print("-" * 65)
    print(f'Training for epoch {epoch}')
    for i, mini_batch in enumerate(train_loader):
        RGB_image = mini_batch['Image'].cuda()
        sceneLabelGT = mini_batch['Scene Index'].cuda()
        if USE_CUDA:
            RGB_image = mini_batch['Image'].cuda()
            semantic_mask = mini_batch['Semantic'].cuda()
            semantic_scores = mini_batch['Semantic Scores'].cuda()
            sceneLabelGT = mini_batch['Scene Index'].cuda()
        # Create tensor of probabilities from semantic_mask
        semanticTensor = utils.make_one_hot(semantic_mask, semantic_scores, C=CONFIG['DATASET']['N_CLASSES_SEM'])
        # Model Forward
        optimizer.zero_grad()
        outputSceneLabel, feature_conv, _, outputSceneLabelSEM = model(RGB_image, semanticTensor)

        # Compute Loss
        loss = model.loss(outputSceneLabelSEM, sceneLabelGT)
        loss.backward()
        optimizer.step(lambda: float(loss))

        # Update values
        losses.update(loss.item(), batch_size)

        if i % CONFIG['TRAINING']['PRINT_FREQ'] == 0:
            print('Training Epoch {} \t'
                  'Batch {}/{}\t'
                  'Loss {loss.val:.3f} (avg: {loss.avg:.3f})'.format(epoch, i, len(train_loader), loss=losses))
    if scheduler is not None:
        scheduler.step()

    # eval and save model
    if epoch % 5 == 0:
        val_top1, val_top2, val_top5, val_loss = evaluationDataLoader(val_loader, model, set='Validation')
        print(' Validation results: Loss {val_loss:.3f}, Prec@1 {top1:.3f}, Prec@2 {top2:.3f}, Prec@5 {top5:.3f}'
              .format(val_loss=val_loss, top1=val_top1, top2=val_top2, top5=val_top5))

        # save the best model
        if val_top1 - best_top1 > 0.5:
            # remove the old model
            ckpt_name = os.path.join(CONFIG["MODEL"]["PATH"], "sem-model-best-epoch-{}.ckpt".format(best_epoch))
            if os.path.isfile(ckpt_name):
                os.remove(ckpt_name)
            # save the new model
            ckpt_name = os.path.join(CONFIG["MODEL"]["PATH"], "sem-model-best-epoch-{}.ckpt".format(epoch))
            torch.save(model.state_dict(), ckpt_name)
            print("update the sem-model with {} top1 value in {} epoch.".format(val_top1, epoch))

            best_top1 = val_top1
            best_epoch = epoch

if name == 'main':

global USE_CUDA, classes, CONFIG

# Decode CONFIG file information
args = parser.parse_args()
CONFIG = yaml.safe_load(open(args.ConfigPath, 'r'))
USE_CUDA = torch.cuda.is_available()

print('-' * 65)
print("Training started starting...")
print('-' * 65)
# Instantiate network
print('Training ONLY SEM branch')
print('Selected SEM backbone architecture: ' + CONFIG['MODEL']['ARCH'])
model = SemBranch(scene_classes=CONFIG['DATASET']['N_CLASSES_SCENE'], semantic_classes=151)

# Move Model to GPU an set it to evaluation mode
if USE_CUDA:
    model.cuda()
cudnn.benchmark = USE_CUDA
model.train()

print('-' * 65)
print('Loading dataset {}...'.format(CONFIG['DATASET']['NAME']))

traindir = os.path.join(CONFIG['DATASET']['ROOT'], CONFIG['DATASET']['NAME'])
valdir = os.path.join(CONFIG['DATASET']['ROOT'], CONFIG['DATASET']['NAME'])

# train_dataset = MITIndoor67Dataset(traindir, "train")
train_dataset = ADE20KDataset(traindir, "training")
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=CONFIG['TRAINING']['BATCH_SIZE']['TRAIN'],
                                           shuffle=True, num_workers=CONFIG['DATALOADER']['NUM_WORKERS'],
                                           pin_memory=False)

val_dataset = ADE20KDataset(valdir, "validation", tencrops=CONFIG['VALIDATION']['TEN_CROPS'], SemRGB=True)
val_loader = torch.utils.data.DataLoader(val_dataset, batch_size=CONFIG['TRAINING']['BATCH_SIZE']['TEST'],
                                         shuffle=False, num_workers=CONFIG['DATALOADER']['NUM_WORKERS'],
                                         pin_memory=False)
classes = train_dataset.classes
# Print dataset information
print('Train set. Size {}. Batch size {}. Nbatches {}'
      .format(len(train_loader) * CONFIG['VALIDATION']['BATCH_SIZE']['TRAIN'],
              CONFIG['VALIDATION']['BATCH_SIZE']['TRAIN'], len(train_loader)))
print('Train set number of scenes: {}'.format(len(classes)))
optimizer = dfw.DFW(model.parameters(), eta=CONFIG['TRAINING']['LR'], momentum=CONFIG['TRAINING']['MOMENTUM'],
                    weight_decay=CONFIG['TRAINING']['WEIGHT_DECAY'])
train_model(model, train_loader, val_loader, optimizer)

[ChuanxinSong]

I don't quite understand whether you tried our SSRM or SAS-Net?

[BudBudding]

SAS-Net. My GitHub shows that your code has not been published yet.

[ChuanxinSong]

I also encountered the problem you mentioned in my attempt at SAS-Net, and it was not solved. Maybe you can look for "answers" in the Issue of the article.

Sorry I have been busy recently, our other code will be released in the near future, please be patient

[BudBudding]

Thank you for your reply and we look forward to your future work. Wishing you success in your research.

[ChuanxinSong]

Please note that our code is only made with some help from the open-source framework released by SAS-Net. This is not equivalent to the accuracy achieved by our model being correlated with them. So if you are interested, please be patient to check out our processing details, including the model, semantic segmentation technique, training strategy, etc. With the above and SAS-Net code framework, it will not take much time to fully reproduce our method.

[ifundo]

Hello, when can you publish the code? Thank you.