loss become minus when train my own TOM model

[YichengZou626]

Hi, I am trying to train the tractseg model with my own data. But after training the third model (TOM one), the loss was getting minus. Here is the hyperparameter I used: {'BATCH_NORM': False, 'BATCH_SIZE': 44, 'BEST_EPOCH': 0, 'BEST_EPOCH_SELECTION': 'f1', 'CALC_F1': True, 'CLASSES': 'All_Part1', 'CSD_RESOLUTION': 'LOW', 'CV_FOLD': 0, 'DATASET': 'HCP', 'DATASET_FOLDER': '/nas/longleaf/home/zyc626/TractSeg/tractseg/tom', 'DATA_AUGMENTATION': True, 'DAUG_ALPHA': (90.0, 120.0), 'DAUG_BLUR_SIGMA': (0, 1), 'DAUG_ELASTIC_DEFORM': True, 'DAUG_FLIP_PEAKS': False, 'DAUG_GAUSSIAN_BLUR': True, 'DAUG_INFO': '-', 'DAUG_MIRROR': False, 'DAUG_NOISE': True, 'DAUG_NOISE_VARIANCE': (0, 0.05), 'DAUG_RESAMPLE': False, 'DAUG_RESAMPLE_LEGACY': False, 'DAUG_ROTATE': False, 'DAUG_ROTATE_ANGLE': (-0.2, 0.2), 'DAUG_SCALE': True, 'DAUG_SIGMA': (9.0, 11.0), 'DIM': '2D', 'DROPOUT_SAMPLING': False, 'EPOCH_MULTIPLIER': 1, 'EXPERIMENT_TYPE': 'peak_regression', 'EXP_MULTI_NAME': '', 'EXP_NAME': 'my_custom_experiment_3', 'EXP_PATH': '/nas/longleaf/home/zyc626/TractSeg/hcp_exp/my_custom_experiment_3_x3', 'FEATURES_FILENAME': 'mrtrix_peaks', 'FLIP_OUTPUT_PEAKS': False, 'FP16': True, 'GET_PROBS': True, 'INFO': '-', 'INFO_2': 'using AngleLengthLoss, PeakLengthDice', 'INPUT_DIM': (144, 144), 'INPUT_RESCALING': False, 'KEEP_INTERMEDIATE_FILES': False, 'LABELS_FILENAME': 'bundle_masks', 'LABELS_FOLDER': 'bundle_masks', 'LABELS_TYPE': 'float', 'LEARNING_RATE': 0.001, 'LOAD_WEIGHTS': False, 'LOSS_FUNCTION': 'angle_length_loss', 'LOSS_WEIGHT': 5, 'LOSS_WEIGHT_LEN': -1, 'LR_SCHEDULE': True, 'LR_SCHEDULE_MODE': 'min', 'LR_SCHEDULE_PATIENCE': 20, 'METRIC_TYPES': ['loss', 'f1_macro', 'angle_err'], 'MODEL': 'UNet_Pytorch_DeepSup', 'MULTI_PARENT_PATH': '/nas/longleaf/home/zyc626/TractSeg/hcp_exp/', 'NORMALIZE_DATA': True, 'NORMALIZE_PER_CHANNEL': False, 'NR_CPUS': -1, 'NR_OF_CLASSES': 60, 'NR_OF_GRADIENTS': 9, 'NR_SLICES': 1, 'NUM_EPOCHS': 160, 'ONLY_VAL': False, 'OPTIMIZER': 'Adamax', 'OUTPUT_MULTIPLE_FILES': False, 'PAD_TO_SQUARE': True, 'PEAK_DICE_LEN_THR': 0.05, 'PEAK_DICE_THR': [0.95], 'PREDICT_IMG': False, 'PREDICT_IMG_OUTPUT': None, 'PRINT_FREQ': 20, 'P_SAMP': 1.0, 'RESET_LAST_LAYER': False, 'RESOLUTION': '1.25mm', 'SAVE_WEIGHTS': True, 'SEGMENT': False, 'SEG_INPUT': 'Peaks', 'SLICE_DIRECTION': 'y', 'SPATIAL_TRANSFORM': 'SpatialTransform', 'TEST': False, 'TEST_TIME_DAUG': False, 'THRESHOLD': 0.01, 'TRACTSEG_DIR': 'tractseg_output', 'TRAIN': True, 'TRAINING_SLICE_DIRECTION': 'y', 'TYPE': 'single_direction', 'UNET_NR_FILT': 64, 'UPSAMPLE_TYPE': 'nearest', 'USE_DROPOUT': False, 'USE_VISLOGGER': False, 'VERBOSE': True, 'WEIGHTS_PATH': '', 'WEIGHT_DECAY': 0} I changed the threshold from 0.5 to 0.01, because it is for peak regression. Also, I changed the loss_weight to 1, and I am wondering if that matters.

[czq0827]

Hello, I have encountered a similar situation. During the training of TOM, there was a negative loss value. May I ask how you resolved it? Looking forward to your reply. Thank you.

[chenglong0290]

Hello, please tell me which files need to be modified when training my own model. I have encountered problems in this regard. I will be very grateful if you can provide help.