About the decoding_layer function in the file Structural_feature_decoding.py

[popu0]

Hello, author! In this function, x_trn, x_val, y_trn, y_val doesn't seem to correspond. Looking forward to your reply.

[ReedOnePeck]

Hi, x_trn and x_val refer to the fMRI data for the training set and validation set, respectively. The function fetch_ROI_voxel(trn_file_ex, ROIs) is used to retrieve the voxels for each Region of Interest (ROI). On the other hand, y_trn and y_val refer to the CLIP features for the training set and validation set, respectively. The function CLIP_feature_layer(trn_CLIP_feature_all_layer, layer_name) is used to extract the features from each layer of CLIP.

[ReedOnePeck]

If there are any other questions, feel free to contact at any time

[popu0]

x_trn = fetch_ROI_voxel(trn_file_ex, ROIs)  # (8859,11694)
x_mean = np.mean(x_trn, axis=0)
x_std = np.std(x_trn, axis=0)

x_trn = (x_trn - x_mean) / x_std
trn_CLIP_feature_all_layer = fetch_CLIP_feature(trn_CLIP_feature_path + '/trn_stim_CLIP_zscore.npy')

x_val = fetch_ROI_voxel(val_file_ex, ROIs)[:500, :]
x_val = (x_val - x_mean) / x_std
val_CLIP_feature_all_layer = fetch_CLIP_feature(val_CLIP_feature_path + '/val_stim_CLIP_zscore.npy')

if layer_name != 'VisionTransformer-1':
    mask = np.load(mask_path + '/{}/Folder_5_{}.npy'.format(layer_name,remain_rate))
    y_trn = CLIP_feature_layer(trn_CLIP_feature_all_layer, layer_name)[:8000,mask]
    y_val = CLIP_feature_layer(val_CLIP_feature_all_layer, layer_name)[8000:, mask]
else:
    y_trn = CLIP_feature_layer(trn_CLIP_feature_all_layer, layer_name)[:8000,:]
    y_val = CLIP_feature_layer(val_CLIP_feature_all_layer, layer_name)[8000:, :]

Hello, glad to see your reply. x_trn looks like the complete training set, but y_trn takes the first 8000 data. And x_val is 500. I looked at the file Semantic_feature_decoding.py. The two files seem to have different data retrieval methods, which makes me feel a little confused.

[ReedOnePeck]

Hello, thank you for raising this issue. I'm sorry that there was a problem when I was reorganizing the code. According to the description in the paper, when searching for hyperparameters, we used the first 8000 data in the training set for training and the last 859 data in the training set for validation.

We have selected the optimal hyperparameters n and remain_rate, and you can directly use all 8859 training data to train.

[popu0]

Thank you for your reply. I understand. I have two more questions about the file Structural_feature_selection.py. 1.When selecting features, the code seems to have filtered only Linear 1. 2.trn_CLIP_feature_all_layer = np.concatenate([fetch_CLIP_feature(CLIP_feature_saved_path + "/trn_stim_CLIPzscore{}.npy".format(i + 1)) for i in range(1,9)], axis=0).Whether this line of code represents combining the characteristic data of multiple subjects. I don't know if my understanding is correct. Looking forward to your reply.

def feature_select(index_save_path, CLIP_feature_saved_path, x_trn, k_folder=5, layer_name = 'Linear-1',rate = 50): index_save_path = index_save_path + '/{}/'.format(layer_name) if not os.path.exists(model_save_path): os.makedirs(model_save_path) trn_CLIP_feature_all_layer = np.concatenate([fetch_CLIP_feature(CLIP_feature_saved_path + "/trn_stim_CLIPzscore{}.npy".format(i + 1)) for i in range(1,9)], axis=0) y_trn_all = CLIP_feature_layer(trn_CLIP_feature_all_layer, layer_name) cor_all = np.concatenate([feature_select_onefold(x_trn = x_trn, k_folder=k_folder, rate=rate, fold_id=i+1, y_trn_all=y_trn_all) for i in range(4)] , axis = 0) bar = np.percentile(cor_all, rate) selected_index = [] for i in range(cor_all.shape[0]): if cor_all[i] >= bar: selected_index.append(i) np.save(index_save_path + 'Folder5{}.npy'.format(100 - rate), np.array(selected_index)) return for layer in args.Layers: b = feature_select(index_save_path=args.index_save_path, CLIP_feature_saved_path=args.CLIP_feature_saved_path, x_trn=x_trn, k_folder=5, layer_name = 'Linear-1',rate = args.rate)

[ReedOnePeck]

Reply： 1.Linear-1 is the initial assignment of the function. In fact, this code will perform feature selection for 'Linear-2', 'Linear-4', 'Linear-6', 'Linear-8', 'Linear-10', 'Linear-12'

2.Due to the different structural and functional topologies of different subjects, it is currently not possible to combine the data from all eight subjects for computation. Our article trains a model separately for each subject. The for i in range(8) here is because when I used CLIP to extract training features, there were 8,859 pictures, but the CUDA was out of memory, so I could only process 1,200 images at a time and saved eight copies.