fatemehkalantari1993 commented 1 year ago

I had a problem with Caffe installation on ubuntu20.4 and I sent an email to you. You suggested me a code in the Torch framework (https://github.com/KamitaniLab/brain-decoding-cookbook-public/tree/main/reconstruction). I did reconstruction with torch and Keras frameworks but I had bigger a error than Caffe (https://github.com/KamitaniLab/DeepImageReconstruction). when I ran this code, I got a big error too. I ran this code and compare its results with the results of Caffe (https://github.com/KamitaniLab/DeepImageReconstruction).

I said my problem to you and you said that this problem may due to my Cuda and Torch versions. I sent my results with your data, my Python and Torch versions, and python code for you. I have big errors. why????

Torch version: 1.12.1+cu116 python version: 3.10.6 result in iter200: error= 2594284032.0 for n01443537_22563.jpeg, sub-01 `

import argparse import glob from itertools import product import os import pickle

from bdpy.recon.torch.icnn import reconstruct from bdpy.recon.utils import normalize_image, clip_extreme from bdpy.dl.torch.models import VGG19, AlexNetGenerator, layer_map from bdpy.dataform import Features, DecodedFeatures from bdpy.feature import normalize_feature from bdpy.util import dump_info import numpy as np import PIL.Image import scipy.io as sio import torch print(torch.version) import torch.optim as optim import yaml

Functions

def image_preprocess(img, image_mean=np.float32([104, 117, 123])): '''convert to Caffe's input image layout''' return np.float32(np.transpose(img, (2, 0, 1))[::-1]) - np.reshape(image_mean, (3, 1, 1))

def image_deprocess(img, image_mean=np.float32([104, 117, 123])): '''convert from Caffe's input image layout''' return np.dstack((img + np.reshape(image_mean, (3, 1, 1)))[::-1])

Network settings -------------------------------------------------------

features_dir = '/home/mvl/kalantari/data/decoded_features/ImageNetTest/deeprecon_originals/VGG19' output_dir = '/home/mvl/kalantari/results/' subject = 'sub-01' roi = 'VC'

device = 'cuda:0'

encoder_param_file = '/home/mvl/kalantari/data/net/VGG_ILSVRC_19_layers/VGG_ILSVRC_19_layers.pt'

layers = [ 'conv1_1', 'conv1_2', 'conv2_1', 'conv2_2', 'conv3_1', 'conv3_2', 'conv3_3', 'conv3_4', 'conv4_1', 'conv4_2', 'conv4_3', 'conv4_4', 'conv5_1', 'conv5_2', 'conv5_3', 'conv5_4', ]

layer_mapping = layer_map('vgg19') encoder_input_shape = (224, 224, 3)

generator_param_file = '/home/mvl/kalantari/data/net/bvlc_reference_caffenet_generator_ILSVRC2012_Training/generator_relu7.pt'

image_mean_file = '/home/mvl/kalantari/data/net/VGG_ILSVRC_19_layers/ilsvrc_2012_mean.npy' image_mean = np.load(image_mean_file) image_mean = np.float32([image_mean[0].mean(), image_mean[1].mean(), image_mean[2].mean()])

feature_std_file = '/home/mvl/kalantari/data/net/VGG_ILSVRC_19_layers/estimated_cnn_feat_std_VGG_ILSVRC_19_layers_ImgSize_224x224_chwise_dof1.mat' feature_range_file = '/home/mvl/kalantari/data/net/bvlc_reference_caffenet_generator_ILSVRC2012_Training/act_range/3x/relu7.txt'

std_ddof = 1 channel_axis = 0

n_iter = 200

# Reconstruction options -------------------------------------------------

opts = { 'loss_func': torch.nn.MSELoss(reduction='sum'), 'n_iter': n_iter, 'lr': (2., 1e-10), 'momentum': (0.9, 0.9), 'decay': (0.01, 0.01), 'blurring': False, 'channels': None, 'masks': None, 'disp_interval': 1,}

Initial image for the optimization (here we use the mean of ilsvrc_2012_mean.npy as RGB values)

initial_image = np.zeros((224, 224, 3), dtype='float32') initial_image[:, :, 0] = image_mean[2].copy() initial_image[:, :, 1] = image_mean[1].copy() initial_image[:, :, 2] = image_mean[0].copy()

Feature SD estimated from true DNN features of 10000 images

feat_std0 = sio.loadmat(feature_std_file)

Feature upper/lower bounds

cols = 4096 up_size = (4096,) upper_bound = np.loadtxt(feature_range_file, delimiter=' ', usecols=np.arange(0, cols), unpack=True) upper_bound = upper_bound.reshape(up_size)

Initial features -------------------------------------------------------

initial_gen_feat = np.random.normal(0, 1, (4096,))

Setup results directory ------------------------------------------------

if not os.path.exists(output_dir): os.makedirs(output_dir)

Set reconstruction options ---------------------------------------------

opts.update({

The initial image for the optimization (setting to None will use random noise as initial image)

'initial_feature': initial_gen_feat,
'feature_upper_bound': upper_bound,
'feature_lower_bound': 0.,
})

decoded = subject is not None and roi is not None print('----------------------------------------') if decoded: print('Subject: ' + subject) print('ROI: ' + roi) print('')

if decoded: save_dir = os.path.join(output_dir, subject, roi) else: save_dir = os.path.join(output_dir)

if not os.path.exists(save_dir): os.makedirs(save_dir)

Get images if images is None

if decoded: matfiles = glob.glob(os.path.join(features_dir, layers[0], subject, roi, '.mat')) else: matfiles = glob.glob(os.path.join(features_dir, layers[0], '.mat'))

images = [os.path.splitext(os.path.basename(fl))[0] for fl in matfiles]

Load DNN features

if decoded: features = DecodedFeatures(os.path.join(features_dir), squeeze=False) else: features = Features(features_dir)

Images loop

for image_label in images[:1]: print('Image: ' + image_label)

# Encoder model
encoder = VGG19()
encoder.load_state_dict(torch.load(encoder_param_file))
encoder.eval()

# Generator model
generator = AlexNetGenerator()
generator.load_state_dict(torch.load(generator_param_file))
generator.eval()

# Districuted computation control
snapshots_dir = os.path.join(save_dir, 'snapshots', 'image-%s' % image_label)
if os.path.exists(snapshots_dir):
   print('Already done or running. Skipped.')
   continue

# Load DNN features
if decoded:
   feat = {
       layer: features.get(layer=layer, subject=subject, roi=roi, image=image_label)
       for layer in layers
   }
else:
    labels = features.labels
    feat = {
        layer: features.get_features(layer)[np.array(labels) == image_label]
        for layer in layers
    }

for layer, ft in feat.items():

    ft0 = normalize_feature(
          ft[0],
          channel_wise_mean=False, channel_wise_std=False,
          channel_axis=channel_axis,
          shift='self', scale=np.mean(feat_std0[layer]),
          std_ddof=std_ddof
    )
    ft = ft0[np.newaxis]
    feat.update({layer: ft})

                # Norm of the DNN features for each layer
feat_norm = np.array([np.linalg.norm(feat[layer])
                      for layer in layers],
                     dtype='float32')
weights = 1. / (feat_norm ** 2)

        # Normalise the weights such that the sum of the weights = 1
weights = weights / weights.sum()
layer_weights = dict(zip(layers, weights))

opts.update({'layer_weights': layer_weights})

# Reconstruction
snapshots_dir = os.path.join(save_dir, 'snapshots', 'image-%s' % image_label)
recon_image, loss_list = reconstruct(feat,
                                     encoder,
                                     generator=generator,
                                     layer_mapping=layer_mapping,
                                     optimizer=optim.SGD,
                                     image_size=encoder_input_shape,
                                     crop_generator_output=True,
                                     preproc=image_preprocess,
                                     postproc=image_deprocess,
                                     output_dir=save_dir,
                                     save_snapshot=True,
                                     snapshot_dir=snapshots_dir,
                                     snapshot_ext='tiff',
                                     snapshot_postprocess=normalize_image,
                                     return_loss=True,
                                     **opts)

# Save the raw reconstructed image
recon_image_mat_file = os.path.join(save_dir, 'recon_image' + '-' + image_label + '.mat')
sio.savemat(recon_image_mat_file, {'recon_image': recon_image})

recon_image_normalized_file = os.path.join(save_dir, 'recon_image_normalized' + '-' + image_label + '.tiff')
PIL.Image.fromarray(normalize_image(clip_extreme(recon_image, pct=4))).save(recon_image_normalized_file)

print('All done')`

ShuntaroAoki commented 1 year ago

Thank you for the detailed explanation. We have noticed the issue and are currently working on replicating and resolving it. However, due to limited resources, we have not yet been able to resolve the issue. I apologize for the inconvenience. I will try to provide you with a response within a week.

fatemehkalantari1993 commented 1 year ago

Thank you very much. So I am waiting for your help.

On Mon, 12 Tir 1402 06:04 PM, Shuntaro Aoki @.***> wrote:

Thank you for the detailed explanation. We have noticed the issue and are currently working on replicating and resolving it. However, due to limited resources, we have not yet been able to resolve the issue. I apologize for the inconvenience. I will try to provide you with a response within a week. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***> -- This email was Anti Virus checked by Security Gateway.

fatemehkalantari1993 commented 1 year ago

In fact, the problem is that the output of each layer (with the same input) in this part of the code that I have attached is very different in Torch, Keras, and Caffe, and errors become big. The difference between feat and feat0 is very big.

The output of conv1_1 layer in caffe and keras (reconstruction without generator and with icnn_gd):

keras: feature_model = Model(inputs=model.input, outputs=model.get_layer('conv1_1').output) feat = feature_model.predict(input_data) print(feat)

[[[[0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] ... [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ]]

[[0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] ... [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ]]

...

[[0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] ... [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ]]

[[0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] ... [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ] [0.7301776 0.06493629 0.03428847 ... 1.0892007 0.47757268 0.4072122 ]]]]

caffe: net.blobs['data'].data[0] = img.copy() net.forward(end=layer_list[-1]) feat_k = net.blobs['conv1_1'].data print(feat_k)

[[[[ 2.43036518e+01 3.56820130e+00 -1.25920921e+02 ... -4.33597565e+01 -1.83088422e+00 7.47574692e+01] [-6.84213715e+01 -1.48907898e+02 -2.89248047e+02 ... 1.31049988e+02 1.41945602e+02 1.42724915e+02] [ 7.93817596e+01 1.45224857e+01 -1.34395844e+02 ... -3.30107422e+01 -1.26928391e+02 -1.33342409e+01] ... [-1.10655563e+02 -8.89228897e+01 -7.74521942e+01 ... -5.86386414e+01 -8.69777908e+01 6.24097061e+01] [-9.74062729e+01 1.07463585e+02 8.99701233e+01 ... 9.98462372e+01 -3.04186516e+01 8.11108093e+01] [ 4.54988861e+01 1.83784531e+02 2.75571537e+01 ... 9.62042618e+01 -9.59368896e+01 4.36504097e+01]]

[[ 1.15499067e+01 -2.24013348e+01 -3.73924408e+01 ... 1.50041761e+01 3.04891415e+01 2.52738037e+01] [ 3.63860130e+01 -6.41979980e+00 -2.05265980e+01 ... 5.49893045e+00 2.14854107e+01 3.23399696e+01] [ 4.66856346e+01 3.02258148e+01 1.28432207e+01 ... 1.49757957e+01 2.33222175e+00 2.17266674e+01] ... [-2.44258251e+01 -3.08063774e+01 -3.56204872e+01 ... 3.82022738e+00 2.09232273e+01 2.08943443e+01] [ 9.80745602e+00 2.40307999e+00 -2.61969509e+01 ... -6.91338682e+00 3.54264712e+00 3.16898785e+01] [ 2.53129025e+01 1.09556675e+01 -3.44352036e+01 ... -2.21822586e+01 -2.23502159e+01 1.01088934e+01]]

[[-8.68284988e+00 -4.15883102e+01 -6.41782761e+01 ... 3.87116852e+01 5.74583321e+01 6.45260010e+01] [ 4.04361572e+01 -1.04818954e+01 -6.93674393e+01 ... -1.14138107e+01 -1.01125622e+01 2.91498623e+01] [ 4.78804588e+01 5.75603371e+01 2.86385326e+01 ... -3.69166679e+01 -4.01310654e+01 2.23414364e+01] ... [-3.60933800e+01 -2.04775906e+01 -4.45583687e+01 ... -1.01242161e+01 7.26186228e+00 4.17242584e+01] [-2.84511356e+01 7.53477764e+00 -1.90827045e+01 ... -7.98048782e+00 -2.04292011e+01 5.23601265e+01] [ 1.55033617e+01 2.31920414e+01 -5.00152779e+01 ... -2.34005566e+01 -6.15089378e+01 -1.38107700e+01]]

...

[[ 2.03526058e+01 3.00890617e+01 -4.95997906e+00 ... -3.77609596e+01 -6.28957405e+01 -2.35124817e+01] [ 2.98741207e+01 8.29564810e-01 -4.74223251e+01 ... -5.71452026e+01 -7.16696472e+01 -5.98899651e+01] [-2.15872693e+00 1.07022667e+00 -2.14295120e+01 ... -1.88944874e+01 -4.48330193e+01 -2.71363087e+01] ... [-3.58933411e+01 -3.50818558e+01 -7.28711777e+01 ... -4.15425110e+01 -7.34035969e-02 -4.96850853e+01] [-6.60309906e+01 -5.25685959e+01 -5.75856628e+01 ... -7.35267639e+00 3.65869641e+00 8.04949188e+00] [-3.33893700e+01 -3.48442841e+01 -3.35984840e+01 ... 3.50117278e+00 -1.09368682e+00 -1.69529266e+01]]

[[ 2.45372009e+01 -6.45377197e+01 -1.51196106e+02 ... 3.68484840e+01 8.97927933e+01 1.17577881e+02] [ 1.68583527e+02 9.98302078e+01 -1.29920044e+02 ... -1.46675354e+02 -1.31027023e+02 -4.57123566e+00] [-4.94287491e+00 5.06024628e+01 -2.35740738e+01 ... -6.42218170e+01 -7.85061646e+01 7.21934280e+01] ... [-1.74686337e+01 6.09535561e+01 6.37779160e+01 ... -8.50162659e+01 -4.71711693e+01 8.02226028e+01] [-1.67503300e+01 5.58245392e+01 -5.81582680e+01 ... -9.25071239e-02 -1.44030914e+02 3.54952164e+01] [ 1.27125578e+01 5.96785583e+01 -1.20143280e+02 ... 4.99216118e+01 -1.44740875e+02 -1.00630402e+02]]

[[-1.25994310e+01 -8.66987305e+01 -1.65883728e+02 ... 6.69289551e+01 1.31918564e+02 1.59578659e+02] [ 1.27301926e+02 2.47248402e+01 -2.05888214e+02 ... -8.09537964e+01 -6.96694183e+01 2.91447449e+01] [ 2.99536476e+01 6.72342224e+01 -1.95589294e+01 ... -8.37600555e+01 -1.20017563e+02 6.14797974e+01] ... [-3.70817108e+01 5.71986427e+01 5.25190239e+01 ... -1.09286575e+02 -4.54462967e+01 6.40232162e+01] [-2.60610199e+01 1.05097031e+02 6.85374832e+00 ... 2.98404961e+01 -1.13495934e+02 8.43368073e+01] [ 5.74910889e+01 1.05252083e+02 -1.13385933e+02 ... 4.54843941e+01 -1.71796295e+02 -8.44327393e+01]]]]

On Tue, 13 Tir 1402 12:34 AM, @.*** wrote:

Thank you very much. So I am waiting for your help.

On Mon, 12 Tir 1402 06:04 PM, Shuntaro Aoki @.***> wrote:

Thank you for the detailed explanation. We have noticed the issue and are currently working on replicating and resolving it. However, due to limited resources, we have not yet been able to resolve the issue. I apologize for the inconvenience. I will try to provide you with a response within a week. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***> -- This email was Anti Virus checked by Security Gateway.

micchu commented 1 year ago

Hello. Thank you for your question. Unfortunately, we were unable to reproduce the issue. Below is the environment during testing. We can successfully reconstruct without any problems using the script provided in this repository (recon_icnn_image_vgg19_dgn_relu7gen_gd.py).

OS: Ubuntu20.04
GPU driver version: 510.60.02
Python: 3.10.6
Torch+CUDA: 1.12.1+cu116
bdpy: 0.19

We know that there are cases where reconstruction fails with specific versions of Torch, such as Pytorch 1.9.1. However, this was not the case this time.

Furthermore, the caffe output of the conv1_1 layer you provided appears to be correct. For example, the activation of conv1_1 when the input is "n01443537_22563" is as follows, and the scale of values is close:

array([[[[ 2.51477509e+02,  1.94084152e+02,  1.90671219e+02, ...,
           1.94927628e+02,  1.95883209e+02, -2.81779814e+00],
         [ 1.57871948e+02, -1.71377697e+01, -1.65778790e+01, ...,
           3.12216401e+00,  1.02690327e+00, -2.09711792e+02],
         [ 1.51870148e+02, -1.34124165e+01, -6.63964939e+00, ...,
           3.86457968e+00,  3.44568682e+00, -2.05663879e+02],
         ...,
         [-1.31289581e+02, -1.28243113e+01, -3.33981857e+01, ...,
           4.42263079e+00,  5.39223528e+00, -2.66478455e+02],
         [-7.83458023e+01,  4.40543594e+01,  3.59504509e+01, ...,
           4.12912798e+00,  1.47437406e+00, -2.67864655e+02],
         [ 3.04511051e+01,  1.47859955e+02,  1.61898727e+02, ...,
          -2.52271164e+02, -2.55292587e+02, -3.83337128e+02]],

        [[-1.58220177e+01, -3.06688843e+01, -3.00492077e+01, ...,
          -3.19152451e+01, -3.08211956e+01, -4.19827347e+01],
         [-1.85093632e+01, -3.61226463e+01, -3.44978371e+01, ...,
          -3.92450104e+01, -3.79475136e+01, -5.40956421e+01],
         [-1.91209679e+01, -3.49747925e+01, -3.33080101e+01, ...,
          -3.95044479e+01, -3.87038498e+01, -5.48260994e+01],
          ...

On the other hand, your Keras output of conv1_1 seems to have all vectors taking the same value, which is clearly unusual. It is likely that your problem is dependent on the implementation in Keras. Could you please check if there are any errors in the preprocessing of the input image?

fatemehkalantari1993 commented 1 year ago

I have this problem with Torch and Keras. Errors in your code are too bigger than Caffe. why??? I don't have any error in the preprocessing of the input image because I checked the output of every line in Torch and Keras with the output of every line in the Caffe code. In fact, the problem was found from this line onwards.

fatemehkalantari1993 commented 1 year ago

thank you.It seems that you have made corrections in the code because the error in your code has decreased.

On Thu, 29 Tir 1402 04:01 AM, Misato Tanaka @.***> wrote:

Hello. Thank you for your question.

Unfortunately, we were unable to reproduce the issue.

Below is the environment during testing. We can successfully reconstruct without any problems using the script provided in this repository (recon_icnn_image_vgg19_dgn_relu7gen_gd.py).

OS: Ubuntu20.04
GPU driver version: 510.60.02
Python: 3.10.6
Torch+CUDA: 1.12.1+cu116
bdpy: 0.19

We know that there are cases where reconstruction fails with specific versions of Torch, such as Pytorch 1.9.1.

However, this was not the case this time. Furthermore, the caffe output of the conv1_1 layer you provided appears to be correct.

For example, the activation of conv1_1 when the input is "n01443537_22563" is as follows, and the scale of values is close: array([[[[ 2.51477509e+02, 1.94084152e+02, 1.90671219e+02, ..., 1.94927628e+02, 1.95883209e+02, -2.81779814e+00], [ 1.57871948e+02, -1.71377697e+01, -1.65778790e+01, ..., 3.12216401e+00, 1.02690327e+00, -2.09711792e+02], [ 1.51870148e+02, -1.34124165e+01, -6.63964939e+00, ..., 3.86457968e+00, 3.44568682e+00, -2.05663879e+02], ..., [-1.31289581e+02, -1.28243113e+01, -3.33981857e+01, ..., 4.42263079e+00, 5.39223528e+00, -2.66478455e+02], [-7.83458023e+01, 4.40543594e+01, 3.59504509e+01, ..., 4.12912798e+00, 1.47437406e+00, -2.67864655e+02], [ 3.04511051e+01, 1.47859955e+02, 1.61898727e+02, ..., -2.52271164e+02, -2.55292587e+02, -3.83337128e+02]],

    [[-1.58220177e+01, -3.06688843e+01, -3.00492077e+01, ...,
      -3.19152451e+01, -3.08211956e+01, -4.19827347e+01],
     [-1.85093632e+01, -3.61226463e+01, -3.44978371e+01, ...,
      -3.92450104e+01, -3.79475136e+01, -5.40956421e+01],
     [-1.91209679e+01, -3.49747925e+01, -3.33080101e+01, ...,
      -3.95044479e+01, -3.87038498e+01, -5.48260994e+01],
      ...

On the other hand, your Keras output of conv1_1 seems to have all vectors taking the same value, which is clearly unusual.

It is likely that your problem is dependent on the implementation in Keras.

Could you please check if there are any errors in the preprocessing of the input image? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

KamitaniLab / brain-decoding-cookbook-public

big errors #2

Functions

Network settings -------------------------------------------------------

device = 'cuda:0'

Initial image for the optimization (here we use the mean of ilsvrc_2012_mean.npy as RGB values)

Feature SD estimated from true DNN features of 10000 images

Feature upper/lower bounds

Initial features -------------------------------------------------------

Setup results directory ------------------------------------------------

Set reconstruction options ---------------------------------------------

The initial image for the optimization (setting to None will use random noise as initial image)

Get images if images is None

Load DNN features

Images loop