Question about CPU out of memory

[Youskrpig]

Hi, I run the experiments with my own dataset, which the number of training samples is about 1000 and the number of testing samples is about 1500. but in the process of embedding_concat (function F.fold)，cpu out of memory. Any suggestions?

[Youskrpig]

z = F.fold(z, kernel_size=s, output_size=(H1, W1), stride=s) File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 3860, in fold return torch._C._nn.col2im(input, _pair(output_size), _pair(kernel_size), RuntimeError: [enforce fail at CPUAllocator.cpp:65] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 52626456576 bytes. Error code 12 (Cannot allocate memory)

[Chavo11]

I did this as a workaround:

    embedding_vectors = test_outputs['layer1']
    # randomly select d dimension
    condition = idx < embedding_vectors.shape[1]
    idx_1 = idx[condition]
    embedding_vectors = torch.index_select(embedding_vectors, 1, idx_1)

    for layer_name in ['layer2', 'layer3']:
        emb_vect = test_outputs[layer_name]
        condition1 = embedding_vectors.shape[1] <= idx
        condition2 = idx < (embedding_vectors.shape[1] + emb_vect.shape[1])
        condition = torch.logical_and(condition1, condition2)
        idx_1 = idx[condition]
        idx_1 -= embedding_vectors.shape[1]
        emb_vect = torch.index_select(emb_vect, 1, idx_1)
        embedding_vectors = embedding_concat(embedding_vectors, emb_vect)

Practically I'm dividing the expensive concat operation into "multiple stages". It helped me out with the memory issue. Also I moved this out of the for cycle in plot_figs():

fig_img, ax_img = plt.subplots(1, 5, figsize=(12, 3))
fig_img.subplots_adjust(right=0.9)
norm = matplotlib.colors.Normalize(vmin=vmin, vmax=vmax)

Because it was causing a huge memory leak (16-20 gb for the whole dataset)

[Youskrpig]

I did this as a workaround:

    embedding_vectors = test_outputs['layer1']
    # randomly select d dimension
    condition = idx < embedding_vectors.shape[1]
    idx_1 = idx[condition]
    embedding_vectors = torch.index_select(embedding_vectors, 1, idx_1)

    for layer_name in ['layer2', 'layer3']:
        emb_vect = test_outputs[layer_name]
        condition1 = embedding_vectors.shape[1] <= idx
        condition2 = idx < (embedding_vectors.shape[1] + emb_vect.shape[1])
        condition = torch.logical_and(condition1, condition2)
        idx_1 = idx[condition]
        idx_1 -= embedding_vectors.shape[1]
        emb_vect = torch.index_select(emb_vect, 1, idx_1)
        embedding_vectors = embedding_concat(embedding_vectors, emb_vect)

Practically I'm dividing the expensive concat operation into "multiple stages". It helped me out with the memory issue. Also I moved this out of the for cycle in plot_figs():

fig_img, ax_img = plt.subplots(1, 5, figsize=(12, 3))
fig_img.subplots_adjust(right=0.9)
norm = matplotlib.colors.Normalize(vmin=vmin, vmax=vmax)

Because it was causing a huge memory leak (16-20 gb for the whole dataset)

Thanks for your reply. It seems that firstly you index_select for every-layer feature and then concat. It helps. but in my experimental condition, i found the C,H,W of feature is not large，but the B is. in ( z = torch.zeros(B, C1 + C2, x.size(2), H2, W2) ) . I split the training samples into parts and concat them after part feature embedding_cat. In addition, random choose feature dimension is not so good. For details, refer to the latest paper: "Semi-orthogonal Embedding for Efficient Unsupervised Anomaly Segmentation"

[jackft2]

hi i met this problem in training phase. and i change this code to your version. but i met

File "main.py", line 337, in main() File "main.py", line 128, in main condition2 = idx < (embedding_vectors.shape[1] + emb_vect.shape[1]) AttributeError: 'list' object has no attribute 'shape'

[nikkymen]

@Youskrpig

In addition, random choose feature dimension is not so good. For details, refer to the latest paper: "Semi-orthogonal Embedding for Efficient Unsupervised Anomaly Segmentation"

Hello, do you understand the essence proposed by the authors of the article?

As far as I understand, instead of random sampling of d features, calculating and storing covariance matrices of size dxd, they propose to calculate the expression

, , , where S is a Fxk random matrix with standard normal distribution,

and use the result D when calculating the score map.

I'm right?

[Youskrpig]

@Youskrpig

In addition, random choose feature dimension is not so good. For details, refer to the latest paper: "Semi-orthogonal Embedding for Efficient Unsupervised Anomaly Segmentation"

Hello, do you understand the essence proposed by the authors of the article?

As far as I understand, instead of random sampling of d features, calculating and storing covariance matrices of size dxd, they propose to calculate the expression

^{-1}W^T), )), , where S is a Fxk random matrix with standard normal distribution,

and use the result D when calculating the score map.

I'm right?

Yeah，I think you're right! Have you tried it ?

[nikkymen]

@Youskrpig

Yeah，I think you're right! Have you tried it ?

Nope, not yet, but it sounds promising.

[Pangoraw]

S is a Fxk random matrix with standard normal distribution,

and use the result D when calculating the score map.

I think you can simply consider the reduced dataset of size k as being the original dataset of size FxN multiplied by the semi-orthogonal matrix:

You can then compute the covariance matrices as usual:

And the score will be the regular Mahalanobis distance:

Note: you still need to substract the mean vectors but I removed them for an easier reading. I have implemented it at Pangoraw/SemiOrthogonal using the outer product for the covariance so the memory footprint is not dependent on the number of training samples (no concat) which could also be useful for this issue.

[zhangyumo95]

I made the following changes to the code： ` # Embedding concat

embedding_vectors = test_outputs['layer1']

    # for layer_name in ['layer2', 'layer3']:
    #     embedding_vectors = embedding_concat(embedding_vectors, test_outputs[layer_name])

    tmp = test_outputs['layer1'].shape[0]
    embedding_vectors1 = None
    for tmp_i in range(0, tmp, 32):
        if tmp_i + 32 >= tmp:
            embedding_vectors = test_outputs['layer1'][tmp_i:]
        else:
            embedding_vectors = test_outputs['layer1'][tmp_i:tmp_i + 32]
        for layer_name in ['layer2', 'layer3']:
            if tmp_i + 32 >= tmp:
                embedding_vectors = embedding_concat(embedding_vectors, test_outputs[layer_name][tmp_i:])
            else:
                embedding_vectors = embedding_concat(embedding_vectors, test_outputs[layer_name][tmp_i:tmp_i + 32])
        embedding_vectors = torch.index_select(embedding_vectors, 1, idx)
        if tmp_i != 0:
            embedding_vectors1 = torch.cat([embedding_vectors1, embedding_vectors], 0)
        else:
            embedding_vectors1 = embedding_vectors
    embedding_vectors = embedding_vectors1

    # randomly select d dimension
    # embedding_vectors = torch.index_select(embedding_vectors, 1, idx)`

[haobo827]

hi i met this problem in training phase. and i change this code to your version. but i met

File "main.py", line 337, in main() File "main.py", line 128, in main condition2 = idx < (embedding_vectors.shape[1] + emb_vect.shape[1]) AttributeError: 'list' object has no attribute 'shape'

np.array(list).shape[1]