Lycus99 / MT-FiST

2 stars 0 forks source link

mean and std in transform.normalize #3

Closed cascat0 closed 1 year ago

cascat0 commented 1 year ago

https://github.com/Lycus99/MT-FiST/blob/8aaf4fe8c2c0bf254995c457864d86c12eba2e4d/train5.py#L306 您好,这组均值和标准差的计算代码可以release一下吗?我自己写了一个来统计CholecT50 official spilt的训练集的均值和标准差,但是内存爆了……想问问您是怎么计算的。谢谢。

cascat0 commented 1 year ago

import numpy as np from PIL import Image import os import glob

images = [] dataset_dir = '../CholecT50/' train_videos = [1, 15, 26, 40, 52, 65, 79, 2, 18, 27, 43, 56, 66, 92, 4, 22, 31, 47, 57, 68, 96, 5, 23, 35, 48, 60, 70, 103, 13, 25, 36, 49, 62, 75, 110] train_records = ['VID{}'.format(str(v).zfill(2)) for v in train_videos] for video in train_records: folder_path = os.path.join(dataset_dir, 'videos', video) image_paths = glob.glob(os.path.join(folder_path, '*.png')) for image_path in image_paths: image = Image.open(image_path).resize((256, 256)) image = np.array(image) images.append(image)

images = np.array(images)

mean = np.mean(images, axis=(0, 1, 2)) std = np.std(images, axis=(0, 1, 2))

print('均值:', mean) print('标准差:', std)

Traceback (most recent call last): File "/home/cv5/matser/fob/lab.py", line 25, in std = np.std(images, axis=(0, 1, 2)) File "<__array_function__ internals>", line 180, in std File "/home/cv5/anaconda3/envs/torch110/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 3573, in std return _methods._std(a, axis=axis, dtype=dtype, out=out, ddof=ddof, File "/home/cv5/anaconda3/envs/torch110/lib/python3.8/site-packages/numpy/core/_methods.py", line 265, in _std ret = _var(a, axis=axis, dtype=dtype, out=out, ddof=ddof, File "/home/cv5/anaconda3/envs/torch110/lib/python3.8/site-packages/numpy/core/_methods.py", line 233, in _var x = asanyarray(arr - arrmean) numpy.core._exceptions.MemoryError: Unable to allocate 107. GiB for an array with shape (72815, 256, 256, 3) and data type float64

Lycus99 commented 1 year ago

https://github.com/YuemingJin/MTRCNet-CL/blob/2c9f11cda8a6b6f4099a6caffb5f5a6a536cf61f/pytorch1.2.0/train_phase%2Btool.py#L272

均值和标准差参考的是之前的工作 MTRCNet-CL,我认为也可以使用ImageNet统计得到的均值及标准差,应该对结果影响不大