smermet
commented
2 years ago This code was picked up in different projects available on GitHub (eg in RobustMFSRforEO) and I still faced the same error.
I modified the code so that the input data corresponds to what is expected for the conv1d function, hoping not to have introduced an error... Changes are commented with ##.
def lanczos_shift(img, shift, p=3, a=3):
'''
Shifts an image by convolving it with a Lanczos kernel.
Lanczos interpolation is an approximation to ideal sinc interpolation,
by windowing a sinc kernel with another sinc function extending up to a
few number of its lobes (typically a=3).
Args:
img : tensor (batch_size, channels, height, width), the images to be shifted
shift : tensor (batch_size, 2) of translation parameters (dy, dx)
p : int, padding width prior to convolution (default=3)
a : int, number of lobes in the Lanczos interpolation kernel (default=3)
Returns:
I_s: tensor (batch_size, channels, height, width), shifted images
'''
## These lines are added from RobustMFSRforEO
B, C, H, W = img.shape
# Because examples and channels are interleaved in dim 1.
shift = shift.repeat(C, 1).reshape(B * C, 2)
img = img.view(1, B * C, H, W)
dtype = img.dtype
if len(img.shape) == 2:
img = img[None, None].repeat(1, shift.shape[0], 1, 1) # batch of one image
elif len(img.shape) == 3: # one image per shift
assert img.shape[0] == shift.shape[0]
img = img[None, ]
# Apply padding
padder = torch.nn.ReflectionPad2d(p) # reflect pre-padding
I_padded = padder(img)
I_padded_reshape = I_padded.view(I_padded.shape[0],I_padded.shape[1],-1) ## image reshaped to work with conv1d
# Create 1D shifting kernels
y_shift = shift[:, [0]]
x_shift = shift[:, [1]]
k_y = (lanczos_kernel(y_shift, a=a, N=None, dtype=dtype)
.flip(1) # flip axis of convolution
)[:, None, :, None].squeeze(3) ## expand dims to get shape (batch, channels, y_kernel) instead of (batch, channels, y_kernel, 1)
k_x = (lanczos_kernel(x_shift, a=a, N=None, dtype=dtype)
.flip(1)
)[:, None, None, :].squeeze(2) ## shape (batch, channels, x_kernel) instead of (batch, channels, 1, x_kernel)
# Apply kernels
I_s = torch.conv1d(I_padded_reshape,
groups=k_y.shape[0],
weight=k_y,
padding=k_y.shape[2] // 2) ## previously : [k_y.shape[2] // 2, 0]
I_s = torch.conv1d(I_s,
groups=k_x.shape[0],
weight=k_x,
padding=k_x.shape[2] // 2) ## previously : [0, k_x.shape[3] // 2]
I_s = I_s.view(B, C, H+2*p, W+2*p) ## result reshaped in image format
I_s = I_s[..., p:-p, p:-p] # remove padding
return I_sPlus : in DeepNetworks/ShiftNet.py I removed the .transpose(0, 1) whose format does not correspond to the documentation
gbrzeczek
Laymanpython
yunseok624
I want to use your code with the Proba-V dataset, but I'm facing the following error.
$ python src/train.py --config config/config.json 0%| | 0/261 [00:00<?, ?it/s] 0%| | 0/400 [00:00<?, ?it/s] Traceback (most recent call last): File "[...]/HighRes-net/src/train.py", line 308, in <module> main(config) File "[...]/HighRes-net/src/train.py", line 294, in main trainAndGetBestModel(fusion_model, regis_model, optimizer, dataloaders, baseline_cpsnrs, config) File "[...]/HighRes-net/src/train.py", line 180, in trainAndGetBestModel srs_shifted = apply_shifts(regis_model, srs, shifts, device)[:, 0] File "[...]/HighRes-net/src/train.py", line 61, in apply_shifts new_images = shiftNet.transform(thetas, images, device=device) File "[...]/HighRes-net/src/DeepNetworks/ShiftNet.py", line 96, in transform new_I = lanczos.lanczos_shift(img=I.transpose(0, 1), File "[...]/HighRes-net/src/lanczos.py", line 96, in lanczos_shift I_s = torch.conv1d(I_padded, RuntimeError: Expected 2D (unbatched) or 3D (batched) input to conv1d, but got input of size: [1, 4, 202, 202]Here are the different values or shapes which are passed in the conv1d function :
I_paddedinput shape :torch.Size([1, 4, 202, 202])k_y.shape[0]andk_x.shape[0]groups number :4k_yandk_xweights shapes :torch.Size([4, 1, 7, 1])(andtorch.Size([4, 1, 1, 7]))[k_y.shape[2] // 2, 0]and[0, k_x.shape[3] // 2]padding values :[3, 0]and[3, 0]I used the default config.json, except for the following parameters.
I tried to squeeze the 1st dim of img, the 2nd of weights and to specify a simple int value for padding to avoid the different error messages, but all I finally had is this new RuntimeError. 'Given groups=4, weight of size [4, 7, 1], expected input[4, 202, 202] to have 28 channels, but got 202 channels instead'
Any clue to help me?