Closed Laymanpython closed 6 months ago
Can you please share the config file you're using and the structure of your downloaded PROBA-V dataset?
Thanks for your reply,the config file I used was probav_highresnet.yml.A nd I just changed the path of data and dataset file.The PROBA-V dataset I used was from RAMs[https://github.com/EscVM/RAMS/tree/master/probav_data] .And I just use validation and train datasets.There was no problem when I use esrgan to run .Thanks again!
I found your paper didn't give the results of proba-v and mus2 datasets.Will you open it in the future? And I found you used opencv to read the image in probav,every pixel will be a int in [0,255].But when I used skimage to read it, every pixel will be a int in [0,65535]. If you convert a int in [0,65535] to [0,255], it will be a float not int. Will processing data in this way cause a loss of accuracy? Thanks~
Can you please share the config file you're using and the structure of your downloaded PROBA-V dataset?
And when I set os.environ['CUDA_VISIBLE_DEVICES'] = “1,2,3,4” in train.py, I found that the "cuda:0" will be used too. Maybe should I set in terminal?
Thanks for your reply,the config file I used was probav_highresnet.yml.A nd I just changed the path of data and dataset file.The PROBA-V dataset I used was from RAMs[https://github.com/EscVM/RAMS/tree/master/probav_data] .And I just use validation and train datasets.There was no problem when I use esrgan to run .Thanks again!
![]()
Here are screenshots of my PROBA-V data directory. It looks similar to yours, but I took a subset of the imgsets in train/NIR/ and moved them to train/NIR/val/. Looks like you changed the code to work with your setup already. I just ran probav_esrgan.yml and probav_highresnet.yml (from the experiments branch) in my environment and both are working. I'm not sure what is causing the error you are getting.
I found your paper didn't give the results of proba-v and mus2 datasets.Will you open it in the future? And I found you used opencv to read the image in probav,every pixel will be a int in [0,255].But when I used skimage to read it, every pixel will be a int in [0,65535]. If you convert a int in [0,65535] to [0,255], it will be a float not int. Will processing data in this way cause a loss of accuracy? Thanks~
I was worried about that. I ran parallel experiments, one using opencv and the other using skimage. The ultimate performance was very similar. I originally switched to opencv because of some memory leakage issues that I had with skimage, but maybe the risk of accuracy loss is not worth it.
And I did not report results of MuS2 because it is just a test set. I think I will try evaluating on MuS2 with models pretrained on different datasets (S2NAIP, WorldStrat, ...) but I don't know if that will tell us anything interesting.
Can you please share the config file you're using and the structure of your downloaded PROBA-V dataset?
And when I set os.environ['CUDA_VISIBLE_DEVICES'] = “1,2,3,4” in train.py, I found that the "cuda:0" will be used too. Maybe should I set in terminal?
Ah, yeah. I would just export CUDA_VISIBLE_DEVICES=1,2,3,4
in the terminal.
Can you please share the config file you're using and the structure of your downloaded PROBA-V dataset?
And when I set os.environ['CUDA_VISIBLE_DEVICES'] = “1,2,3,4” in train.py, I found that the "cuda:0" will be used too. Maybe should I set in terminal?
Ah, yeah. I would just
export CUDA_VISIBLE_DEVICES=1,2,3,4
in the terminal.
Thanks for your reply! Perhaps errors I met were caused by my environment.When I used python3.8,all packages will be downloaded successfully.But using python3.9 won't.Thanks again,I will give the solution to solve it in this issue,as long as I can solve it haha ~
Can you please share the config file you're using and the structure of your downloaded PROBA-V dataset?
And when I set os.environ['CUDA_VISIBLE_DEVICES'] = “1,2,3,4” in train.py, I found that the "cuda:0" will be used too. Maybe should I set in terminal?
Ah, yeah. I would just
export CUDA_VISIBLE_DEVICES=1,2,3,4
in the terminal.
Could you please tell me your version number of environment? And why is the cPSNR you gave in the paper so small? I train it in probav,finding the val's cpsnr will reach almost 44 .
Can you please share the config file you're using and the structure of your downloaded PROBA-V dataset?
And when I set os.environ['CUDA_VISIBLE_DEVICES'] = “1,2,3,4” in train.py, I found that the "cuda:0" will be used too. Maybe should I set in terminal?
Ah, yeah. I would just
export CUDA_VISIBLE_DEVICES=1,2,3,4
in the terminal.
Hi,sorry to bother you too many times haha,I have solve it! Because the class of highresnet inherits SSRNet class.So there are some block didn't attend the training of HighResnet, their gradient will be none.Just inheriting nn.Module will be okay, and every block were written in Highresnet's file.Perhaps my meaning is not clear,my English is so poor.This is a link to slove it: https://blog.csdn.net/YuhsiHu/article/details/134185740
In the paper, I used a crop border of 0, but realized I should use a crop border of 4 to match previous works. I believe all cPSNR values should be higher with the larger crop border.
Can you please share the config file you're using and the structure of your downloaded PROBA-V dataset?
And when I set os.environ['CUDA_VISIBLE_DEVICES'] = “1,2,3,4” in train.py, I found that the "cuda:0" will be used too. Maybe should I set in terminal?
Ah, yeah. I would just
export CUDA_VISIBLE_DEVICES=1,2,3,4
in the terminal.Hi,sorry to bother you too many times haha,I have solve it! Because the class of highresnet inherits SSRNet class.So there are some block didn't attend the training of HighResnet, their gradient will be none.Just inheriting nn.Module will be okay, and every block were written in Highresnet's file.Perhaps my meaning is not clear,my English is so poor.This is a link to slove it: https://blog.csdn.net/YuhsiHu/article/details/134185740
Oh I wonder if it is because HighResNet inherits the self.resize()
module from SRCNN, but does not use it...
Can you please share the config file you're using and the structure of your downloaded PROBA-V dataset?
And when I set os.environ['CUDA_VISIBLE_DEVICES'] = “1,2,3,4” in train.py, I found that the "cuda:0" will be used too. Maybe should I set in terminal?
Ah, yeah. I would just
export CUDA_VISIBLE_DEVICES=1,2,3,4
in the terminal.Hi,sorry to bother you too many times haha,I have solve it! Because the class of highresnet inherits SSRNet class.So there are some block didn't attend the training of HighResnet, their gradient will be none.Just inheriting nn.Module will be okay, and every block were written in Highresnet's file.Perhaps my meaning is not clear,my English is so poor.This is a link to slove it: https://blog.csdn.net/YuhsiHu/article/details/134185740
Oh I wonder if it is because HighResNet inherits the
self.resize()
module from SRCNN, but does not use it...
emmm. I think it was caused by mask_encoder and some blocks: self.mask_encoder = nn.Sequential( OneHot(num_classes=12), DoubleConv2d(in_channels=self.mask_channels, out_channels=1, kernel_size=3), nn.Sigmoid(), )
# Fusion
self.doubleconv2d = DoubleConv2d(
in_channels=hidden_channels * revisits, # revisits as channels
out_channels=hidden_channels,
kernel_size=kernel_size,
use_batchnorm=self.use_batchnorm,
)
self.residualblocks = nn.Sequential(
*(
ResidualBlock(
in_channels=hidden_channels,
kernel_size=kernel_size,
use_batchnorm=self.use_batchnorm,
)
for _ in range(residual_layers)
)
)
so I rewrite the highresnet inheriting from nn.Module: `""" Adapted from: https://https://github.com/worldstrat/worldstrat/blob/main/src/modules.py Authors: Ivan Oršolić, Julien Cornebise, Ulf Mertens, Freddie Kalaitzis """ from basicsr.utils.registry import ARCH_REGISTRY from .srcnn_arch import SRCNN from .arch_util import * from kornia.geometry.transform import Resize
@ARCH_REGISTRY.register() class HighResNet(nn.Module): """ High-resolution CNN. Inherits as many elements from SRCNN as possible for as fair a comparison:
Encoder: nn.Sequential(DoubleConv2d, ResidualBlock) """
def init(self, in_channels, mask_channels, revisits, hidden_channels, out_channels, kernel_size, residual_layers, output_size, zoom_factor, sr_kernel_size, use_reference_frame=False, skip_paddings=True, **kws) -> None: super().init()
self.in_channels = 2 * in_channels if use_reference_frame else in_channels
self.mask_channels = mask_channels
self.revisits = revisits
self.hidden_channels = hidden_channels
self.out_channels = out_channels
self.kernel_size = kernel_size
self.residual_layers = residual_layers
self.output_size = output_size
self.zoom_factor = zoom_factor
self.sr_kernel_size = sr_kernel_size
self.use_batchnorm = False
self.use_reference_frame = use_reference_frame
self.skip_paddings = skip_paddings
self.fusion = RecursiveFusion(
in_channels=self.hidden_channels,
kernel_size=self.kernel_size,
revisits=self.revisits,
)
self.encoder = DoubleConv2d(
in_channels=self.in_channels,
out_channels=self.hidden_channels,
kernel_size=self.kernel_size,
use_batchnorm=self.use_batchnorm,
)
self.sr = PixelShuffleBlock(
in_channels=self.hidden_channels,
out_channels=self.out_channels,
kernel_size=self.kernel_size,
sr_kernel_size=self.sr_kernel_size,
zoom_factor=self.zoom_factor,
use_batchnorm=self.use_batchnorm,
)
self.resize = Resize(
self.output_size,
interpolation="bilinear",
align_corners=False,
antialias=True,
)
def forward(self, x, y=None, mask=None): """ Forward pass of the High-resolution CNN.
Parameters
----------
x : Tensor
The input tensor (low-res revisits).
Shape: (batch_size, revisits, channels, height, width).
y : Tensor, optional
The target tensor (high-res revisits).
Shape: (batch_size, 1, channels, height, width).
mask : Tensor, optional
The mask tensor.
Shape: (batch_size, revisits, mask_channels, height, width).
Returns
-------
Tensor
The output tensor (super-resolved images).
Shape: (batch_size, 1, channels, height, width).
"""
hidden_channels = self.hidden_channels
if self.use_reference_frame:
x = self.compute_and_concat_reference_frame_to_input(x)
# Note: we could put all these layers in a Sequential, but we
# would lose the logging abilities for inspection and lose
# on readability.
batch_size, revisits, channels, height, width = x.shape
x = x.view(batch_size * revisits, channels, height, width)
# Encoded shape: (batch_size * revisits, hidden_channels, height, width)
x = self.encoder(x)
x, mask_channels = self.encode_and_concatenate_masks_to_input(
x, mask, batch_size, revisits, height, width
)
x = x.view(batch_size, revisits, hidden_channels + mask_channels, height, width)
# Fused shape: (batch_size, hidden_channels, height, width)
x = self.fusion(x)
# Super-resolved shape:
# (batch_size, out_channels, height * zoom_factor, width * zoom_factor)
x = self.sr(x)
# Ensure output size of (batch_size, channels, height, width)
x = self.resize(x)
# Pad with empty revisit dimension: (batch_size, 1, channels, height, width)
x = x[:, None]
return x
def compute_and_concat_reference_frame_to_input(self, x):
reference_frame = self.reference_frame(x).expand_as(x)
# Concatenated shape: (batch_size, revisits, 2*channels, height, width)
x = torch.cat([x, reference_frame], dim=-3)
return x
def encode_and_concatenate_masks_to_input( self, x, mask, batch_size, revisits, height, width ): if mask is not None: mask, mask_channels = mask, self.mask_channels mask = mask.view(batch_size * revisits, mask_channels, height, width)
mask = self.mask_encoder(mask)
mask_channels = mask.shape[-3]
# Concatenated shape:
# (batch_size * revisits, hidden_channels+mask_channels, height, width)
x = torch.cat([x, mask], dim=-3)
else:
mask_channels = 0
return x, mask_channels
`
Ah, okay. I am trying to figure out why I can't replicate your error, but thank you for sharing your code - I will try it and make sure it produces similar results.
Ah, okay. I am trying to figure out why I can't replicate your error, but thank you for sharing your code - I will try it and make sure it produces similar results. The python version I used is 3.8,and I ran the code yesterday-all results are similar.Thank you for opening your work,this is an excellent work!~
Awesome, thank you for digging into the code! I'll push a fix to the error you mention, with your change.
Hi! Dear authors,when I use multi gpus to train highresnet using Probav dataset, I found a bug "RuntimeError: Function BroadcastBackward returned an invalid gradient at index 6 - got [0] but expected shape compatible with [1, 0, 3, 3]", I'm find the method to fix it. If you have solution, please tell me,thanks~