Closed MdRanaSarkar closed 2 weeks ago
Hello,
This is also an apparent issue on the Satlas website, see screenshot below.
We experimented with some post-processing to smooth out the boundaries of each tile, but found that this often over-smoothed and interfered with the semantic accuracy of the super-res outputs. You could try adding a loss to training that enforces some similarity between tile bounds, but we did not get something to work well.
I have managed to resolve this issue by running inference on the tile with overlap. For instance, the size of the patch is 256, and I used a stride of 128. Then I convert the RGB output from each patch to LAB color space and compute the average color with weighting.
To elaborate on the weighting process:
Overlap Count: This tracks the number of times each pixel in the large image has been covered by a patch. As we stitch patches back together, each pixel's value is accumulated, and the overlap count is incremented.
Distance Transform Weights: These weights are created based on the distance from the patch edges, using the Euclidean Distance Transform (EDT). The process is as follows:
scipy.ndimage.distance_transform_edt
function is applied to this matrix to compute the distance of each pixel from the nearest zero (edge of the patch).The final color values are computed by accumulating the weighted LAB color values for each pixel and dividing by the total weight for that pixel. This technique blends the overlapping regions seamlessly and avoids visible seams between patches.
Here is a snippet of the relevant code for applying the weights and stitching the patches together:
def apply_weights(image_shape, chunk_size, overlap):
weights = np.zeros((chunk_size, chunk_size), dtype=np.float32)
center = (chunk_size - overlap) // 2
weights[center:center+overlap, center:center+overlap] = 1
weights = scipy.ndimage.distance_transform_edt(weights)
weights /= weights.max()
return weights
def stitch_in_memory(chunks, img_height, img_width, chunk_size=256, overlap=32):
stitched_image = np.zeros((img_height, img_width, 3), dtype=np.float32)
overlap_count = np.zeros((img_height, img_width), dtype=np.float32)
weights = apply_weights((img_height, img_width), chunk_size, overlap)
grid_size_y = (img_height - overlap) // (chunk_size - overlap)
grid_size_x = (img_width - overlap) // (chunk_size - overlap)
idx = 0
for i in range(grid_size_y + 1):
for j in range(grid_size_x + 1):
load = chunks[idx]
idx += 1
y_start = i * (chunk_size - overlap)
x_start = j * (chunk_size - overlap)
y_end = min(y_start + chunk_size, img_height)
x_end = min(x_start + chunk_size, img_width)
load = load[:y_end - y_start, :x_end - x_start, :]
lab_load = color.rgb2lab(load)
for c in range(3):
stitched_image[y_start:y_end, x_start:x_end, c] += lab_load[..., c] * weights[:y_end-y_start, :x_end-x_start]
overlap_count[y_start:y_end, x_start:x_end] += weights[:y_end-y_start, :x_end-x_start]
overlap_count[overlap_count == 0] = 1
averaged_lab = np.zeros_like(stitched_image)
for c in range(3):
averaged_lab[..., c] = stitched_image[..., c] / overlap_count
stitched_image_rgb = (color.lab2rgb(averaged_lab) * 255).astype(np.uint8)
return stitched_image_rgb
By following this method, the final super-resolved image is free from noticeable seams, providing a smooth and visually consistent output.
Note: I trained the model to super-resolve SPOT satellite imagery to aerial resolution.
I've been experiencing a consistent issue with super-resolution results where the boundaries between separate tiles are clearly visible. Despite following various preprocessing steps and using different model weights, the problem persists. Below are the details of my preprocessing steps and normalization methods.
For TCI Bands (Using 8-S2 Weights) Normalization Steps 1:
Normalization Steps 2:
For TCI + Non-TCI Bands (Using 8-S2 Weights) Normalization Steps 1:
Normalization Steps 2:
Note: To subset & resample, I’ve used SNAP tools.
For Resampling: Upsampling method: nearest Downsampling method: first Flag Downsampling method: First Reference Band: B02
Additional Attempt: I've also tried using min-max normalization but still faced the same issue.
Is there any best & ideal preprocessing step ? Can someone help me understand why the tile boundaries are so visible and how I can mitigate this problem? Any suggestions or insights would be greatly appreciated. Thank you!