[feature request] How to implement torch.nn.functional.grid_sample padding mode in Tensorflow?

Eurus-Holmes commented 5 years ago

https://stackoverflow.com/questions/52888146/what-is-the-equivalent-of-torch-nn-functional-grid-sample-in-tensorflow-numpy

How to implement padding mode?

System information

TensorFlow version (use command below): 1.10.0

Describe the problem

How to implement torch.nn.functional.grid_sample padding mode in Tensorflow?

Source code / logs

https://github.com/tensorflow/models/blob/master/research/transformer/spatial_transformer.py

tensorflowbutler commented 5 years ago

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks. What is the top-level directory of the model you are using Have I written custom code OS Platform and Distribution TensorFlow installed from Bazel version CUDA/cuDNN version GPU model and memory Exact command to reproduce

Eurus-Holmes commented 5 years ago

https://stackoverflow.com/questions/52888146/what-is-the-equivalent-of-torch-nn-functional-grid-sample-in-tensorflow-numpy

How to implement padding mode?

System information

TensorFlow version (use command below): 1.10.0
What is the top-level directory of the model you are using: research/transformer/spatial_transformer.py
Have I written custom code: N/A
OS Platform and Distribution: MacOS10.14.5
TensorFlow installed from: source
Bazel version: N/A
CUDA/cuDNN version: N/A
GPU model and memory: N/A
Exact command to reproduce: N/A

Describe the problem

How to implement torch.nn.functional.grid_sample padding mode in Tensorflow?

Source code / logs

https://github.com/tensorflow/models/blob/master/research/transformer/spatial_transformer.py

OrkhanHI commented 3 years ago

any updates?

Feynman1999 commented 3 years ago

any updates?

dexter2406 commented 3 years ago

Found a solution. Works for me:) @OrkhanHI @Eurus-Holmes @Feynman1999

def bilinear_sampler(img, coords):

    H = tf.shape(img)[1]
    W = tf.shape(img)[2]
    max_y = tf.cast(H - 1, 'int32')
    max_x = tf.cast(W - 1, 'int32')

    # ----------------- Changes below -------------------------
    # -> padding_mode = 'border'
    # "#o" means original,  "#t" means modified
    # zero = tf.zeros([], dtype='int32')     #o
    zero = tf.zeros([1], dtype=tf.int32)     #t
    eps = tf.constant([0.5], 'float32')      #t

    # rescale x and y to [0, W-1/H-1]
    x, y = coords[:, ..., 0], coords[:, ..., 1]
    x = tf.cast(x, 'float32')
    y = tf.cast(y, 'float32')

    x = 0.5 * ((x + 1.0) * tf.cast(max_x - 1, 'float32'))
    y = 0.5 * ((y + 1.0) * tf.cast(max_y - 1, 'float32'))
    x = tf.clip_by_value(x, eps, tf.cast(max_x, tf.float32) - eps)   #t
    y = tf.clip_by_value(y, eps, tf.cast(max_y, tf.float32) - eps)   #t
    # -------------- Changes above --------------------

    # grab 4 nearest corner points for each (x_i, y_i)
    x0 = tf.cast(tf.floor(x), 'int32')
    x1 = x0 + 1
    y0 = tf.cast(tf.floor(y), 'int32')
    y1 = y0 + 1

    # clip to range [0, H-1/W-1] to not violate img boundaries
    x0 = tf.clip_by_value(x0, zero, max_x)
    x1 = tf.clip_by_value(x1, zero, max_x)
    y0 = tf.clip_by_value(y0, zero, max_y)
    y1 = tf.clip_by_value(y1, zero, max_y)

    # get pixel value at corner coords
    Ia = get_pixel_value(img, x0, y0)
    Ib = get_pixel_value(img, x0, y1)
    Ic = get_pixel_value(img, x1, y0)
    Id = get_pixel_value(img, x1, y1)

    # recast as float for delta calculation
    x0 = tf.cast(x0, 'float32')
    x1 = tf.cast(x1, 'float32')
    y0 = tf.cast(y0, 'float32')
    y1 = tf.cast(y1, 'float32')

    # calculate deltas
    wa = (x1 - x) * (y1 - y)
    wb = (x1 - x) * (y - y0)
    wc = (x - x0) * (y1 - y)
    wd = (x - x0) * (y - y0)

    # add dimension for addition
    wa = tf.expand_dims(wa, axis=3)
    wb = tf.expand_dims(wb, axis=3)
    wc = tf.expand_dims(wc, axis=3)
    wd = tf.expand_dims(wd, axis=3)

    # compute output
    out = tf.add_n([wa * Ia, wb * Ib, wc * Ic, wd * Id])
    return out