search

shayzweig / InterpoNet

A brain inspired neural network for optical flow dense interpolation
GNU General Public License v3.0
18 stars 6 forks source link

Unable to regenerate the same FlowFields matches in examples folder #3

Open fperezgamonal opened 4 years ago

fperezgamonal commented 4 years ago

Dear Shay,

I'm sorry for bothering you again but I have a quick question about the FlowFields matches you used (in case you recall).

Do you remember if you used the default parameters of the method to compute the matches? As far as I know, the FlowFields authors provide 3 different binaries, two of which use SIFTflow and one which does not. Each of them uses a different parameter S (number of channels*), (S=10, S=3 or S=30). I have tried to generate the matches for all of them in order to check which matched the sample matches you provide in the 'example' folder to no avail as I obtain around 10% of the total number of matches (~30k instead of 300k in the Temple3 sample from Sintel you provide).

Alternatively, I'll try to implement the training code for InterpoNet as you suggested before and re-train the architecture from scratch in order to more fairly compare different implementations with yours.

Thank you very much in advance!

Cheers,

Ferran.

*as briefly explained in the FlowFields paper

shayzweig commented 4 years ago

Hi Ferran, Terribly sorry but it has been very long since I have done this. I will look into my material but I am not very optimistic about my chances to recover this piece of information. Best, Shay

On Tue, Oct 29, 2019, 11:15 Ferran Pérez Gamonal notifications@github.com wrote:

Dear Shay,

I'm sorry for bothering you again but I have a quick question about the FlowFields matches you used (in case you recall).

Do you remember if you used the default parameters of the method to compute the matches? As far as I know, the FlowFields authors provide 3 different binaries, two of which use SIFTflow and one which does not. Each of them uses a different parameter S (number of channels*), (S=10, S=3 or S=30). I have tried to generate the matches for all of them in order to check which matched the sample matches you provide in the 'example' folder to no avail as I obtain around 10% of the total number of matches (~30k instead of 300k in the Temple3 sample from Sintel you provide).

Alternatively, I'll try to implement the training code for InterpoNet as you suggested before and re-train the architecture from scratch in order to more fairly compare different implementations with yours.

Thank you very much in advance!

Cheers,

Ferran.

*as briefly explained in the FlowFields paper

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/shayzweig/InterpoNet/issues/3?email_source=notifications&email_token=AD3LXSQHFN5KUHACFCO6EATQQ75M7A5CNFSM4JGFTEM2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HU7X4MQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD3LXSUC3LPXL2MSVJOMBSLQQ75M7ANCNFSM4JGFTEMQ .

shayzweig commented 4 years ago

Hi Ferran, Attached is the code I found that I used to create it, specifically look at the function: calc_flowFields_map

I hope this helps. Shay

https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail בלי וירוסים. www.avast.com https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Wed, Oct 30, 2019 at 11:10 PM Shay Zweig shay@intuitionrobotics.com wrote:

Hi Ferran, Terribly sorry but it has been very long since I have done this. I will look into my material but I am not very optimistic about my chances to recover this piece of information. Best, Shay

On Tue, Oct 29, 2019, 11:15 Ferran Pérez Gamonal notifications@github.com wrote:

Dear Shay,

I'm sorry for bothering you again but I have a quick question about the FlowFields matches you used (in case you recall).

Do you remember if you used the default parameters of the method to compute the matches? As far as I know, the FlowFields authors provide 3 different binaries, two of which use SIFTflow and one which does not. Each of them uses a different parameter S (number of channels*), (S=10, S=3 or S=30). I have tried to generate the matches for all of them in order to check which matched the sample matches you provide in the 'example' folder to no avail as I obtain around 10% of the total number of matches (~30k instead of 300k in the Temple3 sample from Sintel you provide).

Alternatively, I'll try to implement the training code for InterpoNet as you suggested before and re-train the architecture from scratch in order to more fairly compare different implementations with yours.

Thank you very much in advance!

Cheers,

Ferran.

*as briefly explained in the FlowFields paper

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/shayzweig/InterpoNet/issues/3?email_source=notifications&email_token=AD3LXSQHFN5KUHACFCO6EATQQ75M7A5CNFSM4JGFTEM2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HU7X4MQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD3LXSUC3LPXL2MSVJOMBSLQQ75M7ANCNFSM4JGFTEMQ .

--

[image: IR_Logo_2lines_1024px-wide_300dpi_Black.png]

Shay Zweig

Head of AI

+972-54-7479977 | www.intuitionrobotics.com

https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail בלי וירוסים. www.avast.com https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

-- coding: utf-8 --

""" Created on Mon Nov 30 13:09:25 2015

@author: shayzweig """ import csv import numpy as np import os import skimage as sk

import matplotlib.pyplot as plt

from flow_utils import flowToColor

from loadFlowSintel import get_MPI_pairs, load_allFiles_SINTEL, SINTEL_HOME, SINTEL_HOME_TEST from loadFlowKitti2015 import get_kitti_pairs, load_kitti_meta, KITTI2015_HOME, KITTI2015_HOME_GS, KITTI_NUM_OF_PAIRS, KITTI2012_HOME, KITTI2012_NUM_OF_PAIRS from loadFlowFC import get_FC_pairs, load_FC_meta, FC_HOME, FC_NUM_OF_PAIRS

import string import cPickle as pickle

import marshal

import time from scipy.interpolate import interp2d, NearestNDInterpolator, LinearNDInterpolator from matplotlib.mlab import griddata

import h5py

import logging logger = logging.getLogger('root')

from scipy import interpolate

SINTEL_FLOW_FIELDS_PATH = 'flowFields/SINTEL/training' SINTEL_FLOW_FIELDS_PATH_PRE = 'flowFields/SINTEL/training/preprocess' SINTEL_FLOW_FIELDS_PATH_CONSISTENCY = 'flowFields/SINTEL/training/Consistency' SINTEL_FLOW_FIELDS_PATH_SALIENCY = 'flowFields/SINTEL/training/Saliency' SINTEL_FLOW_FIELDS_PATH_NET_NOISE = 'flowFields/SINTEL/training/netOutput' SINTEL_WIDTH = 1024 SINTEL_HEIGHT = 436 SINTEL_FF_ATOMIC_SIZE = 1 SINTEL_DATA_PATH = 'Data/SintelFlow/training/'

SINTEL_DATA_PATH_TEST = 'Data/SintelFlow/test/' SINTEL_FLOW_FIELDS_PATH_TEST = 'flowFields/SINTEL/test'

KITTI_FLOW_FIELDS_PATH = 'flowFields/kitti2015/' KITTI_WIDTH = 1242 KITTI_HEIGHT = 375 KITTI_IMAGE_HEIGHT = 46 KITTI_IMAGE_WIDTH = 153

KITTI_FF_ATOMIC_SIZE = 1 KITTI_DATA_PATH = 'Data/kitti2015Flow/'

KITTI_DATA_PATH_TEST = 'Data/kitti2015Flow/test/' KITTI_FLOW_FIELDS_PATH_TEST = 'flowFields/kitti2015/test'

KITTI2012_FLOW_FIELDS_PATH = 'flowFields/kitti2012/' KITTI2012_WIDTH = 1242 KITTI2012_HEIGHT = 375 KITTI2012_IMAGE_HEIGHT = 46 KITTI2012_IMAGE_WIDTH = 153 KITTI2012_FF_ATOMIC_SIZE = 1 KITTI2012_DATA_PATH = 'Data/kitti2012Flow/'

KITTI2012_DATA_PATH_TEST = 'Data/kitti2012Flow/test/' KITTI2012_FLOW_FIELDS_PATH_TEST = 'flowFields/kitti2012/test'

FC_FLOW_FIELDS_PATH = 'flowFields/flyingChairs' FC_WIDTH = 512 FC_HEIGHT = 384 FC_FF_ATOMIC_SIZE = 1 FC_DATA_PATH = 'Data/FCFlow/'

def read_flowFieldsOutputFile_DMformat(atomicPatchSize, filename, width, height): img = np.zeros([height/atomicPatchSize,width/atomicPatchSize,2]) mask = -np.ones([height/atomicPatchSize,width/atomicPatchSize])

x1,y1,x2,y2 = np.loadtxt(filename, dtype=np.float32, delimiter=' ', unpack=True, usecols=(0,1,2,3))

if np.any((x1 - atomicPatchSize / 2) % atomicPatchSize != 0) or np.any((y1 - atomicPatchSize / 2)  % atomicPatchSize != 0):
    logger.error('Index values are: (%s,%s) not a multiple of %d' % (x1, x2, atomicPatchSize))

img[(np.array(y1, dtype=int)/atomicPatchSize),(np.array(x1, dtype=int)/atomicPatchSize),:] = np.stack((x2 - x1, y2 - y1), axis=1)
mask[(np.array(y1, dtype=int)/atomicPatchSize),(np.array(x1, dtype=int)/atomicPatchSize)] = 1
if np.any(np.isnan(img)):
    logger.error("Nan value found")

return img, mask

def read_flowFieldsOutputFile(filename): from loadFlowSintel import load_flow_SINTEL data2d = load_flow_SINTEL(filename) img = np.zeros_like(data2d) mask = -np.ones_like(data2d) img[~np.isinf(data2d)] = data2d[~np.isinf(data2d)] mask[~np.isinf(data2d)] = 1 mask = mask[:, :, 0]

return img, mask

def calc_flowFields_map(imgA_filename,imgB_filename, out_filename, dataset, epsilon=1, for_epic=False, full_map=True, with_outliers=False): param_str = '' if for_epic: param_str += ' -osm %s.txt' % out_filename if full_map: param_str += ' -offf %s.flo' % out_filename if with_outliers: param_str += ' -ouff %s_with_outliers.flo' % out_filename

if dataset == 'SINTEL_clean':
    shell_command = './flowFields/Code/FlowFields ' + imgA_filename + ' ' + imgB_filename + param_str
    shell_command += ' -r 6 -r2 5 -epsilon %f -e 4 -s 50' % epsilon
elif dataset == 'kitti':
    # shell_command = './flowFields/Code/FlowFields ' + imgA_filename + ' ' + imgB_filename + param_str
    shell_command = './flowFields/Code/FlowFields_S_12_S2_18  ' + imgA_filename + ' ' + imgB_filename + param_str
    shell_command += ' -r 3 -r2 2 -epsilon %f -e 9 -s 150 -SIFTFlow' % epsilon
elif dataset == 'middleburry':
    shell_command = './flowFields/Code/FlowFields ' + imgA_filename + ' ' + imgB_filename + param_str
    shell_command += ' -r 5 -r2 4 -epsilon %f -e 7 -s 50' % epsilon
else:
    shell_command = './flowFields/Code/FlowFields ' + imgA_filename + ' ' + imgB_filename + param_str
    shell_command += ' -r 8 -r2 6 -epsilon %f -e 4 -s 50' % epsilon
print('executing shell command:',shell_command)
exit_code = os.system(shell_command)
logger.debug('exit_code',exit_code)

def calc_flowFields_full_map(imgA_filename,imgB_filename, out_filename, dataset, epsilon=1):

shell_command = './flowFields/Code/FlowFields ' + imgA_filename + ' ' + imgB_filename + ' -offf ' + out_filename

if dataset == 'SINTEL_clean':

shell_command += ' -r 6 -r2 5 -epsilon %f -e 4 -s 50' % epsilon

elif dataset == 'kitti':

shell_command += ' -r 5 -r2 4 -epsilon %f -e 9 -s 150' % epsilon

elif dataset == 'middleburry':

shell_command += ' -r 5 -r2 4 -epsilon %f -e 7 -s 50' % epsilon

else:

shell_command += ' -r 8 -r2 6 -epsilon %f -e 4 -s 50' % epsilon

print('executing shell command:',shell_command)

exit_code = os.system(shell_command)

logger.debug('exit_code',exit_code)

#

def calc_flowFields_full_map_with_outliers(imgA_filename,imgB_filename, out_filename, dataset, epsilon=1):

shell_command = './flowFields/Code/FlowFields ' + imgA_filename + ' ' + imgB_filename + ' -ouff ' + out_filename

if dataset == 'SINTEL_clean':

shell_command += ' -r 6 -r2 5 -epsilon %f -e 4 -s 50' % epsilon

elif dataset == 'kitti':

shell_command += ' -r 5 -r2 4 -epsilon %f -e 9 -s 150' % epsilon

elif dataset == 'middleburry':

shell_command += ' -r 5 -r2 4 -epsilon %f -e 7 -s 50' % epsilon

else:

shell_command += ' -r 8 -r2 6 -epsilon %f -e 4 -s 50' % epsilon

print('executing shell command:',shell_command)

exit_code = os.system(shell_command)

logger.debug('exit_code',exit_code)

def calc_allFlowFieldsMapSINTEL(sintel_dir=SINTEL_HOME, ff_dir=SINTEL_FLOW_FIELDS_PATH, passType='final', AtoB=True, downsample=0, recalc=True, with_outliers=False, epsilon=5, for_epic=False): pairs = get_MPI_pairs(sintel_dir + passType + '/') for path1,path2 in pairs: pathSplit = string.split(path1,sep='/') sintelDir = pathSplit[-2] sintelFileName = string.split(pathSplit[-1],'.')[-2] if downsample > 0: img1 = sk.io.imread(path1) img2 = sk.io.imread(path2)

        img1 = img1[:(img1.shape[0] - (img1.shape[0] % 2**downsample)),:(img1.shape[1] - (img1.shape[1] % 2**downsample)),:]
        img2 = img2[:(img2.shape[0] - (img2.shape[0] % 2**downsample)),:(img2.shape[1] - (img2.shape[1] % 2**downsample)),:]

        img1 = sk.transform.rescale(img1, 1/float(2**downsample))
        img2 = sk.transform.rescale(img2, 1/float(2**downsample))

        sk.io.imsave('temp/img1.png', img1)
        sk.io.imsave('temp/img2.png', img2)
        path1 = 'temp/img1.png'
        path2 = 'temp/img2.png'
        saveDir = ff_dir  + '/' + passType + ('_%d/' % downsample) + sintelDir
    else:
        saveDir = ff_dir  + '/' + passType + '/'  + sintelDir
    if not os.path.exists(saveDir):
        os.makedirs(saveDir)
    if AtoB:
        if recalc or (not os.path.exists(saveDir + '/' + sintelFileName + '.txt') and for_epic) or (not os.path.exists(saveDir + '/' + sintelFileName + '.flo')):
            calc_flowFields_map(path1,path2, saveDir + '/' + sintelFileName, dataset='SINTEL_' + passType, epsilon=epsilon, for_epic=for_epic, with_outliers=with_outliers)
    else:
        if recalc or (not os.path.exists(saveDir + '/' + sintelFileName + '_BA.txt') and for_epic) or (not os.path.exists(saveDir + '/' + sintelFileName + '_BA.flo')):
            calc_flowFields_map(path2,path1, saveDir + '/' + sintelFileName + '_BA', dataset='SINTEL_' + passType, epsilon=epsilon, for_epic=for_epic, with_outliers=with_outliers)

def calc_allFlowFieldsMapKitti2015(kitti_dir=KITTI2015_HOME_GS, ff_dir=KITTI_FLOW_FIELDS_PATH, AtoB=True, downsample=0, recalc=True, epsilon=1, for_epic=False): pairs = get_kitti_pairs(kitti_dir) for path1,path2 in pairs: pathSplit = string.split(path1,sep='/') kittiFileName = string.split(pathSplit[-1],'.')[-2] saveDir = ff_dir if not os.path.exists(saveDir): os.makedirs(saveDir) if AtoB: if recalc or (not os.path.exists(saveDir + '/' + kittiFileName + '.txt') and for_epic) or not os.path.exists(saveDir + '/' + kittiFileName + '.flo'): calc_flowFields_map(path1,path2, saveDir + '/' + kittiFileName, dataset='kitti', epsilon=epsilon,for_epic=for_epic, with_outliers=False) else: if recalc or (not os.path.exists(saveDir + '/' + kittiFileName + '_BA.txt') and for_epic) or not os.path.exists(saveDir + '/' + kittiFileName + '_BA.flo'): calc_flowFields_map(path2,path1, saveDir + '/' + kittiFileName + '_BA', dataset='kitti', epsilon=epsilon,for_epic=for_epic, with_outliers=False)

def calc_allFlowFieldsMapFC(AtoB=True, recalc=True, epsilon=1, ff_dir=FC_FLOW_FIELDS_PATH): pairs = get_FC_pairs(FC_HOME) for path1,path2 in pairs: pathSplit = string.split(path1,sep='/') FCFileName = string.split(pathSplit[-1],'.')[-2] if AtoB: if recalc or not os.path.exists(ff_dir + '/' + FCFileName + '.flo'): calc_flowFields_full_map(path1,path2, ff_dir + '/' + FCFileName + '.flo', dataset='fc', epsilon=epsilon) else: if recalc or not os.path.exists(ff_dir + '/' + FCFileName + '_BA.flo'): calc_flowFields_full_map(path2,path1, ff_dir + '/' + FCFileName + '_BA.flo', dataset='fc', epsilon=epsilon)

def load_allFlowFieldsMapSINTEL(sintelFlowFieldsDir, data_dir=SINTEL_DATA_PATH, passType = 'final', doReload = False, interp=False, interp_type='lin', suffix='', downsample_source_img=0, downscale=1, with_outliers=False, epic_ds=False, min_pixels_for_downscale=1, epsilon=5): logger.info("loading flow fields maps for MPI sintel") start_time = time.time()

allFiles = load_allFiles_SINTEL(data_dir)
if with_outliers:
    base_filename = 'allFFMatches_with_outliers_'
else:
    base_filename = 'allFFMatches_'
if epic_ds:
    base_filename += '_epic_'
if epsilon != 1:
    base_filename += 'eps_%.1f_' % epsilon
if interp:
    saveFileName = base_filename + passType + '_interp_' + interp_type + '.dump'
elif suffix != '':
    saveFileName = base_filename + passType + '_' + suffix + '.dump'
else:
    if downsample_source_img > 0:
        saveFileName = base_filename  + passType + ('_%d.dump' % downsample_source_img)
    elif downscale != 1:
        saveFileName = base_filename  + passType + ('_d_%d.dump' % downscale)
    else:
        saveFileName = base_filename + passType + '.dump'

print (saveFileName)
if os.path.exists(data_dir + '/' + saveFileName) and not doReload:
    with open(data_dir + '/' + saveFileName, 'r') as f:
        allImg = pickle.load(f)
        allImg_BA = pickle.load(f)
        allMask = pickle.load(f)
        allMask_BA = pickle.load(f)
        # allData = pickle.load(f)
        # allScores = pickle.load(f)
else:
    allImg = {}
    allImg_BA = {}
    allMask_BA = {}
    allMask = {}
    # allData = {}
    for key, currFile in allFiles.items():
        if epic_ds:
            file_suff='.txt'
        else:
            file_suff='.flo'

        if with_outliers:
            currFile_full  = currFile  + '_with_outliers'
            currFile_BA_full  = currFile  + '_BA_with_outliers'
        else:
            currFile_full  = currFile
            currFile_BA_full  = currFile  + '_BA'

        if epsilon != 1:
            sintelFlowFieldsDir_full  = sintelFlowFieldsDir  + '/eps_%.1f/' % epsilon
        else:
            sintelFlowFieldsDir_full  = sintelFlowFieldsDir

        if downsample_source_img > 0:
            ffFileName = sintelFlowFieldsDir_full + '/' + passType + ('_%d/' % downsample_source_img) + currFile_full + file_suff
            ffFileName_BA = sintelFlowFieldsDir_full + '/' + passType + ('_%d/' % downsample_source_img) + currFile_BA_full + file_suff
        else:
            ffFileName = sintelFlowFieldsDir_full + '/' + passType + '/' + currFile_full + file_suff
            ffFileName_BA = sintelFlowFieldsDir_full + '/' + passType + '/' + currFile_BA_full + file_suff
        print("Loading file: " + ffFileName)
        if epic_ds:
            img,mask = read_flowFieldsOutputFile_DMformat(SINTEL_FF_ATOMIC_SIZE, ffFileName, SINTEL_WIDTH, SINTEL_HEIGHT)
            img_BA,mask_BA = read_flowFieldsOutputFile_DMformat(SINTEL_FF_ATOMIC_SIZE, ffFileName_BA, SINTEL_WIDTH, SINTEL_HEIGHT)
        else:
            img,mask = read_flowFieldsOutputFile(ffFileName)
            img_BA,mask_BA = read_flowFieldsOutputFile(ffFileName_BA)
            # img_BA,mask_BA = img,mask

        img = img * (2 ** downsample_source_img)

        img_BA = img_BA * (2 ** downsample_source_img)

        # img_BA,mask_BA = img,mask
        if np.any(np.isnan(img)) or np.any(np.isnan(mask)):
            logger.warning("found nan values")

        if downscale != 1:
            from skimage.util.shape import view_as_blocks

            # from skimage import transform
            # img = img[downscale/2::downscale,downscale/2::downscale]
            # img = transform.rescale(img,1/float(downscale), preserve_range=True)
            img[:,:,0][mask==-1]=np.nan
            img[:,:,1][mask==-1]=np.nan

            img = img[:(img.shape[0] - (img.shape[0] % downscale)),:(img.shape[1] - (img.shape[1] % downscale)),:]

            blocks = view_as_blocks(img,(downscale,downscale,2))
            img = np.nanmean(blocks, axis=(-2,-3,-4))
            # img[np.sum(np.isnan(blocks), axis=(-2,-3,-4)) < min_pixels_for_downscale] = np.nan

            mask = np.ones_like(img)
            mask[np.isnan(img)] = -1
            mask = mask[:,:,0]
            img[np.isnan(img)]=0

            # img_BA = img_BA[downscale/2::downscale,downscale/2::downscale]
            # img_BA = transform.rescale(img_BA,1/float(downscale), preserve_range=True)

            img_BA [:,:,0][mask_BA ==-1]=np.nan
            img_BA [:,:,1][mask_BA ==-1]=np.nan

            img_BA = img_BA[:(img_BA.shape[0] - (img_BA.shape[0] % downscale)),:(img_BA.shape[1] - (img_BA.shape[1] % downscale)),:]

            blocks = view_as_blocks(img_BA ,(downscale,downscale,2))
            img_BA  = np.nanmean(blocks, axis=(-2,-3,-4))
            # img_BA [np.sum(np.isnan(blocks), axis=(-2,-3,-4)) > downscale*downscale*0.25] = np.nan

            mask_BA  = np.ones_like(img_BA )
            mask_BA [np.isnan(img_BA)] = -1
            mask_BA  = mask_BA [:,:,0]
            img_BA [np.isnan(img_BA)]=0

            # mask = mask[:(mask.shape[0] - (mask.shape[0] % downscale)),:(mask.shape[1] - (mask.shape[1] % downscale))]
            # mask = mask[downscale/2::downscale,downscale/2::downscale]

            # mask = transform.downscale_local_mean(mask,(downscale, downscale))
            # mask[np.where(mask > 0.25)]=1
            # mask[np.where(mask <= 0.25)]=0

            # mask_BA = mask_BA[:(mask_BA.shape[0] - (mask_BA.shape[0] % downscale)),:(mask_BA.shape[1] - (mask_BA.shape[1] % downscale))]
            # mask_BA = mask_BA[downscale/2::downscale,downscale/2::downscale]
            # mask_BA = transform.downscale_local_mean(mask_BA,(downscale, downscale))

            # mask_BA[np.where(mask_BA > 0.25)]=1
            # mask_BA[np.where(mask_BA <= 0.25)]=0

        allImg[key] = np.asarray(img, np.float32)
        allImg_BA[key] = np.asarray(img_BA, np.float32)

        allMask[key] = np.asarray(mask, np.float32)
        allMask_BA[key] = np.asarray(mask_BA, np.float32)
        # allData[key] = np.asarray(np.concatenate([img, np.reshape(mask,[len(mask),len(mask[0]),1],2)],2),np.float32)

    with open(data_dir + '/' + saveFileName, 'w') as f:
        pickle.dump(allImg, f,protocol=-1)
        pickle.dump(allImg_BA, f,protocol=-1)
        pickle.dump(allMask, f,protocol=-1)
        pickle.dump(allMask_BA, f,protocol=-1)
        # pickle.dump(allData, f,protocol=-1)

logger.info("Finished loading flow fields maps for MPI sintel after %f seconds" %  (time.time() - start_time))
return allImg, allImg_BA, allMask, allMask_BA

def load_allFlowFieldsMapKitti2015(kittiFlowFieldsDir, kitti_dir=KITTI2015_HOME, data_dir=KITTI_DATA_PATH, doReload = False, downscale=1, num_of_pairs=KITTI_NUM_OF_PAIRS): logger.info("loading flow fields maps for kitti 2015") start_time = time.time() base_filename = 'allFFMatches' if downscale != 1: saveFileName = base_filename + ('d%d.dump' % downscale) else: saveFileName = base_filename + '.dump'

if os.path.exists(data_dir+ saveFileName) and not doReload:
    with open(data_dir + saveFileName, 'rb') as f:
        allImg = pickle.load(f)
        allMask = pickle.load(f)
        allImg_BA = pickle.load(f)
        allMask_BA = pickle.load(f)
else:
    allImg = {}
    allMask = {}
    allImg_BA = {}
    allMask_BA = {}

    for j in xrange(num_of_pairs):
        ffFileName = kittiFlowFieldsDir + '/%06d_10.flo' % j
        ffFileName_BA = kittiFlowFieldsDir + '/%06d_10_BA.flo' % j

        logger.debug("Loading file: " + ffFileName)
        img,mask = read_flowFieldsOutputFile(ffFileName)

        logger.debug("Loading file BA: " + ffFileName)
        img_BA,mask_BA = read_flowFieldsOutputFile(ffFileName_BA)

        if downscale != 1:
            from skimage.util.shape import view_as_blocks

            img[:,:,0][mask==-1]=np.nan
            img[:,:,1][mask==-1]=np.nan

            img = img[:(img.shape[0] - (img.shape[0] % downscale)),:(img.shape[1] - (img.shape[1] % downscale)),:]

            blocks = view_as_blocks(img,(downscale,downscale,2))
            img = np.nanmean(blocks, axis=(-2,-3,-4))

            mask = np.ones_like(img)
            mask[np.isnan(img)] = -1
            mask = mask[:,:,0]
            img[np.isnan(img)]=0

            img_BA [:,:,0][mask_BA ==-1]=np.nan
            img_BA [:,:,1][mask_BA ==-1]=np.nan

            img_BA = img_BA[:(img_BA.shape[0] - (img_BA.shape[0] % downscale)),:(img_BA.shape[1] - (img_BA.shape[1] % downscale)),:]

            blocks = view_as_blocks(img_BA ,(downscale,downscale,2))
            img_BA  = np.nanmean(blocks, axis=(-2,-3,-4))

            mask_BA  = np.ones_like(img_BA )
            mask_BA [np.isnan(img_BA)] = -1
            mask_BA  = mask_BA [:,:,0]
            img_BA [np.isnan(img_BA)]=0

        if np.any(np.isnan(img)) or np.any(np.isnan(mask)):
            logger.warning("found nan values")
        allImg[j] = np.asarray(img, np.float32)
        allMask[j] = np.asarray(mask, np.float32)
        allImg_BA[j] = np.asarray(img_BA, np.float32)
        allMask_BA[j] = np.asarray(mask_BA, np.float32)
    with open(data_dir + saveFileName, 'wb') as f:
        pickle.dump(allImg, f,protocol=-1)
        pickle.dump(allMask, f,protocol=-1)
        pickle.dump(allImg_BA, f,protocol=-1)
        pickle.dump(allMask_BA, f,protocol=-1)
logger.info("Finished loading flow fields maps for kitti 2015 after %f seconds" %  (time.time() - start_time))
return allImg, allImg_BA, allMask, allMask_BA

def load_allFlowFieldsMapFC(fcFlowFieldsDir, data_dir=FC_DATA_PATH, doReload = False, interp=False, interp_type='lin', downscale=8, epsilon=5): logger.info("loading flow fields maps for MPI sintel") start_time = time.time()

base_filename = 'allFFMatches_'

if epsilon != 1:
    base_filename += 'eps_%.1f_' % epsilon

if interp:
    base_filename += ('_interp_' + interp_type)

saveFileName = base_filename  + '.dump'

if os.path.exists(data_dir + '/' + saveFileName) and not doReload:
    with open(data_dir + '/' + saveFileName, 'r') as f:
        allImg = pickle.load(f)
        allImg_BA = pickle.load(f)
        allMask = pickle.load(f)
        allMask_BA = pickle.load(f)
else:
    allImg = {}
    allImg_BA = {}
    allMask = {}
    allMask_BA = {}

    for j in xrange(FC_NUM_OF_PAIRS):
        if epsilon != 1:
            fcFlowFieldsDir_full  = fcFlowFieldsDir  + '/eps_%.1f/' % epsilon
        else:
            fcFlowFieldsDir_full  = fcFlowFieldsDir

        ffFileName = fcFlowFieldsDir_full + '/%05d_img1.flo' % (j + 1)
        ffFileName_BA = fcFlowFieldsDir_full + '/%05d_img1_BA.flo' % (j + 1)
        logger.debug("Loading file: " + ffFileName)
        img,mask = read_flowFieldsOutputFile(ffFileName)
        img_BA,mask_BA= read_flowFieldsOutputFile(ffFileName_BA)

        if np.any(np.isnan(img)) or np.any(np.isnan(mask)):
            logger.warning("found nan values")
        if downscale != 1:
            from skimage.util.shape import view_as_blocks

            # from skimage import transform
            # img = img[downscale/2::downscale,downscale/2::downscale]
            # img = transform.rescale(img,1/float(downscale), preserve_range=True)
            img[:,:,0][mask==-1]=np.nan
            img[:,:,1][mask==-1]=np.nan

            img = img[:(img.shape[0] - (img.shape[0] % downscale)),:(img.shape[1] - (img.shape[1] % downscale)),:]

            blocks = view_as_blocks(img,(downscale,downscale,2))
            img = np.nanmean(blocks, axis=(-2,-3,-4))
            # img[np.sum(np.isnan(blocks), axis=(-2,-3,-4)) > downscale*downscale*0.25] = np.nan

            mask = np.ones_like(img)
            mask[np.isnan(img)] = -1
            mask = mask[:,:,0]
            img[np.isnan(img)]=0

            # img_BA = img_BA[downscale/2::downscale,downscale/2::downscale]
            # img_BA = transform.rescale(img_BA,1/float(downscale), preserve_range=True)

            img_BA [:,:,0][mask_BA ==-1]=np.nan
            img_BA [:,:,1][mask_BA ==-1]=np.nan

            img_BA = img_BA[:(img_BA.shape[0] - (img_BA.shape[0] % downscale)),:(img_BA.shape[1] - (img_BA.shape[1] % downscale)),:]

            blocks = view_as_blocks(img_BA ,(downscale,downscale,2))
            img_BA  = np.nanmean(blocks, axis=(-2,-3,-4))
            # img_BA [np.sum(np.isnan(blocks), axis=(-2,-3,-4)) > downscale*downscale*0.25] = np.nan

            mask_BA  = np.ones_like(img_BA )
            mask_BA [np.isnan(img_BA)] = -1
            mask_BA  = mask_BA [:,:,0]
            img_BA [np.isnan(img_BA)]=0

        allImg[j] = np.asarray(img, np.float32)
        allImg_BA[j] = np.asarray(img_BA, np.float32)
        allMask[j] = np.asarray(mask, np.float32)
        allMask_BA[j] = np.asarray(mask_BA, np.float32)

    with open(data_dir + '/' + saveFileName, 'w') as f:
        pickle.dump(allImg, f,protocol=-1)
        pickle.dump(allImg_BA, f,protocol=-1)
        pickle.dump(allMask, f,protocol=-1)
        pickle.dump(allMask_BA, f,protocol=-1)

logger.info("Finished loading flow fields maps for flying chairs after %f seconds" %  (time.time() - start_time))
return allImg, allImg_BA, allMask, allMask_BA
fperezgamonal commented 4 years ago

Hello Shay,

Thank you so much for looking into it. I'm sure this script will help me reproduce the input matches you used. I'll update you once I've successfully done so.

Cheers,

Ferran.