ND imagery

[dbuscombe-usgs]

now, 3- and 4-band images are accommodated, but needs to be generalized to ND images, stored as nparrays or multiband tiffs

[dbuscombe-usgs]

• makedatasets needs to be updated so it can store the image as an ND array

• train model needs to be updated so it can use the ND array (and detect automatically / generalize to any N)

• seg_images needs a utility pre-processing script to get multiple coincident bands into an input npz

[dbuscombe-usgs]

function that

• points at a directory containing multiple coincident/concurrent bands with the same file root
• pairs them up and puts them into npz files
• also put any geosptial metadata into that npz file

[dbuscombe-usgs]

3 options

1. preprocessing + make datasets reads config > 3 bands
2. separate make and train routines
3. super function / loaders --> calls appropriate functions - remove the if/else options

all of this made sense in the call (!) - evan is wishing for a solution that won't break his current workflow ... see below... this is more complicated than previously though, because of the a) need to pad disparate sized images, and b) the extreme limitations of keras' built0in image generator, that is not amenable to applying the same augmentations on multiple coincident and co-occuring images

the solution is a single function that combines all 3 elements listed above

[dbuscombe-usgs]

having spent a couple of hours playing with ideas, i conclude The only sane way to deal with this is to have one "makedatasets" function that deals with all cases

1. unpadded 1-3 band imagery
2. padded 1-3 band imagery
3. unpadded 4+ band imagery
4. padded 4+ band imagery (basically, satellite data i.e. most of the data I will use going forward)

currently, only 1 and 2 implemented and tested

However, I will implement a "make_nd_datasets" function first that will generalize all 4 cases. Case 1 users (above) can keep using 'make_datasets' for now

[dbuscombe-usgs]

1. images need to be padded first
2. then a non-augmented set created (just 1! current implementation makes AUG LOOPS copies...)
3. then (and only then) an augmented set
4. generators need to be generalized to deal with more than 1 image file

ideally, we move away from the keras imagedatagenerator routine entirely

[dbuscombe-usgs]

makedatasets should eventually

1. automatically resize all imagery (all bands) to a common size if not already so (use target size)
2. generate a non-augmented set
3. generate an augmented set
4. generate additional augmented sets according to AUG_LOOPS (allows AUG_LOOPS to equal 1)
5. create identical npz files for 1, 3 and >3 band images

[dbuscombe-usgs]

I notice that all augmentation and image rescaling options used so far can introduce quite horrible interpolation artifacts for the discrete label images, even rescale from skimage.io with zero-order interpolation, anti-aliasing, and clipping to original range because it first recodes to float .... grrr.

Labels dont ever need interpolation, only resizing through nearest neighbour (up-pad, then infilling, or a spatial replacement operation for downsizing). By definition, interpolation only functions on continuous variables stored as floats, never on discrete arrays

Labels will now therefore adopt this approach which does not create floats or use interpolation

def scale(im, nR, nC):
  '''
  for reszing 2d integer arrays 
  '''
  nR0 = len(im)     # source number of rows 
  nC0 = len(im[0])  # source number of columns 
  tmp = [[ im[int(nR0 * r / nR)][int(nC0 * c / nC)]  
             for c in range(nC)] for r in range(nR)]
  return np.array(tmp).reshape((nR,nC))

lab2 = scale_label(lab,TARGET_SIZE[0],TARGET_SIZE[1])

modified from here

corresponding rgb images can be dealt with like this:

def scale_rgb(img, nR, nC, nD):
  '''
  for reszing 3d integer arrays
  '''
  imgout = np.zeros((nR, nC, nD))
  for k in range(3):
      im = img[:,:,k]
      nR0 = len(im)     # source number of rows
      nC0 = len(im[0])  # source number of columns
      tmp = [[ im[int(nR0 * r / nR)][int(nC0 * c / nC)]
                 for c in range(nC)] for r in range(nR)]
      imgout[:,:,k] = np.array(tmp).reshape((nR,nC))
  return imgout

[dbuscombe-usgs]

also, padded images and labels will be stored as lossless pngs, instead of jpegs

imsave(..., check_contrast=False, compression=0)

[dbuscombe-usgs]

before resize

after resize with no interpolation

[dbuscombe-usgs]

"makedatasets ND":

• [x] automatically resize all imagery (all bands) to a common size if not already so (use target size)
• [x] generate a non-augmented set
• [x] generate an augmented set for individual bands
• [x] generate an augmented set for all merged bands
• [x] generate additional augmented sets according to AUG_LOOPS (allows AUG_LOOPS to equal 1)
• [x] create identical npz files for 1, 3 and >3 band images

[dbuscombe-usgs]

new function make_nd_dataset,py is for N-D imagery that is stored across multiple files, each of which are a separate band or combination of bands from the same sensor, or otherwise coincident and concurrent bands stored in different files (e.g. orthomosaics and digital elevation models). The program has been tested with a set of 3-band rgb images, a coincident set of 1-band NIR images, and a set of 1-band SWIR images, fused to form a 5-band raster stack

1. user prompted for config file
2. user prompted for directory to store output files (npzs for model training)
3. user prompted for directory of label images
4. user prompted for directory of images
5. user is prompted for directory of additional images until the 'no' button is pressed in response to the on-screen dialog box "More directories of images?"
6. if the entire set of label images are the same size, do_resize will be set of False, and images will not be padded or resized, otherwise do_resize will be True, and all images smaller than TARGET_SIZE will be padded and all images larger than TARGET_SIZE will be resized. Note that this version produces much better labels, due to the use of remapping rather than interpolation of 2d label arrays larger than TARGET_SIZE
7. if do_resize is True, lists of padded images and labels are compiled, otherwise lists of the original unpadded/unresized imagery are compiled (not yet tested the condition do_resize is False)
8. non-augmented set of npz files are created from the input lists. if do_resize is True, these will be padded/resized, otherwise the original size
9. non-augmented npzs are read back in as a tf.dataset, to simulate model training, and examples are printed to file (as before)
10. augmented sets are created. if do_resize is True, these will be padded/resized, otherwise the original size
11. augmented npzs are read back in as a tf.dataset, to simulate model training, and examples are printed to file (as before)

new function make_nd_dataset,py is supposed to be generic and should eventually replace make_dataset.py, now for the N_DATA_BANDS <= 3 situation only

[dbuscombe-usgs]

some random augmented examples with REMAP_CLASSES

[dbuscombe-usgs]

SOme ranodm non-augmented samples with REMAP_CLASSES

[dbuscombe-usgs]

The next stage of this task is to create a utility that a user can merge (and optionally pad/resize) ND sample imagery for use in deployment/prediction mode

New function utils\merge_nd_inputs4pred.py . It is essentially the same as make_nd_datasets except:

1. it only operates on sample imagery (not corresponding labels)
2. it does not carry out augmentation (therefore no config file is required)

Prompts the user for

1. imagery (keep adding bands from file using tkinter dialogs)
2. a place to store outputs (merged data as npz file)

root = Tk()
root.filename =  filedialog.askdirectory(initialdir = os.getcwd(),title = "Select directory for storing OUTPUT files")
output_data_path = root.filename
print(output_data_path)
root.withdraw()

root = Tk()
root.filename =  filedialog.askdirectory(initialdir = os.getcwd(),title = "Select FIRST directory of IMAGE files")
data_path = root.filename
print(data_path)
root.withdraw()

W=[]
W.append(data_path)

result = 'yes'
while result == 'yes':
    result = messagebox.askquestion("More directories of images?", "More directories of images?", icon='warning')
    if result == 'yes':
        root = Tk()
        root.filename =  filedialog.askdirectory(initialdir = os.getcwd(),title = "Select directory of image files")
        data_path = root.filename
        print(data_path)
        root.withdraw()
        W.append(data_path)

3. target size (if the program detects variable image size) for padding/resizing

## neeed resizing?
szs = [imread(f).shape for f in files[:,0]]
szs = np.vstack(szs)[:,0]
if len(np.unique(szs))>1:
    do_resize=True
else:
    do_resize=False

from tkinter import simpledialog
application_window = Tk()
TARGET_X = simpledialog.askinteger("Imagery are different sizes and will be resized.",
                                "What is the TARGET_SIZE (X) of the intended model?",
                                 parent=application_window,
                                 minvalue=32, maxvalue=8192)

TARGET_Y = simpledialog.askinteger("Imagery are different sizes and will be resized.",
                                "What is the TARGET_SIZE (Y) of the intended model?",
                                 parent=application_window,
                                 minvalue=32, maxvalue=8192)

TARGET_SIZE = [TARGET_X,TARGET_Y]     

the same padding/resiizng commands are used as make_nd_dataset. Padded imagery is created and put into folders

why is this important? Imagery of the same spatial extent may have different grid sizes. Take these two Landsat scenes, downloaded using CoastSat

This one is 906x792 pixels

and this is the same spatial extent, but 830 x 790 px

the same two images padded/resized

If I apply the same procedure to all bands, then I can stack all bands into coincident N-D rasters. Non-augmented imagery is simply stored in npz files for prediction

[dbuscombe-usgs]

This is now implemented as utils\merge_nd_inputs4pred.py

Next, I need to modify seg_images_in_folder.py and ensemble_seg_images_in_folder to accept the new file formats when N_DATA_BANDS > 3

[dbuscombe-usgs]

• [x] merge functionality of seg_images_in_folder.py and ensemble_seg_images_in_folder
• [x] test new seg_images_in_folder.py with 5D hatteras / solo and ensemble models
• [x] merge functionality of make_dataset and make_nd_dataset
• [ ] end-to end test (make datasets, train_model, and seg_images_in_folder) of 1-band imagery + binary classes (use sidescan data from doodler paper)
• [ ] end-to end test of 1-band imagery + multiple classes (use sidescan data from doodler paper)
• [ ] end-to end test of padded/resized 3-band imagery + multiple classes (use C/T L8 data)
• [x] end-to end test of padded/resized 5-band imagery + multiple classes (use C/T L8 data)
• [ ] end-to end test of padded/resized 3-band imagery + binary classes (use C/T L8 data and REMAP classes)
• [ ] end-to end test of no-padded 3-band imagery + multiple classes (use NOAA data)
• [ ] end-to end test of no-padded 3-band imagery + binary classes (use NOAA data and REMAP classes)