Basic proposed workflow

[dbuscombe-usgs]

@2320sharon this is a description of the workflow I have developed and intend to build here in SDStools, fyi

Currently the Coastseg workflow results in a merged tidally corrected shoreline timeseries file, which I believe most end-users will view as 'the result', transect_time_series_tidally_corrected.csv. Then, as discussed here, we want the option of reordering the data so it is time (rows) x transect (cols), with missing values as NaN. I believe Coastseg should do that, but I also think we should do it for all numeric variables, i.e., tide, x, and y. So, if cs is your dataframe

df_distances_by_time_and_transect = cs.pivot(index='dates',columns='transect_id', values='cross_distance')
df_tides_by_time_and_transect = cs.pivot(index='dates',columns='transect_id', values='tide')
df_x_by_time_and_transect = cs.pivot(index='dates',columns='transect_id', values='x')
df_y_by_time_and_transect = cs.pivot(index='dates',columns='transect_id', values='y')

Simple! this is what those matrices look like for my Elwha site

(I'm not sure why my transect numbers are non-ordinal w.r.t x and y...)

df_x_by_time_and_transect.index = pd.DatetimeIndex(df_x_by_time_and_transect.index)
df_y_by_time_and_transect.index = pd.DatetimeIndex(df_y_by_time_and_transect.index)
df_tides_by_time_and_transect.index = pd.DatetimeIndex(df_tides_by_time_and_transect.index)
df_distances_by_time_and_transect.index = pd.DatetimeIndex(df_distances_by_time_and_transect.index)
times = df_tides_by_time_and_transect.index

plt.figure(figsize=(12,12))

plt.subplot(221)
plt.pcolormesh(np.arange(df_distances_by_time_and_transect.shape[1]), df_distances_by_time_and_transect.index, df_distances_by_time_and_transect, shading='nearest')
plt.xlabel('Transect #'); plt.ylabel('Time')
plt.title('a) Raw tidally corrected shorelines')
plt.colorbar()

plt.subplot(222)
plt.pcolormesh(np.arange(df_tides_by_time_and_transect.shape[1]), df_tides_by_time_and_transect.index, df_tides_by_time_and_transect, shading='nearest')
plt.xlabel('Transect #'); plt.ylabel('Time')
plt.title('b) Tide')
plt.colorbar()

plt.subplot(223)
plt.pcolormesh(np.arange(df_tides_by_time_and_transect.shape[1]), df_tides_by_time_and_transect.index, df_x_by_time_and_transect, shading='nearest')
plt.xlabel('Transect #'); plt.ylabel('Time')
plt.title('c) X')
plt.colorbar()

plt.subplot(224)
plt.pcolormesh(np.arange(df_tides_by_time_and_transect.shape[1]), df_tides_by_time_and_transect.index, df_y_by_time_and_transect, shading='nearest')
plt.xlabel('Transect #'); plt.ylabel('Time')
plt.title('d) Y')
plt.colorbar()

Ok, and I propose Coastseg stop there, and this toolbox (SDStools) takes over the reins for basic analyses and plotting, etc.

The first thing I propose here is a new way to interpolate and filter the data

1. Shoreline data inpainting: Use biharmonic inpainting

from skimage.restoration import inpaint
mask = np.isnan(df_distances_by_time_and_transect_use)
cs_matrix_inpaint = inpaint.inpaint_biharmonic(df_distances_by_time_and_transect_use, mask)

3. denoising Use a wavelet filter, calibrated over a number of scales

   
   from functools import partial
   from skimage.restoration import calibrate_denoiser, denoise_wavelet
   # rescale_sigma=True required to silence deprecation warnings
   _denoise_wavelet = partial(denoise_wavelet, rescale_sigma=True)

Parameters to test when calibrating the denoising algorithm

parameter_ranges = {'sigma': np.arange(0.02, 0.2, 0.02), 'wavelet': ['db1', 'db2'], 'convert2ycbcr': [False, False]}

Denoised image using default parameters of denoise_wavelet

default_output = denoise_wavelet(cs_matrix_inpaint, rescale_sigma=True)

Calibrate denoiser

calibrated_denoiser = calibrate_denoiser(cs_matrix_inpaint, _denoise_wavelet, denoise_parameters=parameter_ranges)

Denoised image using calibrated denoiser

cs_inpaint_denoised = calibrated_denoiser(cs_matrix_inpaint)

Application of these result in
![image](https://github.com/Doodleverse/SDStools/assets/3596509/2db9f1e7-942a-49e5-bd65-4933d4e3b5f0)

4. finally, computing shoreline change relative to initial

I propose instead of subtracting the initial shoreline, we subtract an average of the first N shorelines, due to noise considerations. It efficiently does this as a matrix-vector operation rather than apply the correction to each shoreline in turn

N = 10 shore_change = (cs_inpaint_denoised - cs_inpaint_denoised[:N,:].mean(axis=0)).T

This results in
![image](https://github.com/Doodleverse/SDStools/assets/3596509/821b8062-5f7a-4a17-9f53-5d2c3f4ca543)

Mean shoreline diaplecement across all transects is

![image](https://github.com/Doodleverse/SDStools/assets/3596509/797c2c2b-f8f4-413d-b81b-e5a6b83ae35a)

plt.plot(df_distances_by_time_and_transect.index, shore_change.mean(axis=0), 'k') plt.ylabel('Mean shoreline displacement (m)') plt.show()

Mean shore displacement across all time is
![image](https://github.com/Doodleverse/SDStools/assets/3596509/c9535830-6990-4315-a723-26828d7e298f)

plt.plot(np.arange(30,100), shore_change.mean(axis=1), 'k') plt.ylabel('Mean shoreline displacement (m)') plt.show()

[2320sharon]

Hi @dbuscombe-usgs Sorry for the late response this got lost in my notifications. I've implemented most of the changes in https://github.com/Doodleverse/CoastSeg/issues/224 which means coastseg generates 5 files now:

1. transect_time_series.csv

   • matrix of date vs transect id (raw)

2. transect_time_series_merged.csv

   • dataframe of date, transect id, transect x,y , cross distance

3. transect_time_series_tidally_corrected.csv

   • dataframe of date, transect id, transect x,y , tide prediction, cross distance

4. transect_time_series_tidally_corrected_matrix.csv

   • matrix of date vs transect id (tide corrected)

5. predicted_tides.csv

   • matrix of date, transect id and the predicted tide (AKA date x transect id) I have a few clarification questions now that I've seen this issue.

6. Currently the transect_time_series_tidally_corrected_matrix.csv drops any dates where none of the shoreline points on that date intersected any of the transects, is that okay or should they be rows with only NaNs like in transect_time_series.csv?

7. Do you want CoastSeg to generate 2 extra files being the ones below? I'm not sure if these are necessary since it seems like they can easily be generated by script in SDS_tools, but if you think that end users will find them helpful, then I'm happy to add them.

   df_x_by_time_and_transect = cs.pivot(index='dates',columns='transect_id', values='x')
   df_y_by_time_and_transect = cs.pivot(index='dates',columns='transect_id', values='y')

8. I tagged you in a question on https://github.com/Doodleverse/CoastSeg/issues/224 I'm sure it got lost in your notifications too.