Problems flattening mixed height images

Describe the bug I have some images of samples which have a large variation in height. The molecule has some sections at 2-3nm and others at 10-15nm. Therefore when flattening with TopoStats, streaks are still present where some of the low sections are. The config file also doesn't seem to facilitate the variation in threshold required to form the correct mask.

To Reproduce

Command used to run topostats e.g. python pygwytracing.py / python topostats/topotracing.py
Config file (you can attach this or paste it below)

base_dir: ./
output_dir: ./output
warnings: ignore
cores: 2
quiet: false
file_ext: .spm
loading:
  channel: Height
filter:
  run: true # Options : true, false
  threshold_method: absolute # Options : otsu, std_dev, absolute
  otsu_threshold_multiplier: 1.0
  threshold_std_dev:
    lower: 10.0
    upper: 0.5
  threshold_absolute:
    lower: 0.0
    upper: -7
  gaussian_size: 1.0121397464510862 # in px
  gaussian_mode: nearest
grains:
  run: true # Options : true, false
  # Thresholding by height
  threshold_method: std_dev # Options : std_dev, otsu, absolute
  otsu_threshold_multiplier: 1.0
  threshold_std_dev:
    lower: 10.0
    upper: 1.0
  threshold_absolute:
    lower: -1.0
    upper: 1.0
  direction: upper # Options: upper, lower, both (defines whether to look for grains above or below thresholds or both)
  # Thresholding by area
  absolute_smallest_grain_size: 100
  absolute_area_threshold:
    upper: [ 500, 800 ] # above surface [Low, High] in nm^2 (also takes null)
    lower: [ null, null ] # below surface [Low, High] in nm^2 (also takes null)
grainstats:
  run: true # Options : true, false
  cropped_size: 40.0 # Length (in nm) of square cropped images (can take -1 for grain-sized box)
  save_cropped_grains: true # Options : true, false
dnatracing:
  run: true # Options : true, false
plotting:
  run: true # Options : true, false
  save_format: png # Options : see https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.savefig.html
  image_set: all  # Options : all, core
  zrange: [0, 3]  # low and high height range for core images (can take [null, null])
  colorbar: true  # Options : true, false
  axes: true # Options : true, false (due to off being a bool when parsed)
  cmap: nanoscope # Options : nanoscope, afmhot
  histogram_log_axis: false # Options : true, false

Expected behavior Flattening of the background around the whole molecule

Output

**(Not required if using Docker) Your computer configuration (please complete the following information):** - OS: windows - topostats version: best way to get this at the moment is to paste the results of running `git show --oneline -s` in your topostats folder. - Python version: best way to get this is to paste the results of typing `python --version`. - Optionally, your installed packages: best way to get this is to paste the results of typing `pip freeze`. **Additional context** Add any other context about the problem here.

I have done a preliminary exploration of the issue, and found the following:

The reason that they won’t flatten nicely I think is due to the grains taking up so much of the image. Due to this, the initial median row alignment stage considers the rows where the grain is thickest, to have a median that lies on the grain rather than the background. This results in part of the image being centred incorrectly. (See image attached). There was an assumption made during development that any image would not be more grain than background, as that is how we determine which is which. I will consider this and speak with Alice to try to come up with a solution. I know that this is not a solution, however if you are doing more imaging, taking more zoomed out images (or just larger images) so that any grain is not more than say 1/4 of the image, would prevent this from occurring I think. You wouldn’t have to sacrifice on resolution, but it would take longer to scan 😞

TLDR: I know why it happens Going to talk to people about potential solutions, but it will probably mean major-ish changes, so I don’t think it will be quick… Taking bigger images or ensuring the grains are 1/4 of the image or less should prevent this in future

AFM-SPM / TopoStats

Problems flattening mixed height images #432