search

AFM-SPM / TopoStats

An AFM image analysis program to batch process data and obtain statistics from images
https://afm-spm.github.io/TopoStats/
GNU Lesser General Public License v3.0
55 stars 10 forks source link

DNA image not traced #751

Closed derollins closed 7 months ago

derollins commented 7 months ago

NB TopoStats < 2.0.0 is no longer actively maintained. If you have encountered an issue using this deprecated version please try TopoStats >= 2.0.0, there are instructions on installation, usage and configuration.

By default TopoStats will process images in parallel to speed up processing. In order to identify the problematic image you should re-run your analyses using the --cores 1 option so that images are processed sequentially. This makes it possible to identify which image causes the problem.

Checklist

Please try and tick off each of these items when filing the bug report. There are further instructions on each below.

Describe the bug

This image: https://drive.google.com/file/d/1TT7UR7419EbSOaITvhpcuG7sNYKd6F-3/view?usp=drive_link is well masked however it does not get traced.

20231130_DNA_126bpminicircle_lin2ng_NiCl2_3mM_HEPES_20mM 0_00016_above_masked

Copy of the output

Please copy and paste the output that is shown below within the ``` (triple-backticks).

Get this error:

[Thu, 07 Dec 2023 12:11:12] [WARNING ] [topostats] [20231130_DNA_126bpminicircle_lin2ng_NiCl2_3mM_HEPES_20mM.0_00016] : Errors occurred whilst calculating DNA tracing statistics, returning grain statistics

Include the configuration file

If no configuration file was specified with the -c/--config-file option the defaults were used, please use the run_topostats --create-config-file crash.yaml to save these to the crash.yaml file and copy the contents below.

# Config file generated 2023-11-29 00:15:04
# For more information on configuration and how to use it:
# https://afm-spm.github.io/TopoStats/main/configuration.html
base_dir: X:\pyne_group\Shared\AFM_Data\Cas9_Minicircles\20231130_Cas9Minicircles/image16 # Directory in which to search for data files
output_dir: X:\pyne_group\Shared\AFM_Data\Cas9_Minicircles\20231130_Cas9Minicircles/image16 # Directory to output results to
log_level: info # Verbosity of output. Options: warning, error, info, debug
cores: 1 # Number of CPU cores to utilise for processing multiple files simultaneously.
file_ext: .spm # File extension of the data files.
loading:
  channel: Height # Channel to pull data from in the data files.
filter:
  run: true # Options : true, false
  row_alignment_quantile: 0.5 # below values may improve flattening of larger features
  threshold_method: std_dev # Options : otsu, std_dev, absolute
  otsu_threshold_multiplier: 1.0
  threshold_std_dev:
    below: 10.0 # Threshold for data below the image background
    above: 1.0 # Threshold for data above the image background
  threshold_absolute:
    below: -1.0 # Threshold for data below the image background
    above: 1.0 # Threshold for data above the image background
  gaussian_size: 1.0121397464510862 # Gaussian blur intensity in px
  gaussian_mode: nearest
  # Scar remvoal parameters. Be careful with editing these as making the algorithm too sensitive may
  # result in ruining legitimate data.
  remove_scars:
    run: true
    removal_iterations: 2 # Number of times to run scar removal.
    threshold_low: 0.250 # below values make scar removal more sensitive
    threshold_high: 0.666 # below values make scar removal more sensitive
    max_scar_width: 4 # Maximum thichness of scars in pixels.
    min_scar_length: 16 # Minimum length of scars in pixels.
grains:
  run: true # Options : true, false
  # Thresholding by height
  threshold_method: absolute # Options : std_dev, otsu, absolute
  otsu_threshold_multiplier: 1.0
  threshold_std_dev:
    below: 10.0 # Threshold for grains below the image background
    above: 1.6 # Threshold for grains above the image background
  threshold_absolute:
    below: -1.0 # Threshold for grains below the image background
    above: 1.65 # Threshold for grains above the image background
  direction: above # Options: above, below, both (defines whether to look for grains above or below thresholds or both)
  # Thresholding by area
  smallest_grain_size_nm2: 50 # Size in nm^2 of tiny grains/blobs (noise) to remove, must be > 0.0
  absolute_area_threshold:
    above: [ 100, 300 ] # above surface [Low, High] in nm^2 (also takes null)
    below: [ null, null ] # below surface [Low, High] in nm^2 (also takes null)
  remove_edge_intersecting_grains: true # Whether or not to remove grains that touch the image border
grainstats:
  run: true # Options : true, false
  edge_detection_method: binary_erosion # Options: canny, binary erosion. Do not change this unless you are sure of what this will do.
  cropped_size: 40.0 # Length (in nm) of square cropped images (can take -1 for grain-sized box)
dnatracing:
  run: true # Options : true, false
  min_skeleton_size: 10 # Minimum number of pixels in a skeleton for it to be retained.
  skeletonisation_method: topostats # Options : zhang | lee | thin | topostats
  pad_width: 1 # Cells to pad grains by when tracing
#  cores: 1 # Number of cores to use for parallel processing
plotting:
  run: true # Options : true, false
  save_format: png # Options : see https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.savefig.html
  pixel_interpolation: null # Options : https://matplotlib.org/stable/gallery/images_contours_and_fields/interpolation_methods.html
  image_set: core  # Options : all, core
  zrange: [-4, 4]  # low and high height range for core images (can take [null, null]). low <= high
  colorbar: true  # Options : true, false
  axes: true # Options : true, false (due to off being a bool when parsed)
  cmap: nanoscope # Options : nanoscope, afmhot, gwyddion
  mask_cmap: blu # Options : blu, jet_r and any in matplotlib
  histogram_log_axis: false # Options : true, false
  histogram_bins: 200 # Number of bins for histogram plots to use
summary_stats:
  run: true # Whether to make summary plots for output data
  config: null

To Reproduce

https://drive.google.com/file/d/1TT7UR7419EbSOaITvhpcuG7sNYKd6F-3/view?usp=sharing

topostats process -c .\DNA126_config.yaml

Output

20231130_DNA_126bpminicircle_lin2ng_NiCl2_3mM_HEPES_20mM 0_00016_above_masked 20231130_DNA_126bpminicircle_lin2ng_NiCl2_3mM_HEPES_20mM 0_00016

TopoStats version

Please report the version of TopoStats you are using. There are several ways of doing this, either with pip or run_topostats. Please copy and paste all output from either of the following commands.

Installed version of TopoStats : 2.1.2

Operating System and Python Version

Operating System

Please let us know what operating system you are using, if you have used more than one then tick all boxes.

Python Version

Please let us know the version of Python you are using, paste the results of python --version

Python 3.10.9

Python Packages

If you are able to provide a list of your installed packages that may be useful. The best way to get this is to copy and paste the results of typing pip freeze.

aiofiles @ file:///C:/b/abs_9ex6mi6b56/croot/aiofiles_1683773603390/work
aiosqlite @ file:///C:/b/abs_9djc_0pyi3/croot/aiosqlite_1683773915844/work
anyio @ file:///C:/ci/anyio_1644481856696/work/dist
argon2-cffi @ file:///opt/conda/conda-bld/argon2-cffi_1645000214183/work
argon2-cffi-bindings @ file:///C:/ci/argon2-cffi-bindings_1644569876605/work
asttokens @ file:///opt/conda/conda-bld/asttokens_1646925590279/work
attrs @ file:///C:/b/abs_09s3y775ra/croot/attrs_1668696195628/work
Babel @ file:///C:/b/abs_a2shv_3tqi/croot/babel_1671782804377/work
backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work
beautifulsoup4 @ file:///C:/b/abs_0agyz1wsr4/croot/beautifulsoup4-split_1681493048687/work
bleach @ file:///opt/conda/conda-bld/bleach_1641577558959/work
brotlipy==0.7.0
certifi @ file:///C:/b/abs_4a0polqwty/croot/certifi_1683875377622/work/certifi
cffi @ file:///C:/b/abs_49n3v2hyhr/croot/cffi_1670423218144/work
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
colorama @ file:///C:/b/abs_a9ozq0l032/croot/colorama_1672387194846/work
comm @ file:///C:/b/abs_1419earm7u/croot/comm_1671231131638/work
contextlib2==21.6.0
contourpy==1.0.7
cryptography @ file:///C:/b/abs_8ecplyc3n2/croot/cryptography_1677533105000/work
cycler==0.11.0
debugpy @ file:///C:/ci_310/debugpy_1642079916595/work
decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
defusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/work
entrypoints @ file:///C:/ci/entrypoints_1649926676279/work
executing @ file:///opt/conda/conda-bld/executing_1646925071911/work
fastjsonschema @ file:///C:/Users/BUILDE~1/AppData/Local/Temp/abs_ebruxzvd08/croots/recipe/python-fastjsonschema_1661376484940/work
fonttools==4.38.0
h5py==3.8.0
idna @ file:///C:/b/abs_bdhbebrioa/croot/idna_1666125572046/work
igor==0.3
igor2==0.5.3
imageio==2.25.0
ipykernel @ file:///C:/b/abs_b4f07tbsyd/croot/ipykernel_1672767104060/work
ipython @ file:///C:/b/abs_d1yx5tjhli/croot/ipython_1680701887259/work
ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work
ipywidgets @ file:///C:/b/abs_5awapknmz_/croot/ipywidgets_1679394824767/work
jedi @ file:///C:/ci/jedi_1644315428305/work
Jinja2 @ file:///C:/b/abs_7cdis66kl9/croot/jinja2_1666908141852/work
joblib==1.2.0
json5 @ file:///tmp/build/80754af9/json5_1624432770122/work
jsonschema @ file:///C:/b/abs_6ccs97j_l8/croot/jsonschema_1676558690963/work
jupyter @ file:///C:/Windows/TEMP/abs_56xfdi__li/croots/recipe/jupyter_1659349053177/work
jupyter-console @ file:///C:/b/abs_82xaa6i2y4/croot/jupyter_console_1680000189372/work
jupyter-events @ file:///C:/b/abs_4cak_28ewz/croot/jupyter_events_1684268050893/work
jupyter-ydoc @ file:///C:/b/abs_e7m6nh5lao/croot/jupyter_ydoc_1683747253535/work
jupyter_client @ file:///C:/b/abs_059idvdagk/croot/jupyter_client_1680171872444/work
jupyter_core @ file:///C:/b/abs_9d0ttho3bs/croot/jupyter_core_1679906581955/work
jupyter_server @ file:///C:/b/abs_3eh8sm27tx/croot/jupyter_server_1686059851383/work
jupyter_server_fileid @ file:///C:/b/abs_f1yjnmiq_6/croot/jupyter_server_fileid_1684273602142/work
jupyter_server_terminals @ file:///C:/b/abs_ec0dq4b50j/croot/jupyter_server_terminals_1686870763512/work
jupyter_server_ydoc @ file:///C:/b/abs_8ai39bligw/croot/jupyter_server_ydoc_1686767445888/work
jupyterlab @ file:///C:/b/abs_c1msr8zz3y/croot/jupyterlab_1686179674844/work
jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work
jupyterlab-widgets @ file:///C:/b/abs_38ad427jkz/croot/jupyterlab_widgets_1679055289211/work
jupyterlab_server @ file:///C:/b/abs_e0qqsihjvl/croot/jupyterlab_server_1680792526136/work
kiwisolver==1.4.4
lxml @ file:///C:/b/abs_c2bg6ck92l/croot/lxml_1679646459966/work
MarkupSafe @ file:///C:/ci/markupsafe_1654508036328/work
matplotlib==3.6.3
matplotlib-inline @ file:///C:/ci/matplotlib-inline_1661934094726/work
mistune @ file:///C:/ci_310/mistune_1642084168466/work
nbclassic @ file:///C:/b/abs_c8_rs7b3zw/croot/nbclassic_1681756186106/work
nbclient @ file:///C:/ci/nbclient_1650308592199/work
nbconvert @ file:///C:/b/abs_4av3q4okro/croot/nbconvert_1668450658054/work
nbformat @ file:///C:/b/abs_85_3g7dkt4/croot/nbformat_1670352343720/work
nest-asyncio @ file:///C:/b/abs_3a_4jsjlqu/croot/nest-asyncio_1672387322800/work
networkx==3.0
notebook @ file:///C:/b/abs_49d8mc_lpe/croot/notebook_1681756182078/work
notebook_shim @ file:///C:/b/abs_ebfczttg6x/croot/notebook-shim_1668160590914/work
numpy==1.26.0
opencv-python==4.7.0.72
packaging @ file:///C:/b/abs_ed_kb9w6g4/croot/packaging_1678965418855/work
pandas==1.5.3
pandocfilters @ file:///opt/conda/conda-bld/pandocfilters_1643405455980/work
parso @ file:///opt/conda/conda-bld/parso_1641458642106/work
pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work
Pillow==9.4.0
platformdirs @ file:///C:/b/abs_73cc5cz_1u/croots/recipe/platformdirs_1662711386458/work
ply==3.11
prometheus-client @ file:///C:/Windows/TEMP/abs_ab9nx8qb08/croots/recipe/prometheus_client_1659455104602/work
prompt-toolkit @ file:///C:/b/abs_6coz5_9f2s/croot/prompt-toolkit_1672387908312/work
psutil @ file:///C:/Windows/Temp/abs_b2c2fd7f-9fd5-4756-95ea-8aed74d0039flsd9qufz/croots/recipe/psutil_1656431277748/work
pure-eval @ file:///opt/conda/conda-bld/pure_eval_1646925070566/work
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
pyfiglet==0.8.post1
Pygments @ file:///C:/b/abs_fay9dpq4n_/croot/pygments_1684279990574/work
pyOpenSSL @ file:///C:/b/abs_552w85x1jz/croot/pyopenssl_1677607703691/work
pyparsing==3.0.9
PyQt5==5.15.7
PyQt5-sip @ file:///C:/Windows/Temp/abs_d7gmd2jg8i/croots/recipe/pyqt-split_1659273064801/work/pyqt_sip
pyrsistent @ file:///C:/ci_310/pyrsistent_1642117077485/work
PySocks @ file:///C:/ci_310/pysocks_1642089375450/work
pySPM==0.2.23
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
python-json-logger @ file:///C:/b/abs_cblnsm6puj/croot/python-json-logger_1683824130469/work
pytz==2022.7.1
PyWavelets==1.4.1
pywin32==305.1
pywinpty @ file:///C:/b/abs_73vshmevwq/croot/pywinpty_1677609966356/work/target/wheels/pywinpty-2.0.10-cp310-none-win_amd64.whl
PyYAML @ file:///C:/b/abs_d0g7dqt2xw/croot/pyyaml_1670514768165/work
pyzmq @ file:///C:/b/abs_655zk4a3s8/croot/pyzmq_1686601465034/work
qtconsole @ file:///C:/b/abs_eb4u9jg07y/croot/qtconsole_1681402843494/work
QtPy @ file:///C:/ci/qtpy_1662015130233/work
requests @ file:///C:/b/abs_41owkd5ymz/croot/requests_1682607524657/work
rfc3339-validator @ file:///C:/b/abs_ddfmseb_vm/croot/rfc3339-validator_1683077054906/work
rfc3986-validator @ file:///C:/b/abs_6e9azihr8o/croot/rfc3986-validator_1683059049737/work
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.7
schema==0.7.5
scikit-image==0.19.2
scikit-learn==1.2.1
scipy==1.10.0
seaborn==0.12.2
Send2Trash @ file:///tmp/build/80754af9/send2trash_1632406701022/work
sip @ file:///C:/Windows/Temp/abs_b8fxd17m2u/croots/recipe/sip_1659012372737/work
six @ file:///tmp/build/80754af9/six_1644875935023/work
sniffio @ file:///C:/ci_310/sniffio_1642092172680/work
soupsieve @ file:///C:/b/abs_a989exj3q6/croot/soupsieve_1680518492466/work
stack-data @ file:///opt/conda/conda-bld/stack_data_1646927590127/work
terminado @ file:///C:/b/abs_25nakickad/croot/terminado_1671751845491/work
threadpoolctl==3.1.0
tifffile==2023.2.3
tinycss2 @ file:///C:/b/abs_52w5vfuaax/croot/tinycss2_1668168823131/work
toml @ file:///tmp/build/80754af9/toml_1616166611790/work
tomli @ file:///C:/Windows/TEMP/abs_ac109f85-a7b3-4b4d-bcfd-52622eceddf0hy332ojo/croots/recipe/tomli_1657175513137/work
-e git+https://github.com/AFM-SPM/TopoStats.git@14cd56dd19eecc6af0ae105b076d51d3afe0bad3#egg=topostats
tornado @ file:///C:/ci/tornado_1662476985533/work
tqdm==4.64.1
traitlets @ file:///C:/b/abs_e5m_xjjl94/croot/traitlets_1671143896266/work
typing_extensions @ file:///C:/b/abs_5em9ekwz24/croot/typing_extensions_1686602003259/work
urllib3 @ file:///C:/b/abs_889_loyqv4/croot/urllib3_1686163174463/work
versioneer @ file:///C:/b/abs_03auryi04q/croot/versioneer_1683056059505/work
wcwidth @ file:///Users/ktietz/demo/mc3/conda-bld/wcwidth_1629357192024/work
webencodings==0.5.1
websocket-client @ file:///C:/ci_310/websocket-client_1642093970919/work
widgetsnbextension @ file:///C:/b/abs_882k4_4kdf/croot/widgetsnbextension_1679313880295/work
win-inet-pton @ file:///C:/ci_310/win_inet_pton_1642658466512/work
y-py @ file:///C:/b/abs_b7f5go6r0j/croot/y-py_1683662173571/work
ypy-websocket @ file:///C:/b/abs_4e65ywlnv8/croot/ypy-websocket_1684172103529/work```
MaxGamill-Sheffield commented 7 months ago

Is a result of the dnatracing padding not being enough. Changing this vaue in the config from 1 to 3 fixes this:

In get_fitted_traces the variable index_width determines how far from the trace coordinate to look for a higher region to better align the trace onto the DNA backbone. However, in some cases this value can be too large, and indexes outside the cropped images' limits.

It seems like the region of the code which checks if it is within the bounds:

elif trace_coordinate[0] >= (self.number_of_rows - index_width):
...
trace_coordinate[0] = self.number_of_rows - index_width # needs to add -1 to be within the max image index