search

AFM-SPM / TopoStats

An AFM image analysis program to batch process data and obtain statistics from images
https://afm-spm.github.io/TopoStats/
GNU Lesser General Public License v3.0
60 stars 11 forks source link

Add support for partial config files #981

Closed SylviaWhittle closed 2 weeks ago

SylviaWhittle commented 4 weeks ago

As discussed in the recent TopoStats catchup, the config file is about to explode with parameters. This will result in a rather cumbersome config file for users which will be intimidating.

After fielding a few options to the experimentalists, it was decided that the least bad way forward might be to add support for partial config files, where any missing fields are added in from the default_config.yaml.

This PR is attempting to do just that. Though I'm sure I've forgotten something.

Oh also this PR is entirely type safe (I think?, at least according to mypy!) It brings some type safety to the beginning of run_topostats.py and marks my first foray into generic typing!

SylviaWhittle commented 4 weeks ago

I'm sure I've probably forgotten to do a chore or add something so please let me know what they are 😅

But the tests pass (so far) so I think that's a good sign.

Always scary editing the early code in TopoStats

SylviaWhittle commented 3 weeks ago

Rebase onto main after #985

MaxGamill-Sheffield commented 2 weeks ago

Failing test:

FAILED tests/tracing/test_disordered_tracing.py::test_disordered_trace_grain[test height bias: thick curve height weighting outer, strong height bias] - TypeError: Can only insert double* at [4] in {i8*, i8*, i64, i64, double*, [1 x i64], [1 x i64]}: got float*

on MacOS 3.9 & 3.11 (so probably 3.10 too)

Annoyingly I cannot recreate this using the testing function in VSCode but I can using pytest in the terminal directly.

Seems to be due to Skan version 0.12.0. Using the main's pinned version at 0.11.0 solves the failing tests.

I'll work out how to do a cherry pick commit now :)

SylviaWhittle commented 2 weeks ago

Thank you very much @MaxGamill-Sheffield

SylviaWhittle commented 2 weeks ago

Windows 🙃

image
MaxGamill-Sheffield commented 2 weeks ago

Windows 3.9 test fails but looks fishy and nothing:

INFO     topostats:io.py:1108 Extracting image from tests\resources\test_image\minicircle_small.topostats
Windows fatal exception: code 0x80000003

...(multi-threading logs)

Thread 0x00000dc8 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 312 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\multiprocessing\pool.py", line 858 in next
  File "D:\a\TopoStats\TopoStats\topostats\run_topostats.py", line 190 in run_topostats
  File "D:\a\TopoStats\TopoStats\topostats\entry_point.py", line 372 in entry_point
  File "D:\a\TopoStats\TopoStats\tests\test_run_topostats.py", line 100 in test_run_topostats_process_all
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\python.py", line 159 in pytest_pyfunc_call
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\python.py", line 1627 in runtest
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 174 in pytest_runtest_call
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 242 in <lambda>
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 341 in from_call
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 241 in call_and_report
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 132 in runtestprotocol
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 113 in pytest_runtest_protocol
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\main.py", line 362 in pytest_runtestloop
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\main.py", line 337 in _main
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\main.py", line 283 in wrap_session
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\main.py", line 330 in pytest_cmdline_main
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_callers.py", line 103 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_hooks.py", line 513 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\config\__init__.py", line 175 in main
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\config\__init__.py", line 201 in console_main
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\Scripts\pytest.exe\__main__.py", line 7 in <module>
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\runpy.py", line 87 in _run_code
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\runpy.py", line 197 in _run_module_as_main
Error: Process completed with exit code 1.

but I can't test this on my machine

SylviaWhittle commented 2 weeks ago

How about this for a simple config?

image

This is probably something that will be argued about before deciding on a single simple config, so we can always edit it later once there is consensus.

alicepyne commented 2 weeks ago

Looks brill to me (&max)

On 7 Nov 2024, at 15:48, Sylvia Whittle @.***> wrote:

How about this for a simple config? image.png (view on web) https://github.com/user-attachments/assets/8d3e60b1-a992-4571-beb5-30ab743b8763 This is probably something that will be argued about before deciding on a single simple config, so we can always edit it later once there is consensus.

— Reply to this email directly, view it on GitHub https://github.com/AFM-SPM/TopoStats/pull/981#issuecomment-2462576584, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADJSFMOTAQCDVAMTERNQPMDZ7ODT5AVCNFSM6AAAAABQPSXZMSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRSGU3TMNJYGQ. You are receiving this because you are subscribed to this thread.

SylviaWhittle commented 2 weeks ago

Looks like the tests pass.

@MaxGamill-Sheffield I was unable to replicate your issue with the null overrides. It appears to work on my end, sent you the details in slack since I don't get notifications on here reliably.