python ./preprocessing/patch_extraction/main_extraction.py

[Transformer-man]

when I run python ./preprocessing/patch_extraction/main_extraction.py ,having error.

[FabianHoerst]

Could you provide me with more information? What command did you used? What was your cli command? Did you pass over a config file or directly passed over arguments via the cli? It seems like all the fields in the pydantic cli configuration are missing, which should not happen. I am going to investigate it

Could you also please run the example file (python3 ./preprocessing/patch_extraction/main_extraction.py --config ./example/preprocessing_example.yaml) or did the error occurred while running the example script for patch extraction?

[FabianHoerst]

I also tested the cli for the example file with:

python3 ./preprocessing/patch_extraction/main_extraction.py \
    --wsi_paths ./example/TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE.svs \
    --output_path ./example/output/preprocessing \
    --patch_size 1024 \
    --patch_overlap 6.25 \
    --target_mag 40 \
    --processes 6 \
    --patches_per_batch 4 \
    --overwrite \
    --min_intersection_ratio 0.05

Which is working well for me

[FabianHoerst]

Eine andere Idee: Versuchst du den Code selber zu debuggen? Dann würde ich dir empfehlen, dies über eine eigene debug Datei zu machen, um die Parameter über die CLI zu übergeben. Es muss dann die launch.json-Datei im .vscode Ordner angepasst werden, und der Debugger für diese datei gestartet werden (see: https://code.visualstudio.com/docs/editor/debugging#_launch-versus-attach-configurations). The launch.json file for the example file could look like this:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Preprocessing",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal",
            "justMyCode": false,
            "subProcess": true,
            "args": [
                "--wsi_paths", "./example/TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE.svs",
                "--output_path", "./example/output/preprocessing",
                "--patch_size", "1024",
                "--patch_overlap", "6.25",
                "--target_mag", "40",
                "--processes", "6",
                "--patches_per_batch", "4",
                "--overwrite",
                "--min_intersection_ratio", "0.05"
            ]
        }
    ]
}

You can start debugging then via:

Debugging this gives me the following output:

2023-07-05 06:01:16,516 [INFO] - Data store directory: /homes/fhoerst/histo-projects/CellViT/example/output/preprocessing
2023-07-05 06:01:16,518 [INFO] - Images found: 1
2023-07-05 06:01:16,521 [INFO] - Annotations found: 0
2023-07-05 06:01:20,935 [INFO] - Removing complete dataset! This may take a while.
2023-07-05 06:01:21,841 [INFO] - Using 6 processes.
2023-07-05 06:01:21,906 [INFO] - **************************************************************************************************************************************
2023-07-05 06:01:21,908 [INFO] - 1/1: TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE.svs
2023-07-05 06:01:21,910 [INFO] - Computing patches for TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE.svs
2023-07-05 06:02:21,552 [INFO] - TCGA-V5-A7RE-11A-01-TS1.57401526-EF9E-49AC-8FF6-B4F9652311CE.svs: Processing 58 patches.
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:15<00:00,  1.04s/it]
2023-07-05 06:02:42,571 [INFO] - Total patches sampled: 58
2023-07-05 06:02:43,437 [INFO] - Patches saved to: /homes/fhoerst/histo-projects/CellViT/example/output/preprocessing
2023-07-05 06:02:43,438 [INFO] - Total patches sampled for all WSI: 58
2023-07-05 06:02:43,450 [INFO] - Time usage: 0:01:21.543524

[Transformer-man]

Thank you very much for your help. I will try again following your steps

[Transformer-man]

It seems like the same mistake

[Transformer-man]

When I run python3 ./preprocessing/patch_extraction/main_extraction.py --config ./example/preprocessing_example.yaml，having the same mistake.

[FabianHoerst]

Have you checked the debugging launch.json I provided? Could you please set a breakpoint in line 534 in file preprocessing/patch_extraction/src/cli.py and check the output of the opt_dict?

[Transformer-man]

Copy that, I'll do it right away

[Transformer-man]

[FabianHoerst]

Maybe we should take this discussion offline. Contact me at fabian.hoerst@uk-essen.de.

Screenshots itself are not very helpful for remote debugging. I need to know the exact commands you executed and every entry of the yaml_config dict to reproduce this error. It seems like you tried to debug by using a config file, but the launch json uses the argparse commands and not a yaml file.

[Transformer-man]

[Transformer-man]

An error was reported when running to this line

[Transformer-man]

Okay, I'll contact you right away

[FabianHoerst]

Very strange behavior. My yaml_config dict looks exactly the same as yours:

{'wsi_paths': './example/TCGA-V5-A7...2311CE.svs', 'output_path': './example/output/preprocessing', 'patch_size': 1024, 'patch_overlap': 6.25, 'target_mag': 40, 'processes': 6, 'patches_per_batch': 4, 'overwrite': True, 'normalize_stains': True, 'min_intersection_ratio': 0.05}

Executing PreProcessingYamlConfig(**yaml_config) should return a PreProcessingYamlConfig PyDantic object with all other values initialized by None or a default value.

The PreProcessingYamlConfig is structured such that each value is optional. This should not happen. What is your pydantic version?

[Transformer-man]

Yes, I haven't made any changes, except for using my own data. Everything else is based on your side. But PreProcessingYamlConfig(**yaml_config) has error.

[Transformer-man]

I've found the reason, I'm really stupid. I originally used Pydantic version 2.0, but yours is version 1.10.4, which is due to the version.

[Transformer-man]

Thank you very much for your help. I have wasted so much of your time.

[FabianHoerst]

Ok, good to know the version mismatch. Thanks for resolving this and a note to myself for next projects with new paydantic api! In general, please use provided environments. I am also currently implementing a speedup for inference, should be merged next week, as the code was not optimized right now. I expect to have a 10x speed up for inference with cell_detection.py.

[Transformer-man]

Very good work, thanks for your help