search

BlueBrain / morphoclass

Neuronal morphology preparation and classification using Machine Learning.
https://morphoclass.readthedocs.io
Apache License 2.0
8 stars 4 forks source link

MCAR curation deletes several neurites so resulting morpholiges are unusable #70

Closed FrancescoCasalegno closed 2 years ago

FrancescoCasalegno commented 2 years ago

Issue part 1: MCAR curation runs Sanitize (which deletes whole neurites)

MCAR step Sanitize cannot be skipped, see https://github.com/BlueBrain/morphoclass/issues/4#issuecomment-1154943646. Indeed, even with the line https://github.com/BlueBrain/morphoclass/blob/d8383b5b56eaa393f413346b90d7f417c8d166b1/dvc/data/mcar-luigi.cfg#L42 running 

dvc repro -fs mcar-curation-janelia

produces the following output (including info on Santize being executed!)

Running stage 'mcar-curation-janelia':                                
> MCAR_INPUT_CSV_PATH="data/preprocessed-janelia.csv" MCAR_OUTPUT_DIR="data/mcar-curation/janelia" LUIGI_CONFIG_PATH="data/mcar-luigi.cfg" LUIGI_LOGGING_PATH="data/mcar-luigi-logging.cfg" morphology_workflows --local-scheduler Curate
/usr/local/lib/python3.8/dist-packages/luigi/configuration/core.py:65: UserWarning: Config file does not exist: luigi.cfg
  warnings.warn("Config file does not exist: {path}".format(path=path))
13:25:18 luigi (I) logging configured via *.conf file
13:25:18 luigi-interface (I) Informed scheduler that task   Curate_data_data_preprocesse_True_4b6c62e2e9   has status   PENDING
13:25:18 luigi-interface (I) Informed scheduler that task   PlotMorphologies_data_data_preprocesse_morph_name_ff37344527   has status   PENDING
13:25:18 luigi-interface (I) Informed scheduler that task   Resample_data_data_preprocesse_morph_name_80b35927ba   has status   PENDING
13:25:18 luigi-interface (I) Informed scheduler that task   Orient_data_data_preprocesse_morph_name_1af5e18b1e   has status   PENDING
13:25:18 luigi-interface (I) Informed scheduler that task   EnsureNeuritesOutsideSoma_data_data_preprocesse_morph_name_8820b42a25   has status   PENDING
13:25:18 luigi-interface (I) Informed scheduler that task   Align_data_data_preprocesse_null_2dd8f6bb1f   has status   PENDING
13:25:18 luigi-interface (I) Informed scheduler that task   ErrorsReport_data_data_preprocesse_error_report_csv_7cc5e4f8fd   has status   PENDING
13:25:18 luigi-interface (I) Informed scheduler that task   PlotErrors_data_data_preprocesse_morph_name_7d087863ed   has status   PENDING
13:25:18 luigi-interface (I) Informed scheduler that task   PlotMarkers_data_data_preprocesse_morph_name_7d087863ed   has status   PENDING
13:25:18 luigi-interface (I) Informed scheduler that task   DetectErrors_data_data_preprocesse_morph_name_820cb605ed   has status   PENDING
13:25:18 luigi-interface (I) Informed scheduler that task   Recenter_data_data_preprocesse_morph_name_6dcaf6d874   has status   PENDING
13:25:18 luigi-interface (I) Informed scheduler that task   CheckNeurites_5_data_data_preprocesse_37e31ad2c7   has status   PENDING
13:25:18 luigi-interface (I) Informed scheduler that task   Sanitize_data_data_preprocesse_morph_name_88233f64ef   has status   PENDING
13:25:18 luigi-interface (I) Informed scheduler that task   ExtractMarkers_data_data_preprocesse_morph_name_8820b42a25   has status   PENDING
13:25:18 luigi-interface (I) Informed scheduler that task   Collect_data_data_preprocesse_morph_name_88233f64ef   has status   PENDING
13:25:18 luigi-interface (I) Done scheduling tasks
13:25:18 luigi-interface (I) Running Worker with 1 processes
13:25:18 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) running   Collect(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data)
13:25:18 data_validation_framework.task (I) Input dataset: /workdir/dvc/data/preprocessed-janelia.csv
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 106/106 [00:04<00:00, 22.67it/s]
13:25:23 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) done      Collect(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data)
13:25:23 data_validation_framework.task (I) ==========================================
13:25:23 data_validation_framework.task (I) SUMMARY Collect: 106 / 106 passed
13:25:23 data_validation_framework.task (I) ==========================================
13:25:23 luigi-interface (I) Informed scheduler that task   Collect_data_data_preprocesse_morph_name_88233f64ef   has status   DONE
13:25:23 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) running   CheckNeurites(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, mock_soma_type=spherical, axon_n_section_min=5)
13:25:23 data_validation_framework.task (I) Input dataset: /workdir/dvc/data/preprocessed-janelia.csv
  1%|█▋                                                                                                                                                                            | 1/106 [00:00<00:19,  5.47it/s]13:25:23 morphology_workflows.curation (I) Adding a spherical mock soma at [4842.3955 2272.8381 4161.736 ] of radius 8.584753.
13:25:23 morphology_workflows.curation (I) Adding a spherical mock soma at [4769.5615 1887.6177 4808.7417] of radius 17.688837.
13:25:23 morphology_workflows.curation (I) Adding a spherical mock soma at [2983.603  2536.7551 4621.639 ] of radius 19.009853.
13:25:23 morphology_workflows.curation (I) Adding a spherical mock soma at [6947.3564 2839.913  2662.1196] of radius 16.323946.
  5%|████████▏                                                                                                                                                                     | 5/106 [00:00<00:15,  6.57it/s]13:25:24 morphology_workflows.curation (I) Adding a spherical mock soma at [6965.206  3217.9722 3005.629 ] of radius 14.1033325.
13:25:24 morphology_workflows.curation (I) Adding a spherical mock soma at [6829.0586 2091.2258 3451.9224] of radius 22.648233.
  7%|███████████▍                                                                                                                                                                  | 7/106 [00:00<00:12,  8.06it/s]13:25:24 morphology_workflows.curation (I) Adding a spherical mock soma at [3681.6963 1826.6893 5069.1973] of radius 7.0332794.
  8%|█████████████▏                                                                                                                                                                | 8/106 [00:01<00:14,  6.89it/s]13:25:24 morphology_workflows.curation (I) Adding a spherical mock soma at [6772.697  2261.4219 2847.676 ] of radius 13.481153.
  8%|██████████████▊                                                                                                                                                               | 9/106 [00:01<00:15,  6.22it/s]13:25:24 morphology_workflows.curation (I) Adding a spherical mock soma at [7071.489  3206.688  2557.5164] of radius 13.592491.
13:25:24 morphology_workflows.curation (I) Adding a spherical mock soma at [7090.8594 2379.4612 3258.4294] of radius 21.162338.
 10%|█████████████████▉                                                                                                                                                           | 11/106 [00:01<00:14,  6.52it/s]13:25:24 morphology_workflows.curation (I) Adding a spherical mock soma at [3324.625  2216.2776 4687.8066] of radius 14.723083.
 11%|███████████████████▌                                                                                                                                                         | 12/106 [00:01<00:17,  5.31it/s]13:25:25 morphology_workflows.curation (I) Adding a spherical mock soma at [7101.55  3095.909 2520.919] of radius 12.017457.
13:25:25 morphology_workflows.curation (I) Adding a spherical mock soma at [6908.938  2023.15   3621.3965] of radius 14.4038315.
 13%|██████████████████████▊                                                                                                                                                      | 14/106 [00:02<00:14,  6.54it/s]13:25:25 morphology_workflows.curation (I) Adding a spherical mock soma at [7142.453  2977.193  3272.3047] of radius 11.149239.
 14%|████████████████████████▍                                                                                                                                                    | 15/106 [00:02<00:13,  6.79it/s]13:25:25 morphology_workflows.curation (I) Adding a spherical mock soma at [3980.8523 2312.596  4483.018 ] of radius 13.908157.
 15%|██████████████████████████                                                                                                                                                   | 16/106 [00:02<00:16,  5.57it/s]13:25:25 morphology_workflows.curation (I) Adding a spherical mock soma at [7530.4    2335.9543 3982.9395] of radius 13.1126.
 16%|███████████████████████████▋                                                                                                                                                 | 17/106 [00:02<00:15,  5.85it/s]13:25:25 morphology_workflows.curation (I) Adding a spherical mock soma at [7188.846  2347.553  3264.3901] of radius 18.551317.
 17%|█████████████████████████████▍                                                                                                                                               | 18/106 [00:02<00:13,  6.51it/s]13:25:26 morphology_workflows.curation (I) Adding a spherical mock soma at [6567.63   2183.9429 2745.7468] of radius 11.647071.
 18%|███████████████████████████████                                                                                                                                              | 19/106 [00:02<00:12,  6.75it/s]13:25:26 morphology_workflows.curation (I) Adding a spherical mock soma at [4134.479 2092.271 3855.075] of radius 15.296577.
 19%|████████████████████████████████▋                                                                                                                                            | 20/106 [00:03<00:14,  6.02it/s]13:25:26 morphology_workflows.curation (I) Adding a spherical mock soma at [7414.5913 2568.1753 3688.1655] of radius 13.774002.
 20%|██████████████████████████████████▎                                                                                                                                          | 21/106 [00:03<00:18,  4.52it/s]13:25:26 morphology_workflows.curation (I) Adding a spherical mock soma at [7192.6064 2885.1738 3252.6938] of radius 12.232318.
13:25:26 morphology_workflows.curation (I) Adding a spherical mock soma at [7077.4014 2796.8416 3212.533 ] of radius 13.09981.
13:25:26 morphology_workflows.curation (I) Adding a spherical mock soma at [6895.2783 2125.3193 3607.9685] of radius 17.422413.
 22%|█████████████████████████████████████▌                                                                                                                                       | 23/106 [00:03<00:13,  6.19it/s]13:25:27 morphology_workflows.curation (I) Adding a spherical mock soma at [7386.0776 2622.8152 3736.4163] of radius 19.24956.
 24%|████████████████████████████████████████▊                                                                                                                                    | 25/106 [00:04<00:14,  5.51it/s]13:25:27 morphology_workflows.curation (I) Adding a spherical mock soma at [6854.152  3038.2888 3043.48  ] of radius 10.8120365.
 25%|██████████████████████████████████████████▍                                                                                                                                  | 26/106 [00:04<00:14,  5.48it/s]13:25:27 morphology_workflows.curation (I) Adding a spherical mock soma at [3859.8774 2108.5637 3937.9783] of radius 1.2983407.
13:25:27 morphology_workflows.curation (I) Adding a spherical mock soma at [7515.398  2210.0369 3897.1328] of radius 17.043022.
 26%|█████████████████████████████████████████████▋                                                                                                                               | 28/106 [00:04<00:10,  7.70it/s]13:25:27 morphology_workflows.curation (I) Adding a spherical mock soma at [6996.115  2287.7678 3264.12  ] of radius 16.522657.
 27%|███████████████████████████████████████████████▎                                                                                                                             | 29/106 [00:04<00:13,  5.76it/s]13:25:28 morphology_workflows.curation (I) Adding a spherical mock soma at [7198.291  2981.175  3150.9492] of radius 18.099882.
 28%|████████████████████████████████████████████████▉                                                                                                                            | 30/106 [00:04<00:13,  5.58it/s]13:25:28 morphology_workflows.curation (I) Adding a spherical mock soma at [7234.47   2927.2827 3620.6016] of radius 21.001863.
13:25:28 morphology_workflows.curation (I) Adding a spherical mock soma at [7333.863  2376.8027 3766.3389] of radius 11.346281.
 30%|████████████████████████████████████████████████████▏                                                                                                                        | 32/106 [00:05<00:13,  5.66it/s]13:25:28 morphology_workflows.curation (I) Adding a spherical mock soma at [3763.1748 2367.0574 4157.588 ] of radius 4.340142.
 31%|█████████████████████████████████████████████████████▊                                                                                                                       | 33/106 [00:05<00:12,  5.88it/s]13:25:28 morphology_workflows.curation (I) Adding a spherical mock soma at [6588.8516 2323.77   2663.5122] of radius 11.400868.
 32%|███████████████████████████████████████████████████████▍                                                                                                                     | 34/106 [00:05<00:11,  6.34it/s]13:25:28 morphology_workflows.curation (I) Adding a spherical mock soma at [8183.597  1463.6411 8998.595 ] of radius 9.3544445.
 33%|█████████████████████████████████████████████████████████                                                                                                                    | 35/106 [00:05<00:10,  6.82it/s]13:25:28 morphology_workflows.curation (I) Adding a spherical mock soma at [7166.352  2342.189  4022.2476] of radius 12.238532.
 34%|██████████████████████████████████████████████████████████▊                                                                                                                  | 36/106 [00:06<00:14,  4.87it/s]13:25:29 morphology_workflows.curation (I) Adding a spherical mock soma at [4374.646  2202.8489 4174.637 ] of radius 4.421457.
13:25:29 morphology_workflows.curation (I) Adding a spherical mock soma at [6837.259  2582.749  3314.2239] of radius 10.127118.
13:25:29 morphology_workflows.curation (I) Adding a spherical mock soma at [6884.422  2507.0713 2628.6545] of radius 16.382744.
 37%|███████████████████████████████████████████████████████████████▋                                                                                                             | 39/106 [00:06<00:09,  6.95it/s]13:25:29 morphology_workflows.curation (I) Adding a spherical mock soma at [7133.901  2179.53   3349.8052] of radius 13.538229.
 38%|█████████████████████████████████████████████████████████████████▎                                                                                                           | 40/106 [00:06<00:12,  5.46it/s]13:25:29 morphology_workflows.curation (I) Adding a spherical mock soma at [4510.1123 2291.9841 4244.0625] of radius 2.0333145.
13:25:29 morphology_workflows.curation (I) Adding a spherical mock soma at [6730.175 2546.921 3077.274] of radius 13.411158.
13:25:30 morphology_workflows.curation (I) Adding a spherical mock soma at [6093.5547 2441.8914 2812.3801] of radius 9.244009.
 41%|██████████████████████████████████████████████████████████████████████▏                                                                                                      | 43/106 [00:06<00:09,  6.87it/s]13:25:30 morphology_workflows.curation (I) Adding a spherical mock soma at [6870.3022 2302.6477 3552.168 ] of radius 11.7755785.
 42%|███████████████████████████████████████████████████████████████████████▊                                                                                                     | 44/106 [00:07<00:11,  5.38it/s]13:25:30 morphology_workflows.curation (I) Adding a spherical mock soma at [6999.1416 2222.43   3434.8923] of radius 13.879099.
13:25:30 morphology_workflows.curation (I) Adding a spherical mock soma at [4124.317 2200.784 4591.328] of radius 4.2384386.
13:25:30 morphology_workflows.curation (I) Adding a spherical mock soma at [6792.4053 2243.4988 2761.775 ] of radius 14.64633.
 44%|████████████████████████████████████████████████████████████████████████████▋                                                                                                | 47/106 [00:07<00:07,  7.44it/s]13:25:30 morphology_workflows.curation (I) Adding a spherical mock soma at [6808.253  2447.0986 3286.1873] of radius 19.082016.
 45%|██████████████████████████████████████████████████████████████████████████████▎                                                                                              | 48/106 [00:07<00:10,  5.67it/s]13:25:31 morphology_workflows.curation (I) Adding a spherical mock soma at [6857.3257 2844.3184 3265.5857] of radius 10.601056.
 46%|███████████████████████████████████████████████████████████████████████████████▉                                                                                             | 49/106 [00:08<00:09,  6.07it/s]13:25:31 morphology_workflows.curation (I) Adding a spherical mock soma at [4325.738  2010.1017 4633.217 ] of radius 2.1212962.
13:25:31 morphology_workflows.curation (I) Adding a spherical mock soma at [6424.6885 2193.3687 2768.6543] of radius 10.577901.
 48%|███████████████████████████████████████████████████████████████████████████████████▏                                                                                         | 51/106 [00:08<00:06,  8.00it/s]13:25:31 morphology_workflows.curation (I) Adding a spherical mock soma at [6968.9844 2482.9119 3329.896 ] of radius 8.076381.
13:25:31 morphology_workflows.curation (I) Adding a spherical mock soma at [7080.5283 2201.9673 3539.7268] of radius 16.311554.
 50%|██████████████████████████████████████████████████████████████████████████████████████▌                                                                                      | 53/106 [00:08<00:08,  5.95it/s]13:25:31 morphology_workflows.curation (I) Adding a spherical mock soma at [6730.604 3050.58  2768.282] of radius 13.16303.
 51%|████████████████████████████████████████████████████████████████████████████████████████▏                                                                                    | 54/106 [00:08<00:08,  6.34it/s]13:25:32 morphology_workflows.curation (I) Adding a spherical mock soma at [3980.4119 2358.2268 4726.9   ] of radius 3.562996.
13:25:32 morphology_workflows.curation (I) Adding a spherical mock soma at [7005.1055 2466.446  3349.888 ] of radius 14.556274.
 53%|███████████████████████████████████████████████████████████████████████████████████████████▍                                                                                 | 56/106 [00:09<00:08,  6.04it/s]13:25:32 morphology_workflows.curation (I) Adding a spherical mock soma at [7103.4    2873.4521 3518.2673] of radius 12.732122.
 54%|█████████████████████████████████████████████████████████████████████████████████████████████                                                                                | 57/106 [00:09<00:07,  6.20it/s]13:25:32 morphology_workflows.curation (I) Adding a spherical mock soma at [7011.     2271.5862 2686.552 ] of radius 14.4785595.
 55%|██████████████████████████████████████████████████████████████████████████████████████████████▋                                                                              | 58/106 [00:09<00:07,  6.54it/s]13:25:32 morphology_workflows.curation (I) Adding a spherical mock soma at [7233.918  2988.3247 3330.5176] of radius 13.236062.
13:25:32 morphology_workflows.curation (I) Adding a spherical mock soma at [6934.1763 2007.1012 3940.0603] of radius 10.045484.
 57%|█████████████████████████████████████████████████████████████████████████████████████████████████▉                                                                           | 60/106 [00:09<00:06,  6.63it/s]13:25:32 morphology_workflows.curation (I) Adding a spherical mock soma at [6872.1274 2726.926  3591.9114] of radius 14.268309.
 58%|███████████████████████████████████████████████████████████████████████████████████████████████████▌                                                                         | 61/106 [00:09<00:07,  5.85it/s]13:25:33 morphology_workflows.curation (I) Adding a spherical mock soma at [7206.5083 2300.2334 3458.28  ] of radius 10.550019.
13:25:33 morphology_workflows.curation (I) Adding a spherical mock soma at [7100.4653 2103.4993 3680.0466] of radius 16.447405.
 59%|██████████████████████████████████████████████████████████████████████████████████████████████████████▊                                                                      | 63/106 [00:10<00:05,  7.80it/s]13:25:33 morphology_workflows.curation (I) Adding a spherical mock soma at [6950.426  3128.4019 3013.692 ] of radius 9.674245.
 60%|████████████████████████████████████████████████████████████████████████████████████████████████████████▍                                                                    | 64/106 [00:10<00:06,  6.83it/s]13:25:33 morphology_workflows.curation (I) Adding a spherical mock soma at [7135.5303 2389.328  3537.8   ] of radius 16.266085.
 61%|██████████████████████████████████████████████████████████████████████████████████████████████████████████                                                                   | 65/106 [00:10<00:07,  5.38it/s]13:25:33 morphology_workflows.curation (I) Adding a spherical mock soma at [6806.2866 2453.033  3115.5957] of radius 14.207184.
13:25:33 morphology_workflows.curation (I) Adding a spherical mock soma at [5293.959  1286.079  5824.8076] of radius 13.590938.
13:25:33 morphology_workflows.curation (I) Adding a spherical mock soma at [7083.837  3109.7031 3061.5073] of radius 14.665312.
 64%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▉                                                              | 68/106 [00:10<00:05,  7.24it/s]13:25:34 morphology_workflows.curation (I) Adding a spherical mock soma at [6939.9673 2497.5244 3115.2908] of radius 14.768359.
 65%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                                            | 69/106 [00:11<00:06,  5.34it/s]13:25:34 morphology_workflows.curation (I) Adding a spherical mock soma at [6992.287  3067.     2921.3643] of radius 11.481251.
13:25:34 morphology_workflows.curation (I) Adding a spherical mock soma at [5196.1333 1587.2928 6038.2017] of radius 5.798034.
13:25:34 morphology_workflows.curation (I) Adding a spherical mock soma at [7277.464  2399.2437 3181.0776] of radius 16.837584.
 68%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                                       | 72/106 [00:11<00:04,  7.71it/s]13:25:34 morphology_workflows.curation (I) Adding a spherical mock soma at [6860.643  2450.9612 2886.549 ] of radius 16.504568.
 69%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏                                                     | 73/106 [00:11<00:05,  5.68it/s]13:25:35 morphology_workflows.curation (I) Adding a spherical mock soma at [6874.835  2653.8298 2712.3572] of radius 19.501934.
13:25:35 morphology_workflows.curation (I) Adding a spherical mock soma at [5239.314 1206.47  6034.675] of radius 13.751341.
 71%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍                                                  | 75/106 [00:12<00:05,  6.18it/s]13:25:35 morphology_workflows.curation (I) Adding a spherical mock soma at [5023.621  1745.9634 4121.8086] of radius 26.148209.
 72%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████                                                 | 76/106 [00:12<00:05,  5.71it/s]13:25:35 morphology_workflows.curation (I) Adding a spherical mock soma at [7114.4917 2184.5667 3119.58  ] of radius 16.406845.
 73%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                               | 77/106 [00:12<00:04,  6.28it/s]13:25:35 morphology_workflows.curation (I) Adding a spherical mock soma at [6951.258  3292.6538 3271.8699] of radius 14.908849.
13:25:35 morphology_workflows.curation (I) Adding a spherical mock soma at [4205.9834 2007.6415 3638.79  ] of radius 6.9434924.
 75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉                                            | 79/106 [00:12<00:04,  5.50it/s]13:25:36 morphology_workflows.curation (I) Adding a spherical mock soma at [4773.272  2359.8086 4145.6216] of radius 13.493909.
 75%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                          | 80/106 [00:12<00:04,  5.77it/s]13:25:36 morphology_workflows.curation (I) Adding a spherical mock soma at [6865.496  2292.0315 2509.7349] of radius 17.674744.
13:25:36 morphology_workflows.curation (I) Adding a spherical mock soma at [6830.751  2124.044  2888.1365] of radius 17.642632.
 77%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                                       | 82/106 [00:13<00:04,  5.79it/s]13:25:36 morphology_workflows.curation (I) Adding a spherical mock soma at [4632.234  1861.6147 4036.0645] of radius 7.3514543.
 78%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍                                     | 83/106 [00:13<00:03,  6.32it/s]13:25:36 morphology_workflows.curation (I) Adding a spherical mock soma at [4995.6196 1582.89   4251.5312] of radius 13.200067.
 79%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████                                    | 84/106 [00:13<00:03,  6.47it/s]13:25:36 morphology_workflows.curation (I) Adding a spherical mock soma at [6783.79   2517.5984 2730.94  ] of radius 13.821287.
 80%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                  | 85/106 [00:13<00:03,  6.54it/s]13:25:36 morphology_workflows.curation (I) Adding a spherical mock soma at [6769.1562 3010.73   3418.1064] of radius 15.31764.
 81%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎                                | 86/106 [00:14<00:04,  4.73it/s]13:25:37 morphology_workflows.curation (I) Adding a spherical mock soma at [4959.126  2298.6619 4332.7583] of radius 21.72643.
13:25:37 morphology_workflows.curation (I) Adding a spherical mock soma at [6921.242  2887.962  2809.7659] of radius 10.99052.
 83%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                             | 88/106 [00:14<00:02,  6.35it/s]13:25:37 morphology_workflows.curation (I) Adding a spherical mock soma at [4674.7095 2234.2576 4395.938 ] of radius 1.802695.
13:25:37 morphology_workflows.curation (I) Adding a spherical mock soma at [6800.126  2883.2014 3368.773 ] of radius 12.750242.
 85%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉                          | 90/106 [00:14<00:02,  5.45it/s]13:25:37 morphology_workflows.curation (I) Adding a spherical mock soma at [5063.69   1734.4442 4376.3643] of radius 21.017069.
13:25:38 morphology_workflows.curation (I) Adding a spherical mock soma at [6837.872  2723.6108 2639.571 ] of radius 11.585951.
 87%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏                      | 92/106 [00:14<00:02,  6.89it/s]13:25:38 morphology_workflows.curation (I) Adding a spherical mock soma at [6993.994  2124.5908 3002.6807] of radius 16.954594.
 88%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                     | 93/106 [00:14<00:01,  7.28it/s]13:25:38 morphology_workflows.curation (I) Adding a spherical mock soma at [4669.3945 2257.3887 3921.2878] of radius 3.7015543.
 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍                   | 94/106 [00:15<00:02,  5.45it/s]13:25:38 morphology_workflows.curation (I) Adding a spherical mock soma at [5032.2324 1840.3082 4654.4316] of radius 22.039898.
 90%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████                  | 95/106 [00:15<00:01,  5.91it/s]13:25:38 morphology_workflows.curation (I) Adding a spherical mock soma at [6058.839  2590.8127 2468.953 ] of radius 12.639043.
13:25:38 morphology_workflows.curation (I) Adding a spherical mock soma at [7045.5146 2805.0325 2883.4038] of radius 12.386045.
 92%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎              | 97/106 [00:15<00:01,  6.99it/s]13:25:38 morphology_workflows.curation (I) Adding a spherical mock soma at [4659.543  2644.7402 3893.0142] of radius 2.9787107.
 92%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉             | 98/106 [00:15<00:01,  6.17it/s]13:25:39 morphology_workflows.curation (I) Adding a spherical mock soma at [4694.951  2065.2588 4675.3315] of radius 15.211363.
 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌           | 99/106 [00:16<00:01,  5.18it/s]13:25:39 morphology_workflows.curation (I) Adding a spherical mock soma at [6786.418 2638.536 2699.725] of radius 15.847733.
 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎         | 100/106 [00:16<00:01,  5.88it/s]13:25:39 morphology_workflows.curation (I) Adding a spherical mock soma at [7007.366  2270.8816 2817.25  ] of radius 14.097673.
13:25:39 morphology_workflows.curation (I) Adding a spherical mock soma at [4672.016  2214.7556 4102.199 ] of radius 2.6630929.
 96%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌      | 102/106 [00:16<00:00,  4.32it/s]13:25:40 morphology_workflows.curation (I) Adding a spherical mock soma at [6823.912  2140.2224 2743.367 ] of radius 16.35626.
13:25:40 morphology_workflows.curation (I) Adding a spherical mock soma at [4606.9204 2114.49   3890.7197] of radius 1.0.
 98%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊   | 104/106 [00:17<00:00,  3.91it/s]13:25:40 morphology_workflows.curation (I) Adding a spherical mock soma at [7115.154  2688.67   3006.8655] of radius 16.69183.
 99%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 105/106 [00:17<00:00,  4.13it/s]13:25:40 morphology_workflows.curation (I) Adding a spherical mock soma at [3918.2803 2864.0784 3957.0813] of radius 6.8580246.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 106/106 [00:18<00:00,  3.19it/s]13:25:41 morphology_workflows.curation (I) Adding a spherical mock soma at [4082.51   2098.2026 3519.0073] of radius 17.433077.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 106/106 [00:18<00:00,  5.68it/s]
13:25:41 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) done      CheckNeurites(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, mock_soma_type=spherical, axon_n_section_min=5)
13:25:41 data_validation_framework.task (I) ==========================================
13:25:41 data_validation_framework.task (I) SUMMARY CheckNeurites: 106 / 106 passed
13:25:41 data_validation_framework.task (I) ==========================================
13:25:41 luigi-interface (I) Informed scheduler that task   CheckNeurites_5_data_data_preprocesse_37e31ad2c7   has status   DONE
13:25:41 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) running   DetectErrors(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=True, min_range=50.0)
13:25:41 data_validation_framework.task (I) Input dataset: /workdir/dvc/data/preprocessed-janelia.csv
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 106/106 [00:00<00:00, 1955.75it/s]
13:25:42 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) done      DetectErrors(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=True, min_range=50.0)
13:25:42 data_validation_framework.task (I) ==========================================
13:25:42 data_validation_framework.task (I) SUMMARY DetectErrors: 106 / 106 passed
13:25:42 data_validation_framework.task (I) ==========================================
13:25:42 luigi-interface (I) Informed scheduler that task   DetectErrors_data_data_preprocesse_morph_name_820cb605ed   has status   DONE
13:25:42 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) running   ErrorsReport(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=False, error_report_path=error_report.csv)
13:25:42 data_validation_framework.task (I) Input dataset: /workdir/dvc/data/preprocessed-janelia.csv
13:25:42 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) done      ErrorsReport(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=False, error_report_path=error_report.csv)
13:25:42 data_validation_framework.task (I) ==========================================
13:25:42 data_validation_framework.task (I) SUMMARY ErrorsReport: 106 / 106 passed
13:25:42 data_validation_framework.task (I) ==========================================
13:25:42 luigi-interface (I) Informed scheduler that task   ErrorsReport_data_data_preprocesse_error_report_csv_7cc5e4f8fd   has status   DONE
13:25:42 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) running   Sanitize(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data)
13:25:42 data_validation_framework.task (I) Input dataset: /workdir/dvc/data/preprocessed-janelia.csv
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 106/106 [00:03<00:00, 29.98it/s]
13:25:45 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) done      Sanitize(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data)
13:25:45 data_validation_framework.task (I) ==========================================
13:25:45 data_validation_framework.task (I) SUMMARY Sanitize: 106 / 106 passed
13:25:45 data_validation_framework.task (I) ==========================================
13:25:45 luigi-interface (I) Informed scheduler that task   Sanitize_data_data_preprocesse_morph_name_88233f64ef   has status   DONE
13:25:45 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) running   Recenter(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=False)
13:25:45 data_validation_framework.task (I) Input dataset: /workdir/dvc/data/preprocessed-janelia.csv
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 106/106 [00:02<00:00, 46.39it/s]
13:25:47 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) done      Recenter(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=False)
13:25:47 data_validation_framework.task (I) ==========================================
13:25:47 data_validation_framework.task (I) SUMMARY Recenter: 106 / 106 passed
13:25:47 data_validation_framework.task (I) ==========================================
13:25:47 luigi-interface (I) Informed scheduler that task   Recenter_data_data_preprocesse_morph_name_6dcaf6d874   has status   DONE
13:25:47 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) running   PlotErrors(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=True, with_plotly=True)
13:25:47 data_validation_framework.task (I) Input dataset: /workdir/dvc/data/preprocessed-janelia.csv
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 106/106 [00:00<00:00, 1945.32it/s]
13:25:48 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) done      PlotErrors(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=True, with_plotly=True)
13:25:48 data_validation_framework.task (I) ==========================================
13:25:48 data_validation_framework.task (I) SUMMARY PlotErrors: 106 / 106 passed
13:25:48 data_validation_framework.task (I) ==========================================
13:25:48 luigi-interface (I) Informed scheduler that task   PlotErrors_data_data_preprocesse_morph_name_7d087863ed   has status   DONE
13:25:48 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) running   Orient(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, pia_direction=y)
13:25:48 data_validation_framework.task (I) Input dataset: /workdir/dvc/data/preprocessed-janelia.csv
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 106/106 [00:01<00:00, 53.02it/s]
13:25:50 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) done      Orient(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, pia_direction=y)
13:25:50 data_validation_framework.task (I) ==========================================
13:25:50 data_validation_framework.task (I) SUMMARY Orient: 106 / 106 passed
13:25:50 data_validation_framework.task (I) ==========================================
13:25:50 luigi-interface (I) Informed scheduler that task   Orient_data_data_preprocesse_morph_name_1af5e18b1e   has status   DONE
13:25:50 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) running   Align(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=True, method=whole, neurite_type=apical, direction=null)
13:25:50 data_validation_framework.task (I) Input dataset: /workdir/dvc/data/preprocessed-janelia.csv
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 106/106 [00:00<00:00, 1979.20it/s]
13:25:50 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) done      Align(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=True, method=whole, neurite_type=apical, direction=null)
13:25:50 data_validation_framework.task (I) ==========================================
13:25:50 data_validation_framework.task (I) SUMMARY Align: 106 / 106 passed
13:25:50 data_validation_framework.task (I) ==========================================
13:25:50 luigi-interface (I) Informed scheduler that task   Align_data_data_preprocesse_null_2dd8f6bb1f   has status   DONE
13:25:50 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) running   EnsureNeuritesOutsideSoma(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=True)
13:25:50 data_validation_framework.task (I) Input dataset: /workdir/dvc/data/preprocessed-janelia.csv
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 106/106 [00:00<00:00, 2029.35it/s]
13:25:50 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) done      EnsureNeuritesOutsideSoma(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=True)
13:25:50 data_validation_framework.task (I) ==========================================
13:25:50 data_validation_framework.task (I) SUMMARY EnsureNeuritesOutsideSoma: 106 / 106 passed
13:25:50 data_validation_framework.task (I) ==========================================
13:25:50 luigi-interface (I) Informed scheduler that task   EnsureNeuritesOutsideSoma_data_data_preprocesse_morph_name_8820b42a25   has status   DONE
13:25:50 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) running   Resample(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=False, linear_density=1.0)
13:25:50 data_validation_framework.task (I) Input dataset: /workdir/dvc/data/preprocessed-janelia.csv
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 106/106 [00:18<00:00,  5.78it/s]
13:26:08 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) done      Resample(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=False, linear_density=1.0)
13:26:08 data_validation_framework.task (I) ==========================================
13:26:08 data_validation_framework.task (I) SUMMARY Resample: 106 / 106 passed
13:26:08 data_validation_framework.task (I) ==========================================
13:26:08 luigi-interface (I) Informed scheduler that task   Resample_data_data_preprocesse_morph_name_80b35927ba   has status   DONE
13:26:08 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) running   PlotMorphologies(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=True, with_plotly=False, with_realistic_diameters=True)
13:26:08 data_validation_framework.task (I) Input dataset: /workdir/dvc/data/preprocessed-janelia.csv
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 106/106 [00:00<00:00, 2053.51it/s]
13:26:08 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) done      PlotMorphologies(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=True, with_plotly=False, with_realistic_diameters=True)
13:26:08 data_validation_framework.task (I) ==========================================
13:26:08 data_validation_framework.task (I) SUMMARY PlotMorphologies: 106 / 106 passed
13:26:08 data_validation_framework.task (I) ==========================================
13:26:08 luigi-interface (I) Informed scheduler that task   PlotMorphologies_data_data_preprocesse_morph_name_ff37344527   has status   DONE
13:26:08 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) running   ExtractMarkers(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=True)
13:26:08 data_validation_framework.task (I) Input dataset: /workdir/dvc/data/preprocessed-janelia.csv
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 106/106 [00:00<00:00, 2024.53it/s]
13:26:08 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) done      ExtractMarkers(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=True)
13:26:08 data_validation_framework.task (I) ==========================================
13:26:08 data_validation_framework.task (I) SUMMARY ExtractMarkers: 106 / 106 passed
13:26:08 data_validation_framework.task (I) ==========================================
13:26:08 luigi-interface (I) Informed scheduler that task   ExtractMarkers_data_data_preprocesse_morph_name_8820b42a25   has status   DONE
13:26:08 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) running   PlotMarkers(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=True, with_plotly=True)
13:26:08 data_validation_framework.task (I) Input dataset: /workdir/dvc/data/preprocessed-janelia.csv
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 106/106 [00:00<00:00, 2041.89it/s]
13:26:08 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) done      PlotMarkers(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, input_index_col=morph_name, data_dir=data, skip=True, with_plotly=True)
13:26:08 data_validation_framework.task (I) ==========================================
13:26:08 data_validation_framework.task (I) SUMMARY PlotMarkers: 106 / 106 passed
13:26:08 data_validation_framework.task (I) ==========================================
13:26:08 luigi-interface (I) Informed scheduler that task   PlotMarkers_data_data_preprocesse_morph_name_7d087863ed   has status   DONE
13:26:08 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) running   Curate(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, data_dir=data, report_path=report.pdf, generate_report=True, report_type=pdf, report_config=null, specifications_only=False, input_index_col=morph_name)
13:26:08 data_validation_framework.task (I) Input dataset: /workdir/dvc/data/preprocessed-janelia.csv
/usr/local/lib/python3.8/dist-packages/pandas/core/common.py:241: VisibleDeprecationWarning:

Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.

13:26:08 luigi-interface (I) [pid 17782] Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) done      Curate(tag_output=False, result_path=data/mcar-curation/janelia, dataset_df=data/preprocessed-janelia.csv, data_dir=data, report_path=report.pdf, generate_report=True, report_type=pdf, report_config=null, specifications_only=False, input_index_col=morph_name)
13:26:20 data_validation_framework.report (I) The report was generated here: data/mcar-curation/janelia/report.pdf
13:26:20 data_validation_framework.task (I) ==========================================
13:26:20 data_validation_framework.task (I) SUMMARY Curate: 106 / 106 passed
13:26:20 data_validation_framework.task (I) ==========================================
13:26:20 luigi-interface (I) Informed scheduler that task   Curate_data_data_preprocesse_True_4b6c62e2e9   has status   DONE
13:26:20 luigi-interface (I) Worker Worker(salt=722636757, workers=1, host=4516eecb1d79, username=casalegn, pid=17782) was stopped. Shutting down Keep-Alive thread
13:26:20 luigi-interface (I) 
===== Luigi Execution Summary =====

Scheduled 15 tasks of which:
* 15 ran successfully:
    - 1 Align(...)
    - 1 CheckNeurites(...)
    - 1 Collect(...)
    - 1 Curate(...)
    - 1 DetectErrors(...)
    ...

This progress looks :) because there were no failed tasks or missing dependencies

===== Luigi Execution Summary =====

Issue part 2: feature extraction removes unusable morphologies from dataset

dvc repro -fs features-lida-janelia-L5-diagram-tmd-rd-apical

produces

Running stage 'features-lida-janelia-L5-diagram-tmd-rd-apical':       
> morphoclass -v extract-features data/final/classes-janelia-L5.csv apical diagram-tmd-rd extract-features/lida-janelia-L5/apical/diagram-tmd-rd
13:31:33 morphoclass.console.main (I) Running them morphoclass entrypoint
13:31:33 morphoclass.console.cmd_extract_features (I) Loading modules and libraries
13:31:36 morphoclass.console.cmd_extract_features (I) Starting feature extraction
13:31:36 morphoclass.console.cmd_extract_features (I) Setting up pre-transforms
13:31:36 morphoclass.console.cmd_extract_features (I) Loading data
WARNING: Neuron AA0066 has no apical
WARNING: Neuron AA0415 has no apical
WARNING: Neuron AA0393 has no apical
WARNING: Neuron AA0265 has no apical
WARNING: Neuron AA0397 has no apical
WARNING: Neuron AA0422 has no apical
WARNING: Neuron AA0442 has no apical
WARNING: Neuron AA0332 has no apical
WARNING: Neuron AA0394 has no apical
WARNING: Neuron AA0465 has no apical
WARNING: Neuron AA0466 has no apical
13:31:51 morphoclass.console.cmd_extract_features (E) Some morphologies had neurites with a total neurite node count less than 3. This is too little for feature extraction and we'll therefor remove these morphologies from the dataset. Consider inspecting the data to find the cause. The morphologies to remove are:
* data/final/janelia/L5/TPC_A/AA0066.asc
* data/final/janelia/L5/TPC_C/AA0442.asc
* data/final/janelia/L5/TPC_A/AA0415.asc
* data/final/janelia/L5/UPC/AA0332.asc
* data/final/janelia/L5/UPC/AA0394.asc
* data/final/janelia/L5/UPC/AA0465.asc
* data/final/janelia/L5/TPC_B/AA0393.asc
* data/final/janelia/L5/UPC/AA0466.asc
* data/final/janelia/L5/TPC_C/AA0265.asc
* data/final/janelia/L5/TPC_C/AA0397.asc
* data/final/janelia/L5/TPC_C/AA0422.asc
13:31:51 morphoclass.console.cmd_extract_features (I) Extracting features
13:31:51 morphoclass.console.cmd_extract_features (I) Setting the path attributes
13:31:51 morphoclass.console.cmd_extract_features (I) Saving extracted features to disk
13:31:51 morphoclass.console.cmd_extract_features (I) Done.
FrancescoCasalegno commented 2 years ago

This issue and the bugs that are producing it are discussed in details in this comment: https://github.com/BlueBrain/NeuroR/issues/99#issuecomment-1143499575

Now that the corresponding PRs were created and merged both in NeuroR and morphology-workflows, we should be able to solve our problems by upgrading to the latest releases of these two pacakges.