search

PeterJackNaylor / NN-4-change-detection

MIT License
4 stars 1 forks source link

A train error : multi_class must be in ('ovo', 'ovr') #1

Open flww213 opened 4 months ago

flww213 commented 4 months ago

Thank you for your great work!

I encountered a bug while computing AUC using the "process_diff.py" script.

It seems that the "roc_auc_score()" function should have the parameter "multi_class='ovo'" or "multi_class='ovr'" in order to work correctly. However, this doesn't resolve the issue.

Here is the debug information:

executor >  local (1)
[dd/d6be03] process > prepare_data:from_ply_to_txt (1)       [100%] 1 of 1, cached: 1 ✔
[64/e7f703] process > one_density:one_density_estimation (1) [100%] 1 of 1, cached: 1 ✔
[3c/c2f509] process > one_density:post_processing (1)        [100%] 1 of 1, failed: 1 ✘
[-        ] process > two_density:two_density_estimation     -
[-        ] process > two_density:post_processing            -
ERROR ~ Error executing process > 'one_density:post_processing (1)'

Caused by:
  Process `one_density:post_processing (1)` terminated with an error exit status (1)

Command executed:

  python process_diff.py single_M+TD+TVN MultiSensor_LyonS4_0__FEATUREMETHOD=--fourier__METHOD=M+TD+TVN.pth MultiSensor_LyonS4_0.txt MultiSensor_LyonS4_1.txt MultiSensor_LyonS4_0__FEATUREMETHOD=--fourier__METHOD=M+TD+TVN.npz home.yaml

Command exit status:
  1

Command output:
  [[0.00628275]
   [0.0041347 ]
   [0.00829171]]
  []

Command error:
   26%|██▌       | 12/46 [00:06<00:19,  1.77it/s]
   28%|██▊       | 13/46 [00:07<00:18,  1.79it/s]
   30%|███       | 14/46 [00:08<00:17,  1.80it/s]
   33%|███▎      | 15/46 [00:08<00:17,  1.79it/s]
   35%|███▍      | 16/46 [00:09<00:16,  1.81it/s]
   37%|███▋      | 17/46 [00:09<00:15,  1.83it/s]
   39%|███▉      | 18/46 [00:10<00:15,  1.84it/s]
   41%|████▏     | 19/46 [00:10<00:14,  1.83it/s]
   43%|████▎     | 20/46 [00:11<00:14,  1.85it/s]
   46%|████▌     | 21/46 [00:11<00:13,  1.87it/s]
   48%|████▊     | 22/46 [00:12<00:12,  1.88it/s]
   50%|█████     | 23/46 [00:12<00:12,  1.86it/s]
   52%|█████▏    | 24/46 [00:13<00:11,  1.88it/s]
   54%|█████▍    | 25/46 [00:13<00:11,  1.90it/s]
   57%|█████▋    | 26/46 [00:14<00:10,  1.91it/s]
   59%|█████▊    | 27/46 [00:14<00:10,  1.89it/s]
   61%|██████    | 28/46 [00:15<00:09,  1.91it/s]
   63%|██████▎   | 29/46 [00:16<00:08,  1.91it/s]
   65%|██████▌   | 30/46 [00:16<00:08,  1.92it/s]
   67%|██████▋   | 31/46 [00:17<00:07,  1.90it/s]
   70%|██████▉   | 32/46 [00:17<00:07,  1.91it/s]
   72%|███████▏  | 33/46 [00:18<00:06,  1.92it/s]
   74%|███████▍  | 34/46 [00:18<00:06,  1.93it/s]
   76%|███████▌  | 35/46 [00:19<00:05,  1.91it/s]
   78%|███████▊  | 36/46 [00:19<00:05,  1.92it/s]
   80%|████████  | 37/46 [00:20<00:04,  1.92it/s]
   83%|████████▎ | 38/46 [00:20<00:04,  1.93it/s]
   85%|████████▍ | 39/46 [00:21<00:03,  1.91it/s]
   87%|████████▋ | 40/46 [00:21<00:03,  1.92it/s]
   89%|████████▉ | 41/46 [00:22<00:02,  1.92it/s]
   91%|█████████▏| 42/46 [00:22<00:02,  1.93it/s]
   93%|█████████▎| 43/46 [00:23<00:01,  1.91it/s]
   96%|█████████▌| 44/46 [00:23<00:01,  1.92it/s]
   98%|█████████▊| 45/46 [00:24<00:00,  1.93it/s]
  100%|██████████| 46/46 [00:24<00:00,  1.94it/s]
  100%|██████████| 46/46 [00:24<00:00,  1.85it/s]
  [[0.00628275]
   [0.0041347 ]
   [0.00829171]]
  []
Traceback (most recent call last):
    File "process_diff.py", line 59, in <module>
      auc_score = compute_auc_mc(diff_z_on1, y_on1)
    File "/home/hcf/NN-4-change-detection/NN-4-change-detection/python/src/utils.py", line 122, in compute_auc_mc
      scores[value] = roc_auc_score(y_tmp, diff_tmp)
    File "/home/hcf/anaconda3/envs/nn4cd/lib/python3.10/site-packages/sklearn/utils/_param_validation.py", line 213, in wrapper
      return func(*args, **kwargs)
    File "/home/hcf/anaconda3/envs/nn4cd/lib/python3.10/site-packages/sklearn/metrics/_ranking.py", line 633, in roc_auc_score
      raise ValueError("multi_class must be in ('ovo', 'ovr')")
  ValueError: multi_class must be in ('ovo', 'ovr')

After calling the "roc_auc_score(y_tmp, diff_tmp, multi_class='ovo')" function, a new error occurred. And multi_class='ovr' have the same error.

  Traceback (most recent call last):
    File "process_diff.py", line 59, in <module>
      auc_score = compute_auc_mc(diff_z_on1, y_on1)
    File "/home/hcf/NN-4-change-detection/NN-4-change-detection/python/src/utils.py", line 122, in compute_auc_mc
      scores[value] = roc_auc_score(y_tmp, diff_tmp,multi_class='ovo')
    File "/home/hcf/anaconda3/envs/nn4cd/lib/python3.10/site-packages/sklearn/utils/_param_validation.py", line 213, in wrapper
      return func(*args, **kwargs)
    File "/home/hcf/anaconda3/envs/nn4cd/lib/python3.10/site-packages/sklearn/metrics/_ranking.py", line 634, in roc_auc_score
      return _multiclass_roc_auc_score(
    File "/home/hcf/anaconda3/envs/nn4cd/lib/python3.10/site-packages/sklearn/metrics/_ranking.py", line 706, in _multiclass_roc_auc_score
      if not np.allclose(1, y_score.sum(axis=1)):
    File "/home/hcf/anaconda3/envs/nn4cd/lib/python3.10/site-packages/numpy/core/_methods.py", line 49, in _sum
      return umr_sum(a, axis, dtype, out, keepdims, initial, where)
  numpy.exceptions.AxisError: axis 1 is out of bounds for array of dimension 1

Could you please advise on how to fix this problem?

PeterJackNaylor commented 4 months ago

Hi!

Thank you for your interest in our work and for using the code!

The problem you are encountering, I believe is due to a bad training that leads to something unexpected in the predicted y-labels. When we compute the auc, we do a binary auc (but there are three class?!), indeed, we map the predicted outputs to the class of interest (one of the three) vs the two others before applying the auc.

To maybe have a better idea of what is happening, I would suggest:

Another indication of the bad training is the printed list that you reported, that print shows the threshold values to map the altitude difference in z to the categorical labels (which is written in the std output that you copied, I am referring to this list: [[0.00628275] [0.0041347 ] [0.00829171]]) do not seem very good, and at this stage in the pipeline the program is expecting usually 2 values, corresponding the threshold between "Addition" and "No change" and "Deletion" and "No change", so more reasonable values for this list should be similar to [ [-4], [4] ] or something like that.

Hopefully this should help you, don't hesitate to come back if you need more guidance or advice to get the code properly working!