search

mist-medical / MIST

MIST: A simple, scalable, and end-to-end framework for 3D medical imaging segmentation.
Apache License 2.0
29 stars 11 forks source link

mist_train failing on last fold: IndexError: single positional indexer is out-of-bounds #27

Closed fuentesdt closed 2 months ago

fuentesdt commented 3 months ago

kirby$ pwd /rsrch3/ip/dtfuentes/github/oncopigdata kirby$ /rsrch1/ip/dtfuentes/minicondamist/bin/mist_train --data /rsrch3/ip/dtfuentes/github/oncopigdata/misttrain/dataset.json --numpy /rsrch3/ip/dtfuentes/github/oncopigdata/misttrainnumpy --results /rsrch3/ip/dtfuentes/github/oncopigdata/misttrainresults --gpus 1 --amp --pocket --epochs 100 . . . .

Fold 4: Epoch 98/100 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8/8 • 0:00:06 • loss: 0.1460 Validating ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/2 • 0:00:10 • val_loss: 0.2040 Validation loss did NOT improve from 0.1937

Fold 4: Epoch 99/100 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8/8 • 0:00:06 • loss: 0.1346 Validating ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/2 • 0:00:10 • val_loss: 0.1886 Validation loss IMPROVED from 0.1937 to 0.1886

Fold 4: Epoch 100/100 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8/8 • 0:00:06 • loss: 0.1422 Validating ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/2 • 0:00:10 • val_loss: 0.2077 Validation loss did NOT improve from 0.1886

Testing on fold ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4/4 • 0:00:55

Evaluating ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/36 • 0:00:16 Traceback (most recent call last): File "/rsrch1/ip/dtfuentes/minicondamist/bin/mist_train", line 33, in sys.exit(load_entry_point('mist-medical', 'console_scripts', 'mist_train')()) File "/rsrch1/ip/dtfuentes/github/MIST/mist/scripts/train_entrypoint.py", line 28, in train_entry main(args) File "/rsrch1/ip/dtfuentes/github/MIST/mist/main.py", line 62, in main evaluate(args.data, File "/rsrch1/ip/dtfuentes/github/MIST/mist/evaluate_preds/evaluate.py", line 135, in evaluate truth = paths.loc[paths['id'].astype(str) == patient_id].iloc[0]['mask'] File "/rsrch1/ip/dtfuentes/minicondamist/lib/python3.8/site-packages/pandas/core/indexing.py", line 1103, in getitem return self._getitem_axis(maybe_callable, axis=axis) File "/rsrch1/ip/dtfuentes/minicondamist/lib/python3.8/site-packages/pandas/core/indexing.py", line 1656, in _getitem_axis self._validate_integer(key, axis) File "/rsrch1/ip/dtfuentes/minicondamist/lib/python3.8/site-packages/pandas/core/indexing.py", line 1589, in _validate_integer raise IndexError("single positional indexer is out-of-bounds") IndexError: single positional indexer is out-of-bounds

aecelaya commented 3 months ago

Looks like you're using an earlier version. Try upgrading to v0.4.15. It has more error handling for the evaluation and prediction.

Also, try running ls -a in your predictions folder. The error is telling you that it can't find a certain file. Usually, that means that there might be a hidden file that made its way into the predictions folder, like a .DS_Store of .ipynb_checkpoints. Either way, the latest version can handle it and will let you know which files fail with prediction or evaluation.