TiagoCortinhal / SalsaNext

Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving
MIT License
417 stars 102 forks source link

Error when training the model #53

Closed Moldoteck closed 3 years ago

Moldoteck commented 3 years ago

When running script with: ./train.sh -d "/mnt/d/side projects/KITTI_DATASET/dataset" -a "./salsanext.yml" -l "./logs" it gives the following error: Illegal instruction

Here is the full log:

/home/moldoteck/miniconda3/envs/salsanext/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or "1type" as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / "(1,)type".
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/moldoteck/miniconda3/envs/salsanext/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or "1type" as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / "(1,)type".
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/moldoteck/miniconda3/envs/salsanext/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or "1type" as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / "(1,)type".
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/moldoteck/miniconda3/envs/salsanext/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or "1type" as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / "(1,)type".
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/moldoteck/miniconda3/envs/salsanext/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or "1type" as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / "(1,)type".
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/moldoteck/miniconda3/envs/salsanext/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or "1type" as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / "(1,)type".
  np_resource = np.dtype([("resource", np.ubyte, 1)])
----------
INTERFACE:
dataset /mnt/d/side projects/KITTI_DATASET/dataset
arch_cfg /home/moldoteck/SalsaNext/salsanext.yml
data_cfg config/labels/semantic-kitti.yaml
uncertainty False
Total of Trainable Parameters: 6.71M
log /home/moldoteck/SalsaNext/logs/logs/2021-3-28-16:10
pretrained
----------

----------

Opening arch config file /home/moldoteck/SalsaNext/salsanext.yml
Opening data config file config/labels/semantic-kitti.yaml
No pretrained directory found.
Copying files to /home/moldoteck/SalsaNext/logs/logs/2021-3-28-16:10 for further reference.
Sequences folder exists! Using sequences from /mnt/d/side projects/KITTI_DATASET/dataset/sequences
parsing seq 00
parsing seq 01
parsing seq 02
parsing seq 03
parsing seq 04
parsing seq 05
parsing seq 06
parsing seq 07
parsing seq 09
parsing seq 10
Using 19130 scans from sequences [0, 1, 2, 3, 4, 5, 6, 7, 9, 10]
Sequences folder exists! Using sequences from /mnt/d/side projects/KITTI_DATASET/dataset/sequences
parsing seq 08
Using 4071 scans from sequences [8]
Loss weights from content:  tensor([  0.0000,  22.9317, 857.5627, 715.1100, 315.9618, 356.2452, 747.6170,
        887.2239, 963.8915,   5.0051,  63.6247,   6.9002, 203.8796,   7.4802,
         13.6315,   3.7339, 142.1462,  12.6355, 259.3699, 618.9667])
Training in device:  cpu
Ignoring class  0  in IoU evaluation
[IOU EVAL] IGNORE:  tensor([0])
[IOU EVAL] INCLUDE:  tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
        19])
Illegal instruction
Moldoteck commented 3 years ago

Seems that this is a pytorch error. Using latest version for packages solved the error