PaddlePaddle / PaddleDetection

Object Detection toolkit based on PaddlePaddle. It supports object detection, instance segmentation, multiple object tracking and real-time multi-person keypoint detection.
Apache License 2.0
12.65k stars 2.87k forks source link

训练过程无法保存 #3606

Open AN-ZE opened 3 years ago

AN-ZE commented 3 years ago

环境: win10 + cuda10.2 + paddle2.1.1 + paddledet2.1(今天刚clone)

运行 blazeface 使用voc格式数据训练,训练过程如下

In Paddle <= 2.0, data is in format '[Tensor(shape=(1, 2, 3), dtype=float32)]', and in Paddle >= 2.1, data is in format 'Tensor(shape=(1, 2, 3), dtype=float32)'

[07/06 15:18:11] ppdet.utils.checkpoint INFO: Finish loading model weights: C:\Users\ANZE/.cache/paddle/weights\blazenet_pretrain.pdparams
[07/06 15:18:12] ppdet.engine INFO: Epoch: [0] [ 0/87] learning_rate: 0.000333 loss: 19.039179 eta: 4:35:48 batch_cost: 0.1902 data_cost: 0.0000 ips: 42.0581 images/s
[07/06 15:18:55] ppdet.engine INFO: Epoch: [0] [20/87] learning_rate: 0.000360 loss: 11.350872 eta: 2 days, 1:56:48 batch_cost: 2.1611 data_cost: 2.0163 ips: 3.7018 images/s
[07/06 15:19:43] ppdet.engine INFO: Epoch: [0] [40/87] learning_rate: 0.000387 loss: 9.275948 eta: 2 days, 5:44:07 batch_cost: 2.3897 data_cost: 2.2493 ips: 3.3476 images/s
[07/06 15:20:28] ppdet.engine INFO: Epoch: [0] [60/87] learning_rate: 0.000413 loss: 8.554915 eta: 2 days, 5:57:25 batch_cost: 2.2541 data_cost: 2.1143 ips: 3.5491 images/s
[07/06 15:21:19] ppdet.engine INFO: Epoch: [0] [80/87] learning_rate: 0.000440 loss: 7.966538 eta: 2 days, 7:57:21 batch_cost: 2.5716 data_cost: 2.4313 ips: 3.1109 images/s
Segmentation fault
jerrywgz commented 3 years ago

请问下windows环境中是否使用了conda呢,目前在某些windows环境下确实存在这个问题

AN-ZE commented 3 years ago

没有使用conda,python使用3.6

appdirs             1.4.4
astor               0.8.1
atomicwrites        1.4.0
attrs               21.2.0
Babel               2.9.0
bce-python-sdk      0.8.59
cached-property     1.5.2
certifi             2020.12.5
cfgv                3.2.0
chardet             4.0.0
click               7.1.2
colorama            0.4.4
colorlog            5.0.1
cycler              0.10.0
Cython              0.29.23
dataclasses         0.8
decorator           4.4.2
distlib             0.3.1
easydict            1.9
et-xmlfile          1.1.0
filelock            3.0.12
flake8              3.9.0
flake8-import-order 0.18.1
Flask               1.1.2
Flask-Babel         2.0.0
Flask-Cors          3.0.10
future              0.18.2
gast                0.3.3
gitdb               4.0.7
GitPython           3.1.14
h5py                3.1.0
identify            2.2.0
idna                2.10
imageio             2.9.0
importlib-metadata  3.7.3
importlib-resources 5.1.2
iniconfig           1.1.1
itsdangerous        1.1.0
jieba               0.42.1
Jinja2              2.11.3
joblib              1.0.1
Keras               2.4.3
kiwisolver          1.3.1
lap                 0.4.0
Mako                1.1.4
MarkupSafe          1.1.1
matplotlib          3.3.4
mccabe              0.6.1
motmetrics          1.2.0
networkx            2.5
nodeenv             1.5.0
numpy               1.19.3
opencv-python       4.5.1.48
openpyxl            3.0.7
packaging           20.9
paddlehub           2.0.4
paddlenlp           2.0.0rc17
paddlepaddle-gpu    2.1.1
paddleslim          1.1.1
paddlex             1.3.7
pandas              1.1.5
Pillow              8.1.2
pip                 21.0.1
pluggy              0.13.1
pre-commit          2.11.1
protobuf            3.15.6
psutil              5.8.0
py                  1.10.0
py-cpuinfo          8.0.0
pycocotools         2.0.2
pycodestyle         2.7.0
pycryptodome        3.10.1
pycuda              2020.1
pyflakes            2.3.0
pyparsing           2.4.7
pytest              6.2.4
pytest-benchmark    3.4.1
python-dateutil     2.8.1
pytools             2021.2.1
pytz                2021.1
PyWavelets          1.1.1
PyYAML              5.4.1
pyzmq               22.0.3
rarfile             4.0
requests            2.25.1
scikit-image        0.17.2
scikit-learn        0.24.1
scipy               1.5.4
seqeval             0.0.10
setuptools          57.0.0
Shapely             1.7.1
shellcheck-py       0.7.1.1
six                 1.15.0
sklearn             0.0
smmap               4.0.0
terminaltables      3.1.0
threadpoolctl       2.1.0
tifffile            2020.9.3
toml                0.10.2
torch               1.9.0+cu102
torchaudio          0.9.0
torchvision         0.10.0+cu102
tqdm                4.59.0
typeguard           2.11.1
typing-extensions   3.7.4.3
unknown             0.0.0
urllib3             1.26.4
virtualenv          20.4.3
visualdl            2.1.1
Werkzeug            1.0.1
xlwt                1.3.0
xmltodict           0.12.0
zipp                3.4.1