AIstudio多卡训练导出的log文件无法可视化

把保存的vdl文件下载后通过BML可视化查看数据，可视化显示无法查看。通过BML平台训练的log文件可以正常查看

保存VDL的命令如下

os.system("cd PaddleDetection-release-2.3 && python -m paddle.distributed.launch  tools/train.py -c configs/picodet/picodet_l_640_coco.yml --eval --use_vdl=True --vdl_log_dir='./output_vdl'")

os.system("mv PaddleDetection-release-2.3/output_vdl /root/paddlejob/workspace/output")# 数据集文件目录

保存的Log文件摘要如下

[INFO]: current net device: eth0, ip: 172.28.2.33
[INFO]: paddle job envs:
POD_IP=job-5b4b88d2ffeff1c9002966330390cf75-trainer-0.job-5b4b88d2ffeff1c9002966330390cf75
PADDLE_PORT=12345
PADDLE_TRAINER_ID=0
PADDLE_TRAINERS_NUM=1
PADDLE_USE_CUDA=1
NCCL_SOCKET_IFNAME=eth0
PADDLE_IS_LOCAL=1
OUTPUT_PATH=/root/paddlejob/workspace/output
LOCAL_LOG_PATH=/root/paddlejob/workspace/log
LOCAL_MOUNT_PATH=/mnt/code_20211108133208,/mnt/datasets_20211108133209
JOB_ID=job-5b4b88d2ffeff1c9002966330390cf75
TRAINING_ROLE=TRAINER
[INFO]: user command: python run.py
[INFO]: start trainer
~/paddlejob/workspace/code /mnt
Looking in indexes: https://mirror.baidu.com/pypi/simple, https://pypi.tuna.tsinghua.edu.cn/simple
Requirement already satisfied: tqdm in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from -r requirements.txt (line 1)) (4.36.1)
Collecting typeguard
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/55/55/5438ba6879e7ea539108027cffff5e0933cc395c60ed3ae8eb3f87ebcad6/typeguard-2.13.0-py3-none-any.whl (17 kB)
Collecting visualdl>=2.1.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a0/6d/3598b4adcc75dbb957d36ceda08412ed5646b1a5af73a0fb59ac9eeed3dc/visualdl-2.2.1-py3-none-any.whl (2.7 MB)
Requirement already satisfied: opencv-python in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (4.1.1.26)
Requirement already satisfied: PyYAML in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (5.1)
Collecting shapely
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ae/20/33ce377bd24d122a4d54e22ae2c445b9b1be8240edb50040b40add950cd9/Shapely-1.8.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.1 MB)
Requirement already satisfied: scipy in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (1.1.0)
Collecting terminaltables
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9b/c4/4a21174f32f8a7e1104798c445dacdc1d4df86f2f26722767034e4de4bff/terminaltables-3.1.0.tar.gz (12 kB)
Requirement already satisfied: Cython in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (0.29)
Collecting pycocotools
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/de/df/056875d697c45182ed6d2ae21f62015896fdb841906fe48e7268e791c467/pycocotools-2.0.2.tar.gz (23 kB)
Requirement already satisfied: setuptools>=42.0.0 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (49.2.0)
Collecting lap
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/bf/64/d9fb6a75b15e783952b2fec6970f033462e67db32dc43dfbb404c14e91c2/lap-0.4.0.tar.gz (1.5 MB)
Requirement already satisfied: sklearn in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (0.0)
Collecting motmetrics
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9c/28/9c3bc8e2a87f4c9e7b04ab72856ec7f9895a66681a65973ffaf9562ef879/motmetrics-1.2.0-py3-none-any.whl (151 kB)
Requirement already satisfied: openpyxl in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from -r requirements.txt (line 16)) (3.0.7)
Collecting cython_bbox
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/fa/b9/fc7d60e8c3b29cc0ff24a3bb3c4b7457e10b7610fbb2893741b623487b34/cython_bbox-0.1.3.tar.gz (41 kB)
Requirement already satisfied: requests in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from visualdl>=2.1.0->-r requirements.txt (line 3)) (2.22.0)
Requirement already satisfied: matplotlib in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from visualdl>=2.1.0->-r requirements.txt (line 3)) (2.2.2)
Collecting shellcheck-py
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/77/c9/6f84db444de69e1bdbfd168ae6f3d5af72107c16cd9db3c0a1f8e41204d1/shellcheck_py-0.7.2.1-py2.py3-none-manylinux1_x86_64.whl (2.0 MB)
Collecting bce-python-sdk
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/3f/3e/d7a44d8f43aba0de52d56600fce8ab9ce107fe074dffc36515402bf8983c/bce_python_sdk-0.8.62-py3-none-any.whl (197 kB)
Requirement already satisfied: six>=1.14.0 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from visualdl>=2.1.0->-r requirements.txt (line 3)) (1.16.0)
Requirement already satisfied: numpy in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from visualdl>=2.1.0->-r requirements.txt (line 3)) (1.19.5)
Requirement already satisfied: Flask-Babel>=1.0.0 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from visualdl>=2.1.0->-r requirements.txt (line 3)) (1.0.0)
Collecting Pillow>=7.0.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a7/f3/c81e9c41c6ee4c99ad7436b7a1509f5bce5de4ccf4b83138cf88760d96ff/Pillow-8.4.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
Requirement already satisfied: pre-commit in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from visualdl>=2.1.0->-r requirements.txt (line 3)) (1.21.0)
Requirement already satisfied: flake8>=3.7.9 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from visualdl>=2.1.0->-r requirements.txt (line 3)) (3.7.9)
Requirement already satisfied: pandas in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from visualdl>=2.1.0->-r requirements.txt (line 3)) (1.1.5)
Collecting protobuf>=3.11.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/4a/81/1722e657aea5abf8ed5da6f9ff1c5e4350f64be10de0a4ca6800fa30f06d/protobuf-3.19.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Requirement already satisfied: flask>=1.1.1 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from visualdl>=2.1.0->-r requirements.txt (line 3)) (1.1.1)
Requirement already satisfied: scikit-learn in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from sklearn->-r requirements.txt (line 14)) (0.24.2)
Collecting flake8-import-order
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ab/52/cf2d6e2c505644ca06de2f6f3546f1e4f2b7be34246c9e0757c6048868f9/flake8_import_order-0.18.1-py2.py3-none-any.whl (15 kB)
Collecting pytest-benchmark
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/2c/60/423a63fb190a0483d049786a121bd3dfd7d93bb5ff1bb5b5cd13e5df99a7/pytest_benchmark-3.4.1-py2.py3-none-any.whl (50 kB)
Collecting pytest
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/40/76/86f886e750b81a4357b6ed606b2bcf0ce6d6c27ad3c09ebf63ed674fc86e/pytest-6.2.5-py3-none-any.whl (280 kB)
Collecting xmltodict>=0.12.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/28/fd/30d5c1d3ac29ce229f6bdc40bbc20b28f716e8b363140c26eff19122d8a5/xmltodict-0.12.0-py2.py3-none-any.whl (9.2 kB)
Requirement already satisfied: et-xmlfile in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from openpyxl->-r requirements.txt (line 16)) (1.1.0)
Requirement already satisfied: pycodestyle<2.6.0,>=2.5.0 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl>=2.1.0->-r requirements.txt (line 3)) (2.5.0)
Requirement already satisfied: pyflakes<2.2.0,>=2.1.0 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl>=2.1.0->-r requirements.txt (line 3)) (2.1.1)
Requirement already satisfied: mccabe<0.7.0,>=0.6.0 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl>=2.1.0->-r requirements.txt (line 3)) (0.6.1)
Requirement already satisfied: entrypoints<0.4.0,>=0.3.0 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl>=2.1.0->-r requirements.txt (line 3)) (0.3)
Requirement already satisfied: itsdangerous>=0.24 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from flask>=1.1.1->visualdl>=2.1.0->-r requirements.txt (line 3)) (1.1.0)
Requirement already satisfied: click>=5.1 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from flask>=1.1.1->visualdl>=2.1.0->-r requirements.txt (line 3)) (7.0)
Requirement already satisfied: Werkzeug>=0.15 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from flask>=1.1.1->visualdl>=2.1.0->-r requirements.txt (line 3)) (0.16.0)
Requirement already satisfied: Jinja2>=2.10.1 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from flask>=1.1.1->visualdl>=2.1.0->-r requirements.txt (line 3)) (2.10.3)
Requirement already satisfied: Babel>=2.3 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from Flask-Babel>=1.0.0->visualdl>=2.1.0->-r requirements.txt (line 3)) (2.8.0)
Requirement already satisfied: pytz in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from Flask-Babel>=1.0.0->visualdl>=2.1.0->-r requirements.txt (line 3)) (2018.5)
Requirement already satisfied: MarkupSafe>=0.23 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from Jinja2>=2.10.1->flask>=1.1.1->visualdl>=2.1.0->-r requirements.txt (line 3)) (1.1.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from matplotlib->visualdl>=2.1.0->-r requirements.txt (line 3)) (1.0.1)
Requirement already satisfied: python-dateutil>=2.1 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from matplotlib->visualdl>=2.1.0->-r requirements.txt (line 3)) (2.7.3)
Requirement already satisfied: cycler>=0.10 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from matplotlib->visualdl>=2.1.0->-r requirements.txt (line 3)) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from matplotlib->visualdl>=2.1.0->-r requirements.txt (line 3)) (2.2.0)
Collecting pycryptodome>=3.8.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/b4/7f/4fdf605c688d4afa1264b37fd57edce87313f228facb3d8c869f65b5358e/pycryptodome-3.11.0-cp35-abi3-manylinux2010_x86_64.whl (1.9 MB)
Requirement already satisfied: future>=0.6.0 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from bce-python-sdk->visualdl>=2.1.0->-r requirements.txt (line 3)) (0.18.0)
Requirement already satisfied: toml in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from pre-commit->visualdl>=2.1.0->-r requirements.txt (line 3)) (0.10.0)
Requirement already satisfied: importlib-metadata in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from pre-commit->visualdl>=2.1.0->-r requirements.txt (line 3)) (0.23)
Requirement already satisfied: aspy.yaml in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from pre-commit->visualdl>=2.1.0->-r requirements.txt (line 3)) (1.3.0)
Requirement already satisfied: cfgv>=2.0.0 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from pre-commit->visualdl>=2.1.0->-r requirements.txt (line 3)) (2.0.1)
Requirement already satisfied: virtualenv>=15.2 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from pre-commit->visualdl>=2.1.0->-r requirements.txt (line 3)) (16.7.9)
Requirement already satisfied: identify>=1.0.0 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from pre-commit->visualdl>=2.1.0->-r requirements.txt (line 3)) (1.4.10)
Requirement already satisfied: nodeenv>=0.11.1 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from pre-commit->visualdl>=2.1.0->-r requirements.txt (line 3)) (1.3.4)
Requirement already satisfied: zipp>=0.5 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from importlib-metadata->pre-commit->visualdl>=2.1.0->-r requirements.txt (line 3)) (0.6.0)
Requirement already satisfied: more-itertools in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from zipp>=0.5->importlib-metadata->pre-commit->visualdl>=2.1.0->-r requirements.txt (line 3)) (7.2.0)
Collecting pluggy<2.0,>=0.12
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9e/01/f38e2ff29715251cf25532b9082a1589ab7e4f571ced434f98d0139336dc/pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting py>=1.8.2
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f6/f0/10642828a8dfb741e5f3fbaac830550a518a775c7fff6f04a007259b0548/py-1.11.0-py2.py3-none-any.whl (98 kB)
Collecting packaging
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/b1/09/464d5df9f9ec1ab5054af6d097df6793e542f4aa426ba3062ec64409cab7/packaging-21.2-py3-none-any.whl (40 kB)
Collecting iniconfig
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9b/dd/b3c12c6d707058fa947864b67f0c4e0c39ef8610988d7baea9578f3c48f3/iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Requirement already satisfied: attrs>=19.2.0 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from pytest->motmetrics->-r requirements.txt (line 15)) (19.2.0)
Collecting py-cpuinfo
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e6/ba/77120e44cbe9719152415b97d5bfb29f4053ee987d6cb63f55ce7d50fadc/py-cpuinfo-8.0.0.tar.gz (99 kB)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from requests->visualdl>=2.1.0->-r requirements.txt (line 3)) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from requests->visualdl>=2.1.0->-r requirements.txt (line 3)) (2019.9.11)
Requirement already satisfied: idna<2.9,>=2.5 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from requests->visualdl>=2.1.0->-r requirements.txt (line 3)) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from requests->visualdl>=2.1.0->-r requirements.txt (line 3)) (1.25.6)
Requirement already satisfied: joblib>=0.11 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from scikit-learn->sklearn->-r requirements.txt (line 14)) (0.14.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from scikit-learn->sklearn->-r requirements.txt (line 14)) (2.2.0)
Building wheels for collected packages: terminaltables, pycocotools, lap, cython-bbox, py-cpuinfo
  Building wheel for terminaltables (setup.py): started
  Building wheel for terminaltables (setup.py): finished with status 'done'
  Created wheel for terminaltables: filename=terminaltables-3.1.0-py3-none-any.whl size=15354 sha256=e7e95e2ffd5be1bf1baebea848290ed25eef023a79134f0d4825b66222afa2d9
  Stored in directory: /root/.cache/pip/wheels/1a/45/20/6dd7cbd828c4837b089f050af9f656aa194bf8bb312629d722
  Building wheel for pycocotools (setup.py): started
  Building wheel for pycocotools (setup.py): finished with status 'done'
  Created wheel for pycocotools: filename=pycocotools-2.0.2-cp37-cp37m-linux_x86_64.whl size=305918 sha256=622c8843d73c55034c98c88ccd473620f300b1d49cc4afc5a28b14b88d620dc6
  Stored in directory: /root/.cache/pip/wheels/f0/4e/80/5c67caf3bcf78ccd860093035add754e4d05e2a54b06dd2cb9
  Building wheel for lap (setup.py): started
  Building wheel for lap (setup.py): finished with status 'done'
  Created wheel for lap: filename=lap-0.4.0-cp37-cp37m-linux_x86_64.whl size=1597397 sha256=fe957e52994c8f413428da785e4c3a1d0e157892e07e6073deb583b04f611c65
  Stored in directory: /root/.cache/pip/wheels/5c/d0/d2/e331d17a999666b1e2eb99743cfa1742629f9d26c55c657001
  Building wheel for cython-bbox (setup.py): started
  Building wheel for cython-bbox (setup.py): finished with status 'done'
  Created wheel for cython-bbox: filename=cython_bbox-0.1.3-cp37-cp37m-linux_x86_64.whl size=58324 sha256=c67f32bf33ec2a09b0d68190f72071a632f7665cbb55efd0cbdbfd926dae68f1
  Stored in directory: /root/.cache/pip/wheels/3e/b3/6a/aae8832326545645e65d643a2aaf223ffa3a7d01e1a1bae01b
  Building wheel for py-cpuinfo (setup.py): started
  Building wheel for py-cpuinfo (setup.py): finished with status 'done'
  Created wheel for py-cpuinfo: filename=py_cpuinfo-8.0.0-py3-none-any.whl size=22244 sha256=498267fe7704b487b12f3aca8770c551213c07917f40afce4b400d154acff59e
  Stored in directory: /root/.cache/pip/wheels/88/c7/d0/6309c7cc9929894c11fe8e516c3e2a0d0a53ee4e198eac48b7
Successfully built terminaltables pycocotools lap cython-bbox py-cpuinfo
Installing collected packages: py, pluggy, packaging, iniconfig, pytest, pycryptodome, py-cpuinfo, xmltodict, shellcheck-py, pytest-benchmark, protobuf, Pillow, flake8-import-order, bce-python-sdk, visualdl, typeguard, terminaltables, shapely, pycocotools, motmetrics, lap, cython-bbox
  Attempting uninstall: protobuf
    Found existing installation: protobuf 3.10.0
    Uninstalling protobuf-3.10.0:
      Successfully uninstalled protobuf-3.10.0
  Attempting uninstall: Pillow
    Found existing installation: Pillow 5.2.0
    Uninstalling Pillow-5.2.0:
      Successfully uninstalled Pillow-5.2.0
  Attempting uninstall: visualdl
    Found existing installation: visualdl 2.0.0b4
    Uninstalling visualdl-2.0.0b4:
      Successfully uninstalled visualdl-2.0.0b4
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
parl 1.3.2 requires cloudpickle==1.2.1, but you have cloudpickle 0.5.3 which is incompatible.
parl 1.3.2 requires pyarrow==0.13.0, but you have pyarrow 5.0.0 which is incompatible.
Successfully installed Pillow-8.4.0 bce-python-sdk-0.8.62 cython-bbox-0.1.3 flake8-import-order-0.18.1 iniconfig-1.1.1 lap-0.4.0 motmetrics-1.2.0 packaging-21.2 pluggy-1.0.0 protobuf-3.19.1 py-1.11.0 py-cpuinfo-8.0.0 pycocotools-2.0.2 pycryptodome-3.11.0 pytest-6.2.5 pytest-benchmark-3.4.1 shapely-1.8.0 shellcheck-py-0.7.2.1 terminaltables-3.1.0 typeguard-2.13.0 visualdl-2.2.1 xmltodict-0.12.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/opt/_internal/cpython-3.7.0/bin/python -m pip install --upgrade pip' command.
Looking in indexes: https://mirror.baidu.com/pypi/simple, https://pypi.tuna.tsinghua.edu.cn/simple
Collecting paddleslim
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ee/e7/c6b97eb6809d14634ae5cbf287285584045d6f8949d0b436dc64cbefbf7a/paddleslim-2.1.1-py3-none-any.whl (288 kB)
Requirement already satisfied: tqdm in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from paddleslim) (4.36.1)
Requirement already satisfied: matplotlib in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from paddleslim) (2.2.2)
Requirement already satisfied: pillow in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from paddleslim) (8.4.0)
Requirement already satisfied: opencv-python in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from paddleslim) (4.1.1.26)
Requirement already satisfied: pyzmq in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from paddleslim) (18.0.1)
Requirement already satisfied: six>=1.10 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from matplotlib->paddleslim) (1.16.0)
Requirement already satisfied: cycler>=0.10 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from matplotlib->paddleslim) (0.10.0)
Requirement already satisfied: python-dateutil>=2.1 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from matplotlib->paddleslim) (2.7.3)
Requirement already satisfied: numpy>=1.7.1 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from matplotlib->paddleslim) (1.19.5)
Requirement already satisfied: pytz in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from matplotlib->paddleslim) (2018.5)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from matplotlib->paddleslim) (1.0.1)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from matplotlib->paddleslim) (2.2.0)
Requirement already satisfied: setuptools in /opt/_internal/cpython-3.7.0/lib/python3.7/site-packages (from kiwisolver>=1.0.1->matplotlib->paddleslim) (49.2.0)
Installing collected packages: paddleslim
Successfully installed paddleslim-2.1.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/opt/_internal/cpython-3.7.0/bin/python -m pip install --upgrade pip' command.
WARNING 2021-11-08 13:32:51,778 launch.py:359] Not found distinct arguments and compiled with cuda or xpu. Default use collective mode
INFO 2021-11-08 13:32:51,781 launch_utils.py:510] Local start 4 processes. First process distributed environment info (Only For Debug): 
    +=======================================================================================+
    |                        Distributed Envs                      Value                    |
    +---------------------------------------------------------------------------------------+
    |                       PADDLE_TRAINER_ID                        0                      |
    |                 PADDLE_CURRENT_ENDPOINT                 127.0.0.1:50678               |
    |                     PADDLE_TRAINERS_NUM                        4                      |
    |                PADDLE_TRAINER_ENDPOINTS  ... 0.1:58033,127.0.0.1:56430,127.0.0.1:47799|
    |                     PADDLE_RANK_IN_NODE                        0                      |
    |                 PADDLE_LOCAL_DEVICE_IDS                        0                      |
    |                 PADDLE_WORLD_DEVICE_IDS                     0,1,2,3                   |
    |                     FLAGS_selected_gpus                        0                      |
    |             FLAGS_selected_accelerators                        0                      |
    +=======================================================================================+

INFO 2021-11-08 13:32:51,781 launch_utils.py:514] details abouts PADDLE_TRAINER_ENDPOINTS can be found in log/endpoints.log, and detail running logs maybe found in log/workerlog.0
-----------  Configuration Arguments -----------
gpus: None
heter_worker_num: None
heter_workers: 
http_port: None
ips: 127.0.0.1
log_dir: log
nproc_per_node: None
run_mode: None
server_num: None
servers: 
training_script: tools/train.py
training_script_args: ['-c', 'configs/picodet/picodet_l_640_coco.yml', '--eval', '--use_vdl=True', '--vdl_log_dir=./output_vdl']
worker_num: None
workers: 
------------------------------------------------
launch train in GPU mode!
launch proc_id:596 idx:0
launch proc_id:602 idx:1
launch proc_id:605 idx:2
launch proc_id:608 idx:3
/opt/_internal/cpython-3.7.0/lib/python3.7/site-packages/matplotlib/__init__.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import MutableMapping
/opt/_internal/cpython-3.7.0/lib/python3.7/site-packages/matplotlib/rcsetup.py:20: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Iterable, Mapping
/opt/_internal/cpython-3.7.0/lib/python3.7/site-packages/matplotlib/colors.py:53: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Sized
Warning: import ppdet from source directory without installing, run 'python setup.py install' to install ppdet firstly
W1108 13:32:58.107767   596 gen_comm_id_helper.cc:120] connect addr=127.0.0.1:58033 failed 1 times with reason: Connection refused retry after 0.5 seconds
I1108 13:32:58.608182   596 nccl_context.cc:74] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0
W1108 13:32:59.557065   596 device_context.cc:404] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1
W1108 13:32:59.562562   596 device_context.cc:422] device: 0, cuDNN Version: 7.6.
[32m[11-08 13:33:12 MainThread @logger.py:224][0m Argv: tools/train.py -c configs/picodet/picodet_l_640_coco.yml --eval --use_vdl=True --vdl_log_dir=./output_vdl
/opt/_internal/cpython-3.7.0/lib/python3.7/site-packages/cloudpickle/cloudpickle.py:47: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
/opt/_internal/cpython-3.7.0/lib/python3.7/site-packages/parl/utils/communication.py:35: FutureWarning: 'pyarrow.default_serialization_context' is deprecated as of 2.0.0 and will be removed in a future version. Use pickle or the pyarrow IPC functionality instead.
  context = pyarrow.default_serialization_context()
/opt/_internal/cpython-3.7.0/lib/python3.7/site-packages/paddleslim/common/analyze_helper.py:22: UserWarning: 
This call to matplotlib.use() has no effect because the backend has already
been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot,
or matplotlib.backends is imported for the first time.

The backend was *originally* set to 'TkAgg' by the following code:
  File "tools/train.py", line 32, in <module>
    from ppdet.core.workspace import load_config, merge_config
  File "/root/paddlejob/workspace/code/PaddleDetection-release-2.3/ppdet/__init__.py", line 15, in <module>
    from . import (core, data, engine, modeling, model_zoo, optimizer, metrics,
  File "/root/paddlejob/workspace/code/PaddleDetection-release-2.3/ppdet/data/__init__.py", line 15, in <module>
    from . import source
  File "/root/paddlejob/workspace/code/PaddleDetection-release-2.3/ppdet/data/source/__init__.py", line 19, in <module>
    from . import keypoint_coco
  File "/root/paddlejob/workspace/code/PaddleDetection-release-2.3/ppdet/data/source/keypoint_coco.py", line 21, in <module>
    from pycocotools.coco import COCO
  File "/opt/_internal/cpython-3.7.0/lib/python3.7/site-packages/pycocotools/coco.py", line 49, in <module>
    import matplotlib.pyplot as plt
  File "/opt/_internal/cpython-3.7.0/lib/python3.7/site-packages/matplotlib/pyplot.py", line 71, in <module>
    from matplotlib.backends import pylab_setup
  File "/opt/_internal/cpython-3.7.0/lib/python3.7/site-packages/matplotlib/backends/__init__.py", line 16, in <module>
    line for line in traceback.format_stack()

  matplotlib.use('Agg')
[11/08 13:33:14] ppdet.engine INFO:  Model FLOPs : 5.633368G. (image shape is [1, 3, 704, 704])
/opt/_internal/cpython-3.7.0/lib/python3.7/site-packages/paddle/nn/layer/norm.py:641: UserWarning: When training, we now always track global mean and variance.
  "When training, we now always track global mean and variance.")
[11/08 13:33:16] ppdet.engine INFO: Epoch: [0] [  0/361] learning_rate: 0.000000 loss_vfl: 1.190482 loss_bbox: 0.461772 loss_dfl: 0.213259 loss: 1.865513 eta: 17:36:52 batch_cost: 1.7566 data_cost: 0.0004 ips: 4.5543 images/s
[11/08 13:33:17] ppdet.engine INFO: Epoch: [0] [  1/361] learning_rate: 0.000010 loss_vfl: 1.029632 loss_bbox: 0.433910 loss_dfl: 0.198828 loss: 1.662370 eta: 11:30:46 batch_cost: 0.5397 data_cost: 0.0003 ips: 14.8232 images/s
[11/08 13:33:18] ppdet.engine INFO: Epoch: [0] [  2/361] learning_rate: 0.000020 loss_vfl: 1.299591 loss_bbox: 0.398723 loss_dfl: 0.196930 loss: 1.895245 eta: 10:03:52 batch_cost: 0.7149 data_cost: 0.0004 ips: 11.1900 images/s
[11/08 13:33:18] ppdet.engine INFO: Epoch: [0] [  3/361] learning_rate: 0.000030 loss_vfl: 1.023783 loss_bbox: 0.659205 loss_dfl: loss_dfl: 0.353259 loss: 1.963440 eta: 6:11:26 batch_cost: 0.4957 data_cost: 0.0003 ips: 16.1399 images/s
[11/08 13:38:05] ppdet.utils.checkpoint INFO: Save checkpoint: output/picodet_l_640_coco
[11/08 13:38:08] ppdet.engine INFO:  Model FLOPs : 4.638880G. (image shape is [1, 3, 640, 640])
[11/08 13:38:08] ppdet.engine INFO: Eval iter: 0
[11/08 13:38:19] ppdet.engine INFO: Eval iter: 100
[11/08 13:38:30] ppdet.engine INFO: Eval iter: 200
[11/08 13:38:42] ppdet.engine INFO: Eval iter: 300
[11/08 13:38:53] ppdet.engine INFO: Eval iter: 400
[11/08 13:39:05] ppdet.engine INFO: Eval iter: 500
[11/08 13:39:16] ppdet.engine INFO: Eval iter: 600
[11/08 13:39:28] ppdet.engine INFO: Eval iter: 700
[11/08 13:39:39] ppdet.engine INFO: Eval iter: 800
[11/08 13:39:50] ppdet.engine INFO: Eval iter: 900
[11/08 13:40:02] ppdet.engine INFO: Eval iter: 1000
[11/08 13:40:13] ppdet.engine INFO: Eval iter: 1100
[11/08 13:40:25] ppdet.engine INFO: Eval iter: 1200
[11/08 13:40:36] ppdet.engine INFO: Eval iter: 1300
[11/08 13:40:48] ppdet.engine INFO: Eval iter: 1400
[11/08 13:40:59] ppdet.engine INFO: Eval iter: 1500
[11/08 13:41:11] ppdet.engine INFO: Eval iter: 1600
[11/08 13:41:22] ppdet.engine INFO: Eval iter: 1700
[11/08 13:41:35] ppdet.engine INFO: Eval iter: 1800
[11/08 13:41:46] ppdet.engine INFO: Eval iter: 1900
[11/08 13:41:58] ppdet.engine INFO: Eval iter: 2000
[11/08 13:42:09] ppdet.engine INFO: Eval iter: 2100
[11/08 13:42:20] ppdet.engine INFO: Eval iter: 2200
[11/08 13:42:32] ppdet.engine INFO: Eval iter: 2300
[11/08 13:42:43] ppdet.engine INFO: Eval iter: 2400
[11/08 13:42:54] ppdet.engine INFO: Eval iter: 2500
[11/08 13:43:06] ppdet.engine INFO: Eval iter: 2600
[11/08 13:43:17] ppdet.engine INFO: Eval iter: 2700
[11/08 13:43:29] ppdet.engine INFO: Eval iter: 2800
[11/08 13:43:39] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
[11/08 13:43:40] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 8.46%
[11/08 13:43:40] ppdet.engine INFO: Total sample number: 2888, averge FPS: 8.683230491646983
[11/08 13:43:40] ppdet.engine INFO: Best test bbox ap is 0.085.
[11/08 13:43:40] ppdet.utils.checkpoint INFO: Save checkpoint: output/picodet_l_640_coco

PaddlePaddle / PaddleDetection

AIstudio多卡训练导出的log文件无法可视化 #4519