Before training starts I get this error: pandas.errors.ParserError: Expected 8 fields in line 7, saw 9. Error could possibly be due to quotes being ignored when a multi-char delimiter is used.
It's due to the space in chrome 00type=g line below, which is what you obtain after you process the output of nvidia-smi in the get_nvidia_gpu_power function.
gpu pid type sm mem enc dec command
0 1351 G 0 0 0 0 Xorg
0 1441 G 0 0 0 0 gnome0shell
0 1973 G 0 0 0 0 Xorg
0 2090 G 0 0 0 0 gnome0shell
0 83636 C 0 0 0 0 python
0 105873 G 0 0 0 0 chrome 00type=g
0 1807 C 4 1 0 0 python3
0 2090 G 4 0 0 0 gnome0shell
0 1807 C 0 0 0 0 python3
0 2090 G 6 0 0 0 gnome0shell
0 2090 G 2 0 0 0 gnome0shell
When running that table through pd.read_csv(StringIO(out_str_final), delim_whitespace=True), delim_whitespace turns chrome 00type=g into 2 columns.
To fix, before doing out_str_final = out_str_final.replace("-", "0"), do out_str_final = out_str_final.replace(" --", "").
Before training starts I get this error:
pandas.errors.ParserError: Expected 8 fields in line 7, saw 9. Error could possibly be due to quotes being ignored when a multi-char delimiter is used.
It's due to the space in
chrome 00type=g
line below, which is what you obtain after you process the output ofnvidia-smi
in theget_nvidia_gpu_power
function.When running that table through
pd.read_csv(StringIO(out_str_final), delim_whitespace=True)
,delim_whitespace
turnschrome 00type=g
into 2 columns.To fix, before doing
out_str_final = out_str_final.replace("-", "0")
, doout_str_final = out_str_final.replace(" --", "")
.