Unable to plot validation loss

[cakuba]

Hi,

We are unable to use PlotValidation.m to get the validation loss plot, and the error as below

root@4e2114f90242:~# PlotValidation.m ./soma/trainout/3fm/log
octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
Parsing log file
Traceback (most recent call last):
  File "/root/caffe_nd_sense_segmentation/tools/extra/parse_log.py", line 196, in <module>
    main()
  File "/root/caffe_nd_sense_segmentation/tools/extra/parse_log.py", line 190, in main
    train_dict_list, test_dict_list = parse_log(args.logfile_path)
  File "/root/caffe_nd_sense_segmentation/tools/extra/parse_log.py", line 55, in parse_log
    logfile_year)
  File "/root/caffe_nd_sense_segmentation/tools/extra/extract_seconds.py", line 9, in extract_datetime_from_line
    month = int(line[0][1:3])
ValueError: invalid literal for int() with base 10: 'ou'
Reading CSV files
error: dlmread: unable to open file './soma/trainout/3fm/log/out.log.train'
error: called from
    csvread at line 34 column 5
    /root/PlotValidation.m at line 47 column 14

and the first few line of our log file looks like

Could not create logging file: No such file or directory 
COULD NOT CREATE A LOGGINGFILE 20190410-023032.341!I0410 02:30:32.328800   341 caffe.cpp:93] FLAGS_
gpu.size() = 1
I0410 02:30:32.328913   341 caffe.cpp:96] FLAGS_strings.size() = 1
I0410 02:30:32.328929   341 caffe.cpp:186] Using GPUs 0
I0410 02:30:32.663046   341 solver.cpp:48] Initializing solver from parameters: 
test_iter: 20
test_interval: 10
base_lr: 0.01
display: 3
max_iter: 10
lr_policy: "poly"
power: 0.8
momentum: 0.9
weight_decay: 0.0005
snapshot: 2000
snapshot_prefix: "/root/soma/trainout/3fm/trainedmodel/3fm_classifer"
solver_mode: GPU
device_id: 0
net: "train_val.prototxt"
average_loss: 16
iter_size: 8

not sure if this error is related to the message "COULD NOT CREATE A LOGGINGFILE" in out.log file. But I tried to remove the entire sentence of "Could not create logging file: No such file or directory COULD NOT CREATE A LOGGINGFILE " from the out.log, but still, PlotValidation.m yields the same error. Any hint? Thanks for your help.

Brett

[cakuba]

OK, it seems that the problem is related to the line in caffetrain.sh

GLOG_log_dir=$log_dir caffe.bin train --solver=$model_dir/solver.prototxt --gpu $gpu $snapshot_opts > "${model_dir}/log/out.log" 2>&1

when we use "./soma/trainout" instead of "~/soma/trainout". A tiny difference but did affect the output of the log file. Thanks anyway,