Update logging logic

[xwli-chelsea]

Description

Currently DeText logger logs both network structure and evaluation metrics in the following manner:

• For single node training, there's one logging.txt that has all logs
• For parameter server (worker + evaluator) training, logging.txt logs the network related while eval_log.txt logs the evaluation related from the evaluator.

This PR consolidates the logging method and better arranges the logging output:

• Network related (trainable variables, their shapes, and total deep parameters) are logged in network_structure.txt
• Evaluation logs will be in eval_log.txt for both single node training and ps multi worker training.

Fixes # (issue) AIAF-365

Type of change

Please delete options that are not relevant.

• [x] New feature (non-breaking change which adds functionality)

List all changes

Please list all changes in the commit. Removed unnecessary prints Use network_structure.txt file for logging variable related outputs Removed logging.txt as the useful logging is moved to network_structure.txt Tidied eval logging in best_checkpoint_copier.py

Testing

Tested the training flow with run_detext.sh. Logging files listed below:

$ ls *.txt
eval_log.txt        network_structure.txt

$ cat eval_log.txt 
***** Evaluation on dev set during training *****
## Step 2
loss : 1.2556911706924438
Checking checkpoint model.ckpt-2
keeping checkpoint model.ckpt-2 with metric/ndcg@10 = 0.7103099226951599

## Step 10
loss : 0.9746564030647278
Checking checkpoint model.ckpt-10
keeping checkpoint model.ckpt-10 with metric/ndcg@10 = 1.0
removing old checkpoint model.ckpt-2 with metric/ndcg@10 = 0.7103099226951599

***** Training finished. *****

***** Evaluation on test set with best exported model: *****
global_step = 10
loss = 0.9746564
metric/ndcg@10 = 1.0

$ cat network_structure.txt 
# Trainable variables
  w_pretrained_embedding:0, (14, 32), /device:CPU:0
  cnn/query_cnn_3/kernel:0, (3, 32, 1, 50), /device:CPU:0
  cnn/query_cnn_3/bias:0, (50,), /device:CPU:0
  cnn/doc_cnn_0_3/kernel:0, (3, 32, 1, 50), /device:CPU:0
  cnn/doc_cnn_0_3/bias:0, (50,), /device:CPU:0
  wide_ftr_norm_w:0, (10,), /device:CPU:0
  wide_ftr_norm_b:0, (10,), /device:CPU:0
  hidden_projection_0/kernel:0, (11, 100), /device:CPU:0
  hidden_projection_0/bias:0, (100,), /device:CPU:0
  scoring/kernel:0, (100, 1), /device:CPU:0
  scoring/bias:0, (1,), /device:CPU:0
total cnn parameters: 9700

Test Configuration:

• Firmware version:
• Hardware:
• Toolchain:
• SDK:

Checklist

• [x] My code follows the style guidelines of this project
• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have made corresponding changes to the documentation
• [x] My changes generate no new warnings
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] New and existing unit tests pass locally with my changes
• [x] Any dependent changes have been merged and published in downstream modules