Antimalweb / URLNet

Code for the paper URLNet - Learning a URL Representation with Deep Learning for Malicious URL Detection
Apache License 2.0
152 stars 60 forks source link

During training checpointer is not saving #3

Closed vinayakumarr closed 6 years ago

vinayakumarr commented 6 years ago

Program works properly, but during training checkpointer is not saved.

IOError: [Errno 2] No such file or directory: 'runs/1000000_emb5_dlm1_32dim_minwf1_1conv3456_1ep/train_1000000_test_500000.txt' img_20180405_195242836 img_20180405_195249415

henryhungle commented 6 years ago

Hi @vinayakumarr ,

IN the program, there are two parameters that affect the checkpoint process:

  1. OUTPUT_DIR: the directory to save dictionaries and model checkpoints
  2. CHECKPOINT_EVERY: the number of training steps to save the model.
    if step % FLAGS.CHECKPOINT_EVERY == 0 or idx == (nb_batches-1): 
                    if dev_loss < min_dev_loss: 
                        path = saver.save(sess, checkpoint_prefix, global_step = step) 
                        print("Dev loss improved: {} -> {}".format(min_dev_loss, dev_loss))
                        print("Saved model checkpoint to {}\n".format(path))
                        min_dev_loss = dev_loss 
                    else: 
                        print("Dev loss did not improve: {} -> {}".format(min_dev_loss, dev_loss))

Note that in the current codes, the model checkpoint is saved only after each evaluation step (EVAL_EVERY) on the validation data and the new validation loss is improved/decreased. You can change this saving process yourself according to your own requirements.