envy@ub1404:/media/envy/data1t/os_prj/github/variational-text-tensorflow$ python main.py --dataset ptb
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so locally
{'checkpoint_dir': 'checkpoint',
'dataset': 'ptb',
'decay_rate': 0.96,
'decay_step': 10000,
'embed_dim': 500,
'forward_only': False,
'h_dim': 50,
'learning_rate': 0.001,
'maxiter': 450000,
'model': 'nvdm'}
[] save ./data/ptb/vocab.pkl
[] save ./data/ptb/train.txt.npy
[] save ./data/ptb/valid.txt.npy
[_] save ./data/ptb/test.txt.npy
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:900] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GTX 950M
major: 5 minor: 0 memoryClockRate (GHz) 1.124
pciBusID 0000:01:00.0
Total memory: 4.00GiB
Free memory: 3.57GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0: Y
I tensorflow/core/common_runtime/gpu/gpudevice.cc:755] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 950M, pci bus id: 0000:01:00.0)
WARNING:tensorflow:Passing a GraphDef to the SummaryWriter is deprecated. Pass a Graph object instead, such as sess.graph.
[] Loading checkpoints...
[!] Load failed...
Step: [ 0/450000] time: 0.3199, loss: 234.64193726
[_] Saving checkpoints...
Traceback (most recent call last):
File "main.py", line 50, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv))
File "main.py", line 43, in main
model.train(FLAGS)
File "/media/envy/data1t/os_prj/github/variational-text-tensorflow/models/nvdm.py", line 154, in train
self.save(self.checkpoint_dir, step)
File "/media/envy/data1t/os_prj/github/variational-text-tensorflow/models/base.py", line 28, in save
os.path.join(checkpoint_dir, model_name), global_step=global_step)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1037, in save
{self.saver_def.filename_tensor_name: checkpoint_file})
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 340, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 564, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 637, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 659, in _do_call
e.code)
tensorflow.python.framework.errors.UnimplementedError: File system scheme checkpoint/ptb_h_dim not implemented
[[Node: save_1/save = SaveSlices[T=[DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save_1/Const_0, save_1/save/tensor_names, save_1/save/shapes_and_slices, Variable, beta1_power/_49, beta1_power_1/_51, beta1_power_2/_53, beta2_power/_55, beta2_power_1/_57, beta2_power_2/_59, encoder/l1/Bias/_61, encoder/l1/Bias/Adam/_63, encoder/l1/Bias/Adam_1/_65, encoder/l1/Bias/Adam_2/_67, encoder/l1/Bias/Adam_3/_69, encoder/l1/Matrix/_71, encoder/l1/Matrix/Adam/_73, encoder/l1/Matrix/Adam_1/_75, encoder/l1/Matrix/Adam_2/_77, encoder/l1/Matrix/Adam_3/_79, encoder/l2/Bias/_81, encoder/l2/Bias/Adam/_83, encoder/l2/Bias/Adam_1/_85, encoder/l2/Bias/Adam_2/_87, encoder/l2/Bias/Adam_3/_89, encoder/l2/Matrix/_91, encoder/l2/Matrix/Adam/_93, encoder/l2/Matrix/Adam_1/_95, encoder/l2/Matrix/Adam_2/_97, encoder/l2/Matrix/Adam_3/_99, encoder/log_sigma_sq/Bias/_101, encoder/log_sigma_sq/Bias/Adam/_103, encoder/log_sigma_sq/Bias/Adam_1/_105, encoder/log_sigma_sq/Bias/Adam_2/_107, encoder/log_sigma_sq/Bias/Adam_3/_109, encoder/log_sigma_sq/Matrix/_111, encoder/log_sigma_sq/Matrix/Adam/_113, encoder/log_sigma_sq/Matrix/Adam_1/_115, encoder/log_sigma_sq/Matrix/Adam_2/_117, encoder/log_sigma_sq/Matrix/Adam_3/_119, encoder/mu/Bias/_121, encoder/mu/Bias/Adam/_123, encoder/mu/Bias/Adam_1/_125, encoder/mu/Bias/Adam_2/_127, encoder/mu/Bias/Adam_3/_129, encoder/mu/Matrix/_131, encoder/mu/Matrix/Adam/_133, encoder/mu/Matrix/Adam_1/_135, encoder/mu/Matrix/Adam_2/_137, encoder/mu/Matrix/Adam_3/_139, generator/R/_141, generator/R/Adam/_143, generator/R/Adam_1/_145, generator/R/Adam_2/_147, generator/R/Adam_3/_149, generator/b/_151, generator/b/Adam/_153, generator/b/Adam_1/_155, generator/b/Adam_2/_157, generator/b/Adam_3/_159)]]
Caused by op u'save_1/save', defined at:
File "main.py", line 50, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv))
File "main.py", line 43, in main
model.train(FLAGS)
File "/media/envy/data1t/os_prj/github/variational-text-tensorflow/models/nvdm.py", line 154, in train
self.save(self.checkpoint_dir, step)
File "/media/envy/data1t/os_prj/github/variational-text-tensorflow/models/base.py", line 18, in save
self.saver = tf.train.Saver()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 832, in init
restore_sequentially=restore_sequentially)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 500, in build
save_tensor = self._AddSaveOps(filename_tensor, vars_to_save)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 197, in _AddSaveOps
save = self.save_op(filename_tensor, vars_to_save)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 149, in save_op
tensor_slices=[vs.slice_spec for vs in vars_to_save])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/io_ops.py", line 172, in _save
tensors, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 341, in _save_slices
name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 661, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2154, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1154, in init
self._traceback = _extract_stack()
envy@ub1404:/media/envy/data1t/os_prj/github/variational-text-tensorflow$ python main.py --dataset ptb I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so locally I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so locally I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so locally I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so locally {'checkpoint_dir': 'checkpoint', 'dataset': 'ptb', 'decay_rate': 0.96, 'decay_step': 10000, 'embed_dim': 500, 'forward_only': False, 'h_dim': 50, 'learning_rate': 0.001, 'maxiter': 450000, 'model': 'nvdm'} [] save ./data/ptb/vocab.pkl [] save ./data/ptb/train.txt.npy [] save ./data/ptb/valid.txt.npy [_] save ./data/ptb/test.txt.npy I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:900] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: name: GeForce GTX 950M major: 5 minor: 0 memoryClockRate (GHz) 1.124 pciBusID 0000:01:00.0 Total memory: 4.00GiB Free memory: 3.57GiB I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0: Y I tensorflow/core/common_runtime/gpu/gpudevice.cc:755] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 950M, pci bus id: 0000:01:00.0) WARNING:tensorflow:Passing a
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv))
File "main.py", line 43, in main
model.train(FLAGS)
File "/media/envy/data1t/os_prj/github/variational-text-tensorflow/models/nvdm.py", line 154, in train
self.save(self.checkpoint_dir, step)
File "/media/envy/data1t/os_prj/github/variational-text-tensorflow/models/base.py", line 28, in save
os.path.join(checkpoint_dir, model_name), global_step=global_step)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1037, in save
{self.saver_def.filename_tensor_name: checkpoint_file})
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 340, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 564, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 637, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 659, in _do_call
e.code)
tensorflow.python.framework.errors.UnimplementedError: File system scheme checkpoint/ptb_h_dim not implemented
[[Node: save_1/save = SaveSlices[T=[DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save_1/Const_0, save_1/save/tensor_names, save_1/save/shapes_and_slices, Variable, beta1_power/_49, beta1_power_1/_51, beta1_power_2/_53, beta2_power/_55, beta2_power_1/_57, beta2_power_2/_59, encoder/l1/Bias/_61, encoder/l1/Bias/Adam/_63, encoder/l1/Bias/Adam_1/_65, encoder/l1/Bias/Adam_2/_67, encoder/l1/Bias/Adam_3/_69, encoder/l1/Matrix/_71, encoder/l1/Matrix/Adam/_73, encoder/l1/Matrix/Adam_1/_75, encoder/l1/Matrix/Adam_2/_77, encoder/l1/Matrix/Adam_3/_79, encoder/l2/Bias/_81, encoder/l2/Bias/Adam/_83, encoder/l2/Bias/Adam_1/_85, encoder/l2/Bias/Adam_2/_87, encoder/l2/Bias/Adam_3/_89, encoder/l2/Matrix/_91, encoder/l2/Matrix/Adam/_93, encoder/l2/Matrix/Adam_1/_95, encoder/l2/Matrix/Adam_2/_97, encoder/l2/Matrix/Adam_3/_99, encoder/log_sigma_sq/Bias/_101, encoder/log_sigma_sq/Bias/Adam/_103, encoder/log_sigma_sq/Bias/Adam_1/_105, encoder/log_sigma_sq/Bias/Adam_2/_107, encoder/log_sigma_sq/Bias/Adam_3/_109, encoder/log_sigma_sq/Matrix/_111, encoder/log_sigma_sq/Matrix/Adam/_113, encoder/log_sigma_sq/Matrix/Adam_1/_115, encoder/log_sigma_sq/Matrix/Adam_2/_117, encoder/log_sigma_sq/Matrix/Adam_3/_119, encoder/mu/Bias/_121, encoder/mu/Bias/Adam/_123, encoder/mu/Bias/Adam_1/_125, encoder/mu/Bias/Adam_2/_127, encoder/mu/Bias/Adam_3/_129, encoder/mu/Matrix/_131, encoder/mu/Matrix/Adam/_133, encoder/mu/Matrix/Adam_1/_135, encoder/mu/Matrix/Adam_2/_137, encoder/mu/Matrix/Adam_3/_139, generator/R/_141, generator/R/Adam/_143, generator/R/Adam_1/_145, generator/R/Adam_2/_147, generator/R/Adam_3/_149, generator/b/_151, generator/b/Adam/_153, generator/b/Adam_1/_155, generator/b/Adam_2/_157, generator/b/Adam_3/_159)]]
Caused by op u'save_1/save', defined at:
File "main.py", line 50, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv))
File "main.py", line 43, in main
model.train(FLAGS)
File "/media/envy/data1t/os_prj/github/variational-text-tensorflow/models/nvdm.py", line 154, in train
self.save(self.checkpoint_dir, step)
File "/media/envy/data1t/os_prj/github/variational-text-tensorflow/models/base.py", line 18, in save
self.saver = tf.train.Saver()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 832, in init
restore_sequentially=restore_sequentially)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 500, in build
save_tensor = self._AddSaveOps(filename_tensor, vars_to_save)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 197, in _AddSaveOps
save = self.save_op(filename_tensor, vars_to_save)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 149, in save_op
tensor_slices=[vs.slice_spec for vs in vars_to_save])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/io_ops.py", line 172, in _save
tensors, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 341, in _save_slices
name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 661, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2154, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1154, in init
self._traceback = _extract_stack()
GraphDef
to the SummaryWriter is deprecated. Pass aGraph
object instead, such assess.graph
. [] Loading checkpoints... [!] Load failed... Step: [ 0/450000] time: 0.3199, loss: 234.64193726 [_] Saving checkpoints... Traceback (most recent call last): File "main.py", line 50, inenvy@ub1404:/media/envy/data1t/os_prj/github/variational-text-tensorflow$