search

thushv89 / exercises_thushv_dot_com

Here lies all the exercises I implement and share in my website
http://www.thushv.com
209 stars 156 forks source link

I change to my word embeding and my data but not work help me. #1

Open vhuytdt opened 6 years ago

vhuytdt commented 6 years ago

i train work embedding 300 dimension with 12000 word from https://nlp.stanford.edu/projects/nmt/data/iwslt15.en-vi/. but not work.. help me tks you very much.

Started Training .

InvalidArgumentError Traceback (most recent call last) ~/tensorflow-dev/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, args) 1360 try: -> 1361 return fn(args) 1362 except errors.OpError as e:

~/tensorflow-dev/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata) 1339 return tf_session.TF_Run(session, options, feed_dict, fetch_list, -> 1340 target_list, status, run_metadata) 1341

~/tensorflow-dev/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py in exit(self, type_arg, value_arg, traceback_arg) 515 compat.as_text(c_api.TF_Message(self.status.status)), --> 516 c_api.TF_GetCode(self.status.status)) 517 # Delete the underlying status object from memory otherwise it stays alive

InvalidArgumentError: indices[5] = 12036 is not in [0, 12000) [[Node: embedding_lookup_16 = Gather[Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@Const"], validate_indices=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](Const, _arg_enc_train_inputs_16_0_154)]]

During handling of the above exception, another exception occurred:

InvalidArgumentError Traceback (most recent call last)

in () 51 # ======================= OPTIMIZATION ========================== 52 if step < 10000: ---> 53 _,l,tr_pred = sess.run([adam_optimize,loss,train_prediction], feed_dict=feed_dict) 54 else: 55 _,l,tr_pred = sess.run([sgd_optimize,loss,train_prediction], feed_dict=feed_dict) ~/tensorflow-dev/lib/python3.5/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata) 903 try: 904 result = self._run(None, fetches, feed_dict, options_ptr, --> 905 run_metadata_ptr) 906 if run_metadata: 907 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) ~/tensorflow-dev/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata) 1135 if final_fetches or final_targets or (handle and feed_dict_tensor): 1136 results = self._do_run(handle, final_targets, final_fetches, -> 1137 feed_dict_tensor, options, run_metadata) 1138 else: 1139 results = [] ~/tensorflow-dev/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata) 1353 if handle is None: 1354 return self._do_call(_run_fn, self._session, feeds, fetches, targets, -> 1355 options, run_metadata) 1356 else: 1357 return self._do_call(_prun_fn, self._session, handle, feeds, fetches) ~/tensorflow-dev/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args) 1372 except KeyError: 1373 pass -> 1374 raise type(e)(node_def, op, message) 1375 1376 def _extend_graph(self): InvalidArgumentError: indices[5] = 12036 is not in [0, 12000) [[Node: embedding_lookup_16 = Gather[Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@Const"], validate_indices=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](Const, _arg_enc_train_inputs_16_0_154)]] Caused by op 'embedding_lookup_16', defined at: File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.5/runpy.py", line 85, in _run_code exec(code, run_globals) File "/root/tensorflow-dev/lib/python3.5/site-packages/ipykernel_launcher.py", line 16, in app.launch_new_instance() File "/root/tensorflow-dev/lib/python3.5/site-packages/traitlets/config/application.py", line 658, in launch_instance app.start() File "/root/tensorflow-dev/lib/python3.5/site-packages/ipykernel/kernelapp.py", line 486, in start self.io_loop.start() File "/root/tensorflow-dev/lib/python3.5/site-packages/tornado/platform/asyncio.py", line 127, in start self.asyncio_loop.run_forever() File "/usr/lib/python3.5/asyncio/base_events.py", line 345, in run_forever self._run_once() File "/usr/lib/python3.5/asyncio/base_events.py", line 1312, in _run_once handle._run() File "/usr/lib/python3.5/asyncio/events.py", line 125, in _run self._callback(*self._args) File "/root/tensorflow-dev/lib/python3.5/site-packages/tornado/platform/asyncio.py", line 117, in _handle_events handler_func(fileobj, events) File "/root/tensorflow-dev/lib/python3.5/site-packages/tornado/stack_context.py", line 276, in null_wrapper return fn(*args, **kwargs) File "/root/tensorflow-dev/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 450, in _handle_events self._handle_recv() File "/root/tensorflow-dev/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 480, in _handle_recv self._run_callback(callback, msg) File "/root/tensorflow-dev/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 432, in _run_callback callback(*args, **kwargs) File "/root/tensorflow-dev/lib/python3.5/site-packages/tornado/stack_context.py", line 276, in null_wrapper return fn(*args, **kwargs) File "/root/tensorflow-dev/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher return self.dispatch_shell(stream, msg) File "/root/tensorflow-dev/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell handler(stream, idents, msg) File "/root/tensorflow-dev/lib/python3.5/site-packages/ipykernel/kernelbase.py", line 399, in execute_request user_expressions, allow_stdin) File "/root/tensorflow-dev/lib/python3.5/site-packages/ipykernel/ipkernel.py", line 208, in do_execute res = shell.run_cell(code, store_history=store_history, silent=silent) File "/root/tensorflow-dev/lib/python3.5/site-packages/ipykernel/zmqshell.py", line 537, in run_cell return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs) File "/root/tensorflow-dev/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2662, in run_cell raw_cell, store_history, silent, shell_futures) File "/root/tensorflow-dev/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2785, in _run_cell interactivity=interactivity, compiler=compiler, result=result) File "/root/tensorflow-dev/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2903, in run_ast_nodes if self.run_code(code, result): File "/root/tensorflow-dev/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2963, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 23, in encoder_emb_inp = [tf.nn.embedding_lookup(encoder_emb_layer, src) for src in enc_train_inputs] File "", line 23, in encoder_emb_inp = [tf.nn.embedding_lookup(encoder_emb_layer, src) for src in enc_train_inputs] File "/root/tensorflow-dev/lib/python3.5/site-packages/tensorflow/python/ops/embedding_ops.py", line 327, in embedding_lookup transform_fn=None) File "/root/tensorflow-dev/lib/python3.5/site-packages/tensorflow/python/ops/embedding_ops.py", line 151, in _embedding_lookup_and_transform result = _clip(_gather(params[0], ids, name=name), ids, max_norm) File "/root/tensorflow-dev/lib/python3.5/site-packages/tensorflow/python/ops/embedding_ops.py", line 55, in _gather return array_ops.gather(params, ids, name=name) File "/root/tensorflow-dev/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 2667, in gather params, indices, validate_indices=validate_indices, name=name) File "/root/tensorflow-dev/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1777, in gather validate_indices=validate_indices, name=name) File "/root/tensorflow-dev/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/root/tensorflow-dev/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3271, in create_op op_def=op_def) File "/root/tensorflow-dev/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1650, in __init__ self._traceback = self._graph._extract_stack() # pylint: disable=protected-access InvalidArgumentError (see above for traceback): indices[5] = 12036 is not in [0, 12000) [[Node: embedding_lookup_16 = Gather[Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@Const"], validate_indices=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](Const, _arg_enc_train_inputs_16_0_154)]]
thushv89 commented 6 years ago

@vhuytdt Can you send me the code with changes you did? It's hard to pinpoint this with just the error

vhuytdt commented 6 years ago

hey @thushv89 . i fixed it. because i use word embedding don't same vocabulary size ( dimesion ) with encoder and decoder. Tks u about Article, NMT with tensorflow. <3. I do w2vect fowllow http://adventuresinmachinelearning.com/word2vec-tutorial-tensorflow/. if i use pre-train word embedding, should i do wirte revert-dictionary? will use it in encoder and decoder ?. My data: IWSLT'15 English-Vietnamese data [Small] Train (133K sentence pairs): [https://nlp.stanford.edu/projects/nmt/data/iwslt15.en-vi/train.en] encoder [https://nlp.stanford.edu/projects/nmt/data/iwslt15.en-vi/train.vi] decoder Vocabularies (top 50K frequent words): [https://nlp.stanford.edu/projects/nmt/data/iwslt15.en-vi/vocab.en] encoder [https://nlp.stanford.edu/projects/nmt/data/iwslt15.en-vi/vocab.vi] decoder tks u @thushv89

thushv89 commented 6 years ago

Hi @vhuytdt , Are you asking if you should write the reverse dictionary if you use pre-trained embeddings? To answer that, if you use pretrained-embeddings, make sure your vocabulary in the pretrained-embeddings matches well with the vocabulary provided in the IWSLT'15. For the ones that don't match, you can either have randomly initialized vectors and jointly train it with your model or, replace those words with a special token (), if it's only a few words.

vhuytdt commented 6 years ago

Hi @thushv89 . train.en vocabulary size don't same wwith train.vi vocabulary size. Here : vocab.en -> 15xxx words. vocab.vi -> 7xxx words. how to choose number-vocabulary-size ? => embeddings = tf.Variable( tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0)) loss of w2vec affect to encoder and decoder ?. in present, i use tensorflow to train NMT Model about comic. Can you give me id-skype? many thank.