Comments on a few issues (rnn, python libraries)

[coltekin]

Instead of sending emails, I decided to comment on a few things we discussed yesterday here.

@jonasbiegert: the RNN implementation seems doing what we were aiming to do. A few possible directions to go:

• A hidden layer (or maybe more) between the answer input may be a good idea. For example, in your code:

  questions_input = Input(shape=(len(train_x[0]),))
  choices_input = Input(shape=(1,))
  choice_m = Dense(32, activation='relu')                    ### added
  x = Embedding(input_dim=len(vocab) + 1, output_dim=64, input_length=len(train_x[0]))(questions_input)
  x = GRU(64)(x)
  x = keras.layers.concatenate([x, choice_m], axis=1)        ### changed
  # x = Dense(64, activation='relu')(x)                      ### added, see below
  output = Dense(1, activation='sigmoid')(x)

• A non-linear classifier may also be useful (the last addition in the code above).
• I suggest trying to produce some evaluation results. You have all the ingredients. I think we are not interested in what numbers the the system produce, as long as the max matches with the gold standard. Can you run a few tests, and report both training and test accuracy?
• The strategy I suggest is to try to see if you can get very good accuracy on the training set by manipulating the network architecture (layers and the number of units). Then, start adding measures to prevent overfitting (dropout in a few places in the network). Hopefully this will increase the accuracy on the development set.

@MegamindHenry: I had another quick look at ALGES with no success, but I will give it a try next week (probably Wed). If any of you gets any progress, not to replicate the work, please drop a note here.

[jonasbiegert]

@coltekin: I know theres around 650 train questions and 180 test questions so I can somewhat read accuracy with what I already have (its really low :D). The problem is that currently since I am adding each question as many times as there are answer options and then splitting the data it's hard to keep track how many train and test questions there are exactly, but I was planning to and will fix this by filtering the train.sat.json for numerical answer question and splitting the result of that into a json file containing 80% of those and another json file with the remaining 20% which I will then process to be the train and dev set

[jonasbiegert]

I will try your suggestions, of course.

[MegamindHenry]

@coltekin So far, I don't have errors in jsonrpc, (but what I did may affect other errors which I don't know if it is because of jsonrpc library). However, I had an error on nltk. It said that resource "wordnet_ic" when import nltk and nltk.download('wordnet_ic'). I have google "wordnet_ic", but nothing really helpful. So, I need to dig more into nltk. I hope you may be familiar with nltk :-P.

I have python 3.7. And jsonrpclib is located in python library site-packages. I don't know if it is basic python libraries or other packages.(‎⁨Macintosh HD⁩ ▸ ⁨Library⁩ ▸ ⁨Frameworks⁩ ▸ ⁨Python.framework⁩ ▸ ⁨Versions⁩ ▸ ⁨3.7⁩ ▸ ⁨lib⁩ ▸ ⁨python3.7⁩ ▸ ⁨site-packages⁩ ▸ ⁨jsonrpclib⁩).

Here are what I did: I changed xmlrpclib to .client

from xmlrpc.client import Transport as XMLTransport from xmlrpc.client import SafeTransport as XMLSafeTransport from xmlrpc.client import ServerProxy as XMLServerProxy from xmlrpc.client import _Method as XML_Method

as well as httplib

from http.client import HTTP, HTTPConnection

and I delete HTTP, since there is no more HTTP class in http.client and it may merge into HTTP Connection(I am not sure, but they have similar functions)

from http.client import HTTPConnection

[MegamindHenry]

@coltekin And also, I will try to setup a virtual python enviorment and try to find all libraries during that time. See if it is working.

[jonasbiegert]

So here are my results (every model includes the changes of the ones above it, epochs = 20):

initial model: accuracy on test set: 31 / 167, 18,6% accuracy on training set: 131 / 667, 19,6% with choices_m: accuracy on test set: 33 / 167, 20,0% accuracy on training set: 128 / 667, 19,1% with dense layer after concatenation: accuracy on test set: 31 / 167, 18,6% accuracy on training set: 129 / 667, 19,3% with 4 additional dense layers after concatenation: accuracy on test set: 32 / 167, 19,2% accuracy on training set: 138 / 667, 20,7% with 1024 units in the GRU (cause why not go completely overboard here): accuracy on test set: 33 / 167, 19,8% accuracy on training set: 136 / 667, 20,4%

so clearly, none of these changes made a difference. I'm somewhat stumped on what to do now. Should I change the units in the dense layers, add more of them before or after the recurrent unit, have a different activation function in those layers? Using a SimpleRNN or LSTM instead of a GRU doesnt change the results, even with just one epoch the numbers are just slightly worse, so I'm not sure what's going on here exactly.

[MegamindHenry]

I have fixed this nltk issue with 'wordnet_ic'. It seems that we need to download wordnet_ic package manully. And there is a minor issue when downloading it like "certificate verify failed".

Now, there is something wrong with simplejson. Haha! (Just download it :-P)

Traceback (most recent call last):
  File "parseSdata.py", line 8, in <module>
    from ILPformat import parse_json
  File "/Users/HakunaMatata/Desktop/TACL2015-master/ILPformat.py", line 6, in <module>
    from simplejson import loads
ModuleNotFoundError: No module named 'simplejson'
Traceback (most recent call last):
  File "ILPFormat.py", line 6, in <module>
    from simplejson import loads
ModuleNotFoundError: No module named 'simplejson'

[MegamindHenry]

Then, I fix errors about types.StringTypes and other types. And then another error with json encoding.

Then, another huge error come up:

Xuefengs-MacBook-Air:TACL2015-master HakunaMatata$ ./preSplit.sh questions.json 
0
 Joan found 70 seashells on the beach . she gave Sam some of her seashells . She has 27 seashell left . How many seashells did she give to Sam  ? 
Traceback (most recent call last):
  File "parseSdata.py", line 88, in <module>
    make_eq(q,a,VERBOSE,TRAIN)
  File "parseSdata.py", line 50, in make_eq
    story = nlp.parse(problem)
  File "parseSdata.py", line 17, in parse
    return json.loads(self.server.parse(text))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jsonrpclib/jsonrpc.py", line 288, in __call__
    return self.__send(self.__name, args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jsonrpclib/jsonrpc.py", line 237, in _request
    response = self._run_request(request)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jsonrpclib/jsonrpc.py", line 255, in _run_request
    verbose=self.__verbose
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1154, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1166, in single_request
    http_conn = self.send_request(host, handler, request_body, verbose)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1279, in send_request
    self.send_content(connection, request_body)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jsonrpclib/jsonrpc.py", line 126, in send_content
    connection.endheaders()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 928, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 727, in create_connection
    raise err
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 716, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
0
 Joan found 70 seashells on the beach . she gave Sam some of her seashells . She has 27 seashell left . How many seashells did she give to Sam  ? 
Traceback (most recent call last):
  File "ILPFormat.py", line 108, in <module>
    make_eq(q,a,VERBOSE,TRAIN)
  File "ILPFormat.py", line 56, in make_eq
    story = nlp.parse(problem)
  File "ILPFormat.py", line 20, in parse
    return loads(self.server.parse(text))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jsonrpclib/jsonrpc.py", line 288, in __call__
    return self.__send(self.__name, args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jsonrpclib/jsonrpc.py", line 237, in _request
    response = self._run_request(request)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jsonrpclib/jsonrpc.py", line 255, in _run_request
    verbose=self.__verbose
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1154, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1166, in single_request
    http_conn = self.send_request(host, handler, request_body, verbose)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1279, in send_request
    self.send_content(connection, request_body)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jsonrpclib/jsonrpc.py", line 126, in send_content
    connection.endheaders()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 928, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 727, in create_connection
    raise err
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 716, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

Well, I think I would better look through Alina's papers today instead of try to fix their errors. :-P