Open coltekin opened 5 years ago
@coltekin: I know theres around 650 train questions and 180 test questions so I can somewhat read accuracy with what I already have (its really low :D). The problem is that currently since I am adding each question as many times as there are answer options and then splitting the data it's hard to keep track how many train and test questions there are exactly, but I was planning to and will fix this by filtering the train.sat.json for numerical answer question and splitting the result of that into a json file containing 80% of those and another json file with the remaining 20% which I will then process to be the train and dev set
I will try your suggestions, of course.
@coltekin So far, I don't have errors in jsonrpc, (but what I did may affect other errors which I don't know if it is because of jsonrpc library). However, I had an error on nltk. It said that resource "wordnet_ic" when import nltk and nltk.download('wordnet_ic'). I have google "wordnet_ic", but nothing really helpful. So, I need to dig more into nltk. I hope you may be familiar with nltk :-P.
I have python 3.7. And jsonrpclib is located in python library site-packages. I don't know if it is basic python libraries or other packages.(Macintosh HD ▸ Library ▸ Frameworks ▸ Python.framework ▸ Versions ▸ 3.7 ▸ lib ▸ python3.7 ▸ site-packages ▸ jsonrpclib).
Here are what I did: I changed xmlrpclib to .client
from xmlrpc.client import Transport as XMLTransport from xmlrpc.client import SafeTransport as XMLSafeTransport from xmlrpc.client import ServerProxy as XMLServerProxy from xmlrpc.client import _Method as XML_Method
as well as httplib
from http.client import HTTP, HTTPConnection
and I delete HTTP, since there is no more HTTP class in http.client and it may merge into HTTP Connection(I am not sure, but they have similar functions)
from http.client import HTTPConnection
@coltekin And also, I will try to setup a virtual python enviorment and try to find all libraries during that time. See if it is working.
So here are my results (every model includes the changes of the ones above it, epochs = 20):
initial model: accuracy on test set: 31 / 167, 18,6% accuracy on training set: 131 / 667, 19,6% with choices_m: accuracy on test set: 33 / 167, 20,0% accuracy on training set: 128 / 667, 19,1% with dense layer after concatenation: accuracy on test set: 31 / 167, 18,6% accuracy on training set: 129 / 667, 19,3% with 4 additional dense layers after concatenation: accuracy on test set: 32 / 167, 19,2% accuracy on training set: 138 / 667, 20,7% with 1024 units in the GRU (cause why not go completely overboard here): accuracy on test set: 33 / 167, 19,8% accuracy on training set: 136 / 667, 20,4%
so clearly, none of these changes made a difference. I'm somewhat stumped on what to do now. Should I change the units in the dense layers, add more of them before or after the recurrent unit, have a different activation function in those layers? Using a SimpleRNN or LSTM instead of a GRU doesnt change the results, even with just one epoch the numbers are just slightly worse, so I'm not sure what's going on here exactly.
I have fixed this nltk issue with 'wordnet_ic'. It seems that we need to download wordnet_ic package manully. And there is a minor issue when downloading it like "certificate verify failed".
Now, there is something wrong with simplejson. Haha! (Just download it :-P)
Traceback (most recent call last):
File "parseSdata.py", line 8, in <module>
from ILPformat import parse_json
File "/Users/HakunaMatata/Desktop/TACL2015-master/ILPformat.py", line 6, in <module>
from simplejson import loads
ModuleNotFoundError: No module named 'simplejson'
Traceback (most recent call last):
File "ILPFormat.py", line 6, in <module>
from simplejson import loads
ModuleNotFoundError: No module named 'simplejson'
Then, I fix errors about types.StringTypes and other types. And then another error with json encoding.
Then, another huge error come up:
Xuefengs-MacBook-Air:TACL2015-master HakunaMatata$ ./preSplit.sh questions.json
0
Joan found 70 seashells on the beach . she gave Sam some of her seashells . She has 27 seashell left . How many seashells did she give to Sam ?
Traceback (most recent call last):
File "parseSdata.py", line 88, in <module>
make_eq(q,a,VERBOSE,TRAIN)
File "parseSdata.py", line 50, in make_eq
story = nlp.parse(problem)
File "parseSdata.py", line 17, in parse
return json.loads(self.server.parse(text))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jsonrpclib/jsonrpc.py", line 288, in __call__
return self.__send(self.__name, args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jsonrpclib/jsonrpc.py", line 237, in _request
response = self._run_request(request)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jsonrpclib/jsonrpc.py", line 255, in _run_request
verbose=self.__verbose
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1154, in request
return self.single_request(host, handler, request_body, verbose)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1166, in single_request
http_conn = self.send_request(host, handler, request_body, verbose)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1279, in send_request
self.send_content(connection, request_body)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jsonrpclib/jsonrpc.py", line 126, in send_content
connection.endheaders()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 928, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 727, in create_connection
raise err
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
0
Joan found 70 seashells on the beach . she gave Sam some of her seashells . She has 27 seashell left . How many seashells did she give to Sam ?
Traceback (most recent call last):
File "ILPFormat.py", line 108, in <module>
make_eq(q,a,VERBOSE,TRAIN)
File "ILPFormat.py", line 56, in make_eq
story = nlp.parse(problem)
File "ILPFormat.py", line 20, in parse
return loads(self.server.parse(text))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jsonrpclib/jsonrpc.py", line 288, in __call__
return self.__send(self.__name, args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jsonrpclib/jsonrpc.py", line 237, in _request
response = self._run_request(request)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jsonrpclib/jsonrpc.py", line 255, in _run_request
verbose=self.__verbose
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1154, in request
return self.single_request(host, handler, request_body, verbose)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1166, in single_request
http_conn = self.send_request(host, handler, request_body, verbose)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1279, in send_request
self.send_content(connection, request_body)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jsonrpclib/jsonrpc.py", line 126, in send_content
connection.endheaders()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 928, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 727, in create_connection
raise err
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
Well, I think I would better look through Alina's papers today instead of try to fix their errors. :-P
Instead of sending emails, I decided to comment on a few things we discussed yesterday here.
@jonasbiegert: the RNN implementation seems doing what we were aiming to do. A few possible directions to go:
@MegamindHenry: I had another quick look at ALGES with no success, but I will give it a try next week (probably Wed). If any of you gets any progress, not to replicate the work, please drop a note here.