Upload error with large file

[sr320]

I am trying to upload a file (rather big - 25GB) http://eagle.fish.washington.edu/cnidarian/gp_association_goa_uniprot_2

Below is the error output. It uploaded about half of the data. One thing to point out is if you happen to not see this error, there is no indication that it is incomplete in SQLShare. https://sqlshare.escience.washington.edu/sqlshare/#s=query/sr320%40washington.edu/gp_association_goa_uniprot_121113

Any advice on how to upload- I will point out I really only need column 2 and column 4.

parsing 46E49B03...
processing chunk line 158084694 to 159084473 (33104.2104089 s elapsed)
pushing /Volumes/Bay3 scratch/gp_association_goa_uniprot_2...
Traceback (most recent call last):
  File "/Users/sr320/sqlshare-pythonclient/tools/singleupload.py", line 42, in <module>
    main()
  File "/Users/sr320/sqlshare-pythonclient/tools/singleupload.py", line 39, in main
    conn.uploadone(args[0], datasetname)
  File "/Library/Python/2.7/site-packages/SQLShare-0.1a-py2.7.egg/sqlshare/__init__.py", line 183, in uploadone
    self.upload_chunk(fn, dataset_name, chunk, True, False)
  File "/Library/Python/2.7/site-packages/SQLShare-0.1a-py2.7.egg/sqlshare/__init__.py", line 197, in upload_chunk
    jsonuploadid = self.post_file_chunk(fn, dataset_name, chunk, force_append, force_column_headers) 
  File "/Library/Python/2.7/site-packages/SQLShare-0.1a-py2.7.egg/sqlshare/__init__.py", line 136, in post_file_chunk
    h.request('POST', selector, body, headers)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 941, in request
    self._send_request(method, url, body, headers)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 975, in _send_request
    self.endheaders(body)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 937, in endheaders
    self._send_output(message_body)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 797, in _send_output
    self.send(msg)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 773, in send
    self.sock.sendall(data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 207, in sendall
    v = self.send(data[count:])
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 176, in send
    v = self._sslobj.write(data)
socket.error: [Errno 32] Broken pipe

[dhalperi]

Hi Steven,

I finally got around to cleaning up some of the Python code. Can you

1) upgrade your copy of the code: git fetch 2) checkout the development branch: git checkout dhalperi-cleanup 3) install the development code: python setup.py install 4) try uploading again using singleupload.py as before?

This will either work and create a single table in SQLShare, or fail spectacularly and not show up in SQLShare.