thinkle
commented
9 years ago Ok -- tracked down this bug too. Nothing to do w/ google's JSON -- looks like an update in the code changed the behavior of get_file_content and broke this. I changed the behavior again and I believe I've updated both json and non-json paths to fix. Note, I also fixed an encoding error while I was at it.
Here's a patch that makes 2 fixes: change implementation of get_by_format and how it's called fix unicode error
diff -c /tmp/get.py /usr/local/lib/python2.7/dist-packages/gdcmdtools/get.py *\ /tmp/get.py 2015-10-14 21:44:17.623913045 -0400 --- /usr/local/lib/python2.7/dist-packages/gdcmdtools/get.py 2015-10-14 22:10:52.448583796 -0400
* 67,73 ** file_fullname = "%s.%s" % (file_name, file_ext)
with open(file_fullname, 'wb+') as f:! f.write(file_source)
j.pop("source")
new_json["files"].append(j)--- 67,73 ---- file_fullname = "%s.%s" % (file_name, file_ext)
with open(file_fullname, 'wb+') as f:! f.write(file_source.encode('utf8')) # We need unicode!
j.pop("source")
new_json["files"].append(j)* 102,112 ** self.save_as = title
if self.format == "json":! file_content = self.get_by_format(return_format[self.format])
self.parse_gas_json(file_content, self.save_as)
else:
FIXME: handle return value
! result, local_size = self.get_by_format(self.save_as, return_format[self.format]) if( result == False ): raise Exception("File size check failed, download may be incompleted. local size is %d" % local_size)
--- 102,113 ---- self.save_as = title
if self.format == "json":! result,file_content,local_size = self.get_by_format(self.save_as,return_format[self.format])
! print 'file_content=>',file_content
self.parse_gas_json(file_content, self.save_as)
else:
FIXME: handle return value
! result,content,local_size = self.get_by_format(self.save_as, return_format[self.format]) if( result == False ): raise Exception("File size check failed, download may be incompleted. local size is %d" % local_size)
* 188,197 **
if self.file_size:
if( int(self.file_size) == local_size ):! return True, local_size print "File size: %d" % local_size else: ! return False, local_size else: print "File size in bytes: %d" % local_size ! return True, local_size --- 189,198 ----
if self.file_size:
if( int(self.file_size) == local_size ):! return True, response.content, local_size print "File size: %d" % local_size else: ! return False, response.content, local_size else: print "File size in bytes: %d" % local_size ! return True, response.content, local_size
Diff finished. Wed Oct 14 22:11:11 2015
After applying my patch to fix the issue downloading JSON files, I was able to successfully pull the json file, but then it looks like gdget.py runs into trouble parsing google's files.
Here's an example with an extremely simple test project I created: File location: Simple.json
File size in bytes: 134 Traceback (most recent call last): File "/usr/local/bin/gdget.py", line 65, in
result = get.run()
File "/usr/local/lib/python2.7/dist-packages/gdcmdtools/get.py", line 106, in run
self.parse_gas_json(file_content, self.save_as)
File "/usr/local/lib/python2.7/dist-packages/gdcmdtools/get.py", line 54, in parse_gas_json jsons = json.loads(file_content) File "/usr/lib/python2.7/json/init.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) TypeError: expected string or buffer
And here is the the JSON it's barfing on:
{"files":[{"id":"5d71cce7-1c5b-4cc6-a292-7f04e6251059","name":"Code","type":"server_js","source":"function myFunction() {\n \n}\n"}]}