search

Crystal03 / google-docs-fs

Automatically exported from code.google.com/p/google-docs-fs
GNU General Public License v2.0
0 stars 0 forks source link

Request -URI Too Large #31

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. Mount a Google Apps account with a large number of folder documents.

What is the expected output? What do you see instead?
I would expect the system to mount, instead I receive:

FUSE library version: 2.8.1
nullpath_ok: 0
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.13
flags=0x0000007b
max_readahead=0x00020000
   INIT: 7.12
   flags=0x00000011
   max_readahead=0x00020000
   max_write=0x00020000
   unique: 1, success, outsize: 40
unique: 2, opcode: LOOKUP (1), nodeid: 1, insize: 47
LOOKUP /.Trash
getattr /.Trash
   unique: 2, error: -2 (No such file or directory), outsize: 16
unique: 3, opcode: LOOKUP (1), nodeid: 1, insize: 52
unique: 4, opcode: OPENDIR (27), nodeid: 1, insize: 48
   unique: 4, success, outsize: 32
LOOKUP /.Trash-1000
getattr /.Trash-1000
   unique: 3, error: -2 (No such file or directory), outsize: 16
unique: 5, opcode: LOOKUP (1), nodeid: 1, insize: 45
LOOKUP /BDMV
getattr /BDMV
   unique: 5, error: -2 (No such file or directory), outsize: 16
unique: 6, opcode: LOOKUP (1), nodeid: 1, insize: 57
LOOKUP /.xdg-volume-info
getattr /.xdg-volume-info
   unique: 6, error: -2 (No such file or directory), outsize: 16
unique: 7, opcode: LOOKUP (1), nodeid: 1, insize: 52
LOOKUP /autorun.inf
getattr /autorun.inf
   unique: 7, error: -2 (No such file or directory), outsize: 16
unique: 8, opcode: READDIR (28), nodeid: 1, insize: 80
readdir[0] from 0
Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/googledocsfs/gFile.py", line 164, in readdir
    feed = self.gn.get_docs(filetypes = excludes)
  File "/usr/local/lib/python2.6/dist-packages/googledocsfs/gNet.py", line 61, in get_docs
    return self.gd_client.Query(query.ToUri())
  File "/usr/local/lib/python2.6/dist-packages/gdata/docs/service.py", line 218, in Query
    return self.Get(uri, converter=converter)
  File "/usr/local/lib/python2.6/dist-packages/gdata/service.py", line 1108, in Get
    'reason': server_response.reason, 'body': result_body}
gdata.service.RequestError: {'status': 414, 'body': '\n\n<html><head>\n<meta 
http-equiv="content-type" content="text/html;charset=utf-8">\n<title>414 
Request-URI Too Large</title>

What version of the product are you using? On what operating system?
Ubuntu 10.04, google-docs-fs-1.0rc1

Please provide any additional information below.

I do have a very large number of documents and folder so I'm not sure if it is 
one or both of these issues causing the URI to be too large. I do have more 
debug information but it contains sensitive data in the file/folder names so 
would need to obfuscate before sending.

Original issue reported on code.google.com by phil%lit...@gtempaccount.com on 20 Jun 2010 at 7:36

GoogleCodeExporter commented 9 years ago 
Hi, sorry for the delay in answering.
I think you're right in that it is a problem with the number of documents. 
Version 3.0 might improve things, but I don't want to rely on that library 
until it has been finalised from their Labs.

In the meantime, could you try following the instructions here:
http://code.google.com/apis/documents/docs/1.0/developers_guide_python.html#List
Docs
and see if this PrintFeed function gives you the same problem. Note, you need 
to Authenticate, with the link given in that article.

If you still have the problems there, then it will at least confirm that the 
problem lies in Google's APIs and will allow me to open a ticket on their Issue 
tracker and hopefully get a fix.

Cheers,
Scott W

Original comment by d38dm8nw81k1ng@gmail.com on 20 Jul 2010 at 11:20

GoogleCodeExporter commented 9 years ago 
I created a test.py with the following:

import gdata.docs
import gdata.docs.service

gd_client = gdata.docs.service.DocsService(source='domain.com')

gd_client.ClientLogin('user@domain.com', 'password')

def PrintFeed(feed):
  """Prints out the contents of a feed to the console."""
  print '\n'
  if not feed.entry:
    print 'No entries in feed.\n'
  for entry in feed.entry:
    print '%s %s %s' % (entry.title.text.encode('UTF-8'), entry.GetDocumentType(), entry.resourceId.text)

feed = gd_client.GetDocumentListFeed()
PrintFeed(feed)

PrintFeed function worked fine, no errors and got the full list, no folders but 
I guess that is expected?

Hope that's helped.

Phil.

Original comment by phil%lit...@gtempaccount.com on 20 Jul 2010 at 11:45

GoogleCodeExporter commented 9 years ago 
Yeah, PrintFeed doesn't print folders. It's possible that the folders are what 
is causing the problem. If you could use a query to show all the folders we 
might be able to get to the bottom of this.

The following code I've just ripped from google-docs-fs:

query = gdata.docs.service.DocumentQuery(categories = ['folder'])
query['showfolders'] = 'true'

Just pass that query into PrintFeed and you should be good to go.

Original comment by d38dm8nw81k1ng@gmail.com on 17 Aug 2010 at 9:23

GoogleCodeExporter commented 9 years ago 
I've changed the end of the script to now have:

query = gdata.docs.service.DocumentQuery(categories = ['folder'])
query['showfolders'] = 'true'
feed = gd_client.Query(query.ToUri())
PrintFeed(feed)

That seems to be printing out a list of folders. The list does seem to break 
with the following error:

Traceback (most recent call last):
  File "test.py", line 22, in <module>
    PrintFeed(feed)
  File "test.py", line 14, in PrintFeed
    print '%s %s %s' % (entry.title.text.encode('UTF-8'), entry.GetDocumentType(), entry.resourceId.text)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal 
not in range(128)

I'm not sure if this is a separate issue or not but thought it was worth 
mentioning.

Thanks.

Original comment by phil%lit...@gtempaccount.com on 28 Aug 2010 at 10:59

GoogleCodeExporter commented 9 years ago 
The UnicodeDecodeError isn't a problem since it doesn't occur within 
google-docs-fs; it's just somewhat annoying when it comes to testing.

Try changing the .encode('UTF-8') to .decode('UTF-8') and see if that gives you 
any joy.

Original comment by d38dm8nw81k1ng@gmail.com on 1 Sep 2010 at 11:16

GoogleCodeExporter commented 9 years ago 
That's cracked it. So would I be correct in assuming we just need to wait for 
version 3.0?

Original comment by phil%lit...@gtempaccount.com on 1 Sep 2010 at 11:51

GoogleCodeExporter commented 9 years ago 
So you are able to print all files and folders with PrintFeed?

I think I know why it might be happening. To print the root folder, 
google-docs-fs gets a list of all folders and then asks for files that are NOT 
in those folders. If you have enough folders, this could be causing the request 
URI to be too big to be parsed by the server.

A workaround could be to browse to line 164 in 
/usr/local/lib/python2.6/dist-packages/googledocsfs/gFile.py and, above that 
line, enter `excludes = []` at the same indent level and it should work, though 
it will list all your files and folders, regardless of whether they have a 
"parent" folder or not.

Original comment by d38dm8nw81k1ng@gmail.com on 1 Sep 2010 at 12:01

GoogleCodeExporter commented 9 years ago 
Sorry, spoke too soon, with decode instead of encode I get:

Traceback (most recent call last):
  File "gtest.py", line 17, in <module>
    PrintFeed(feed)
  File "gtest.py", line 14, in PrintFeed
    print '%s %s %s' % (entry.title.text.decode('UTF-8'), entry.GetDocumentType(), entry.resourceId.text)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 
10: ordinal not in range(128)

Sorry, missed stderr :/

Original comment by phil%lit...@gtempaccount.com on 1 Sep 2010 at 12:23

GoogleCodeExporter commented 9 years ago 
=/ I can never remember which to use (decode or encode), but I don't think it 
matters too much. Try having a go at that workaround I posted above:
Open /usr/local/lib/python2.6/dist-packages/googledocsfs/gFile.py and before 
line 164 put the following code:
{{{
excludes = []
}}}
and see what happens.

Original comment by d38dm8nw81k1ng@gmail.com on 1 Sep 2010 at 3:06

GoogleCodeExporter commented 9 years ago 
Ok, ignore the {{{ }}}, just enter the text in between. I forgot this doesn't 
use Wiki syntax.

Original comment by d38dm8nw81k1ng@gmail.com on 1 Sep 2010 at 3:07

GoogleCodeExporter commented 9 years ago 
Hi, same problem here (URI too large). We've had big trouble with Google 
because after deleting an ex-employees account about six momnths ago. All 
document this ex-employee owned were completely lost without any warning. 
Google is aware of this issue but no changes since then.

To overcome this issue we would like to mount our Google Docs into some Linux 
box and backup the documents. Thus it would be great to have the google-docs-fs 
tool working with a large amount of documents.

Original comment by floesc...@gmail.com on 28 Jan 2011 at 9:03

GoogleCodeExporter commented 9 years ago 
I've just had a thought. Do you have a large number of folders? It could be 
that the file system is telling Google Docs to exclude a large list of folders, 
which is too long for the server to parse.

Original comment by d38dm8nw81k1ng@gmail.com on 28 Jan 2011 at 9:50