search

CHJani / google-api-java-client

Automatically exported from code.google.com/p/google-api-java-client
0 stars 0 forks source link

Ability to stream parse a feed of entries #333

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
External references, such as a standards document, or specification?

n/a

Java environments (e.g. Java 6, Android 2.3, App Engine, or All)?

All, but particularly relevant for low-memory devices like Android

Please describe the feature requested.

We need an ability in the generated service-specific libraries to parse a feed 
of entries as a stream, without having to load the whole feed into memory which 
may cause an OutOfMemoryError on low-memory devices.

Here is a code snippet that Alon Albert wrote that demonstrates one possible 
way this can be accomplished today:

          while (true) {
                final JsonParser parser = JsonHttpParser.parserForResponse(jsonFactory, response);
                parser.nextToken();
                JsonToken token = parser.getCurrentToken();
                String nextPageToken = null;
                while (token == JsonToken.FIELD_NAME) {
                    String key = parser.getText();
                    parser.nextToken();
                    if (key.equals("nextPageToken")) {
                        nextPageToken = parser.parse(String.class, null);
                    } else if (key.equals("items")) {
                        while (true) {
                            // skip START_ARRAY/END_OBJECT
                            parser.nextToken();
                            if (parser.getCurrentToken() == JsonToken.END_ARRAY) {
                                break;
                            }
                            final Entry entry = parser.parse(Entry.class, null);
                            System.out.println(entry);
                        }
                    } else {

                        parser.skipChildren();
                    }
                    token = parser.nextToken();
                }
                if (nextPageToken == null) {
                    break;
                }
                response = builder.setPageToken(nextPageToken).executeUnparsed();
            }

Ideally we would add support that would support the that Alon needs here, but 
that wouldn't require him to write any JsonParser low-level parsing code.

I tried to implement something like this long ago with JsonFeedParser:

http://javadoc.google-api-java-client.googlecode.com/hg/1.5.1-beta/com/google/ap
i/client/googleapis/json/JsonFeedParser.html

but for various reasons that doesn't do what we want.  So either we fix up 
JsonFeedParser to work better, or try a different approach.  For example, we 
could try Iterator<Entry> for accessing the entries, though keep in mind we 
still need the ability the ability to parse the feed metadata.

Finally, we need to integrate this into the generated service-specific 
libraries, so this is easy to use and discover.

Original issue reported on code.google.com by yan...@google.com on 28 Oct 2011 at 8:16

GoogleCodeExporter commented 9 years ago

Original comment by rmis...@google.com on 20 Jan 2012 at 4:30

GoogleCodeExporter commented 9 years ago

Original comment by yan...@google.com on 16 Feb 2012 at 2:49

GoogleCodeExporter commented 9 years ago

Original comment by yan...@google.com on 27 Mar 2012 at 2:10

GoogleCodeExporter commented 9 years ago

Original comment by rmis...@google.com on 16 May 2012 at 1:24

GoogleCodeExporter commented 9 years ago

Original comment by yan...@google.com on 30 May 2012 at 10:04

GoogleCodeExporter commented 9 years ago

Original comment by yan...@google.com on 2 Aug 2012 at 2:34

GoogleCodeExporter commented 9 years ago

Original comment by yan...@google.com on 26 Sep 2012 at 12:35

GoogleCodeExporter commented 9 years ago 
This has the potential to have a significant improvement to memory and speed of 
JSON parsing, especially on older Android devices.

Original comment by yan...@google.com on 24 Oct 2012 at 5:25

GoogleCodeExporter commented 9 years ago

Original comment by yan...@google.com on 1 Dec 2012 at 5:27

GoogleCodeExporter commented 9 years ago

Original comment by yan...@google.com on 10 Dec 2012 at 2:47

GoogleCodeExporter commented 9 years ago

Original comment by yan...@google.com on 19 Jan 2013 at 8:16

GoogleCodeExporter commented 9 years ago

Original comment by yan...@google.com on 7 Feb 2013 at 12:05

GoogleCodeExporter commented 9 years ago

Original comment by yan...@google.com on 10 Jun 2013 at 1:24

GoogleCodeExporter commented 9 years ago

Original comment by yan...@google.com on 26 Jul 2013 at 10:19

GoogleCodeExporter commented 9 years ago

Original comment by yan...@google.com on 27 Sep 2013 at 12:00