search

gturri / aXMLRPC

A lightweight Java XML-RPC client for the use with Android. Since there is absolutely no dependency to any android specific library (and I assure, that it never will have any), it also works fine in normal Java applications. Read the README.md file for more information.
www.timroes.de
MIT License
145 stars 74 forks source link

Parsing large XML responses #16

Open zemanel opened 12 years ago

zemanel commented 12 years ago

I'm currently not able to parse large endpoint responses due to memory issues (one of the endpoints is a list with 22,746 value entries and no result pagination).

A solution is apparently (as far as i know at the moment) parsing the response using stream parser like SAX (http://developer.android.com/reference/javax/xml/parsers/SAXParser.html). Taking a look at the library, it would be either refactoring or creating a paralel implementation of the ResponseParser and data type Serializer interfaces and implementations (since they are tied to org.w3c.dom.Element).

What do you think?

timroes commented 12 years ago

I think SAX parser would to be preferred in general, since you are reading all the time from network. So I would replace the DOM parser through a SAX parser, if someone (hint hint) would write it :) If you don't want to give it a look, I would give it a look within the next days. Just give me a hint, if you want to look at it, that we don't do the same work twice.

zemanel commented 12 years ago

I've been reading documentation regarding using the SAX parser on Android [compatibility] but i'm unsure on how to refactor the library at the moment, if the current implementation should be replaced or a SAX based parsed added to it for example.

timroes commented 12 years ago

I will change it so SAX parser tomorrow (German time). So if you have around 24 hours time from now, you will get a working SAX version. As it seem (https://github.com/zemanel/pypi-java-xmlrpc-client/tree/develop#implemented-xmlrpc-calls) you still have some other methods, you can implement first :-)

zemanel commented 12 years ago

No rush and i would be glad to help :-) yes actually i was going over that; currently implementing the "search" method call of:

http://wiki.python.org/moin/PyPiXmlRpc

p.s.: the library you mentioned is a component of another project, https://github.com/zemanel/pypi-android , which will i hope will be a good app for tracking Python package releases, someday :-)

timroes commented 12 years ago

For documentation reasons: I profiled your project, and as it seems most of the objects allocations will be there, even with SAX parser. But I will keep that in mind for later, and might give it a try some time, but not now, as I see no large chance, it will help the problem.