search

schmittjoh / JMSSerializerBundle

Easily serialize, and deserialize data of any complexity (supports XML, JSON, YAML)
http://jmsyst.com/bundles/JMSSerializerBundle
MIT License
1.8k stars 311 forks source link

Batch processing of large graphs. #172

Open henrikbjorn opened 12 years ago

henrikbjorn commented 12 years ago

I have a need for serializing about 20.000 objects witch each have 7 relations that also need serialized (in total that is a lot). Currently it is not possible to batch process a serialization and that makes it keep the data in memory and the refence aswell so running out of memory will happen quite fast (at about 38000 with 500mb memory_limit).

Is it possible to implement something that makes it more memory efficient by storing the already serialized data in /tmp or have it eagerly return the serialized data (would work well with StreamedResponse).

schmittjoh commented 12 years ago

The actual conversion of the in-memory representation to the actual format is done at the very end (for example for JSON, json_encode is called).

We would need to write our own JSON writer if we want to support streaming. Patches welcome though :)