search

wanglingsong / JsonSurfer

A streaming JsonPath processor in Java
MIT License
294 stars 55 forks source link

Supply an OutputStream to a SurfingConfiguration #50

Open arenger opened 5 years ago

arenger commented 5 years ago

First: Good job with JsonSurfer! It's a helpful tool, for sure.

I am looking for a way to read from a large JSON file in a streaming fashion and write an excerpt from that file to a new file in a streaming fashion. The onValue method of the JsonPathListener receives an Object which was obtained in a streaming manner. This is great, but at that point the full Object is loaded in memory. Would it be possible to specify an OutputStream to a SurfingConfiguration, such that the matching portions of the incoming file could be written to the stream instead of memory?

Actually, I really need a different OutputStream for every excerpt that is matched by the specified JSON Path. Perhaps a new SurfingConfiguration.setStreamHandler method would receive a new OutputStreamHandler interface that two methods: OutputStream onNewOutput(ParsingContext context) and void end(OutputStream out)... or something. There are a few ways to do it, I'm sure.

Maybe there is already support for something like this. I looked but couldn't find any.

wanglingsong commented 5 years ago

This feature would require modification to jsurfer-core, probably need to implement a new kind of JsonCollector which to be inserted into the dispatcher. Tips here: https://github.com/jsurfer/JsonSurfer/blob/master/jsurfer-core/src/main/java/org/jsfr/json/SurfingContext.java#L79

If you are interested, feel free to contribute. I'm currently working another issue regarding JsonPath filter during my limited free time

arenger commented 5 years ago

I certainly understand the limited free time. No problem. This issue isn't urgent, currently, but if it becomes urgent I might look into contributing to the project. Thanks.