Add the ability to save and load files with intermediate data

[Rembane]

When developing and troubleshooting it is very useful to be able to save the raw data from the web services to file and then load it from that file instead of using the web service. This lets the developers save the broken data on one day and keep working on the problem the next day.

So the workflow I want to implement is:

1. Call the executable using a flag, which saves the intermediate data to files using some good naming scheme.
2. Call the executable with another flag that loads the intermediate data from the files and then runs the parsing as usual.
3. Hack away...
4. Compile and jump to 2.

If it's easy to refactor the existing code so I can just load one file with intermediate data I will implement that, otherwise you'll have to make do with loading all the data.

I will soon create a PR with a prototype of this.

[adamse]

I started on a refactoring which would save the src json (only in error cases) (in the past when json api was the thing we consumed) all through to the frontend so that data could be attached to the github ticket link. I did not finish it.

I think this is reasonable. Maybe my idea above instead of 1, but 2 is definitely necessary.

[adamse]

Could also be a separate executable for 2 since all the parsing lives in the library.

[Jassob]

I agree with @adamse that if we could save the data that we failed to parse by default rather than by having a flag argument for it, that would probably be better. Partly for simplicity's sake, but also because some of our data sources are probably maintained by hand, which means that they won't always be consistent enough with a one-time capture. By capturing every failure and giving it a unique (timestamp-based?) name we give ourselves the possibility to build up our parsing to be resilient enough to parse all variations of our sources.

Plus, by persisting every failure we get natural test cases that we could build our tests around!

[Rembane]

The primary reason for why I want to do this is to have the data I use when developing on my harddrive. With data offline I can do development on saturdays, on aircrafts and without hitting the API every time I recompile.

More concretely: I will do some refactoring on @kirens code to make it use the data flows and data types already present in the other modules, and since I'm the forest there will be considerable latency if I always hit the API, so to make the development experience better it's very nice to have the data locally.

The secondary reason is to have debug data when things go awry, but that's a later issue.

[The1Penguin]

Update on this,

With the merge of #135, we now save the data when the application hits an error. One way to use this is to then create unit test from this test data and hack away