Reading/Writing JSON file

[agrignard]

It would be great if gama can deal with JSON file as it's getting more and more used.

[AlexisDrogoul]

This has to be included somehow in the serialization plugin, so as to be able to serialize any GAML object into a JSON string and read JSON strings as maps. I add @benoitgaudou as a participant.

[AlexisDrogoul]

Note that json-simple is already included in GeoTools. I have exported the packages so it can be used anywhere.

[agrignard]

What do you mean by it can be used anywhere?

What I 'd like to do is to read/write a Json file from GAML.

I can spend some time on implementing the basic Json feature in term of read and write but I don't remember exactly remmeber the way to handle in a cleaa way file in GAMA.

I'd like a simple API in GAML that does stuff like that: https://processing.org/reference/loadJSONObject_.html and this https://processing.org/reference/saveJSONObject_.html

[AlexisDrogoul]

Hi, I've played 20 minutes with JSON Simple, already included in GeoTools. I've implemented the JSONAware interface in IValue, declared the only method needed in the 16 classes that implement IValue by calling serialize(true) and it works well to output strings in JSON format. Inputting is a bit more complex as there is no deserialization happening (so you only get strings). Probably we'll need some process in IType to handle this.

The problem of JSON Simple is that it is probably too ... simple for what we want. But maybe it is the way to go to allow the maximal flexibility for the user. Lists and maps are outputted as JSON arrays and maps and more exotic values (like geometries) are outputted as genuine GAML expression (i.e. "{0.0, 0.0} as geometry" for instance}, so it is possible to use eval_gaml on them, plus a casting, to recreate the correct object.

If you already know what is inside your files (which is normally the case) and only input/output simple values or handmade values (like maps or lists that you build yourself), then it might be a good idea to implement this. It is a matter of hours. Then, you'll be able to write things like:

file f <- json_file("trial.json");
// f will wrap a container, which will probably be always a list, even if the json file only contains 1 "object".
map object <- map(f[0]); // if it is a map
write object.keys;
object["new_key"] <- 1000;
save object to: "trial.json" type: json;

[agrignard]

Good news.

Well done. When I was speaking about JSON format I was not thinking about the serialization (even if it's a good idea) I was speaking a format for any input/output to initialize a simualtion and/or save any information. A kind of .csv but more "modern" .

For instance I'd like to initialize a grid with that kind of input https://github.com/ChangingPlaces/DataTemplates/blob/master/template.json

The API in gaml would look like this: https://processing.org/reference/loadJSONObject_.html and this https://processing.org/reference/saveJSONObject_.html

[agrignard]

I am ok to implement that, I am just a bit rusted in GAMA I just don't know where to start to make it clean?

Any chance to have an empty nut shell and then I take care of the different implementation usefull for reading and writing?

This is typically the type of JSON that I want to use as an input to instanciate. The strucutre is not so complicated it's only a grid at the end.

http://cityscope.media.mit.edu/BRT.json

[AlexisDrogoul]

I guess the best way would be (at first) to create a new file type (json), so as to be able to load and parse them without changing too many things around. To write json files, as we have still not implemented the file infrastructure completely (cf. #1362 ), the best thing would be to enrich the save statement (I guess we will have to modularize it like create, but not now).

[agrignard]

So I'll start by adding a GamaJSONFile.java and iterate through that. I'll keep you update

[agrignard]

Actually as the structure of the JSON cannot be known by default I am wondering if we shoudl not think about another way to deal with those file (not a JSONFile.java)

Is there a way to have more or less the same syntax as in JAVA but in GAML like this: ////////////////////// SYNTAX 1 JSONParser parser = new JSONParser();

    try {     
        Object obj = parser.parse(new FileReader("c:\\file.json"));

        JSONObject jsonObject =  (JSONObject) obj;

        String name = (String) jsonObject.get("name");
        System.out.println(name);

        String city = (String) jsonObject.get("city");
        System.out.println(city);

        String job = (String) jsonObject.get("job");
        System.out.println(job);

        // loop array
        JSONArray cars = (JSONArray) jsonObject.get("cars");
        Iterator<String> iterator = cars.iterator();
        while (iterator.hasNext()) {
         System.out.println(iterator.next());
        }

////////////////////////////////////////////////

///////////////////SYNTAX 2 //////////// JSONArray a = (JSONArray) parser.parse(new FileReader("c:\exer4-courses.json"));

for (Object o : a) { JSONObject person = (JSONObject) o;

String name = (String) person.get("name");
System.out.println(name);

String city = (String) person.get("city");
System.out.println(city);

String job = (String) person.get("job");
System.out.println(job);

JSONArray cars = (JSONArray) jsonObject.get("cars");

for (Object c : cars)
{
  System.out.println(c+"");
}

}

[hqnghi88]

hi Arnaud, In the fillBUffer of xxxFileType, you can parse, assign, manipulate the json structure. And return a gamamap, so in GAMA, user just need to write:

file my_json <- file("xxx.json"); write my_json["name"]; write my_json["job"];

I did the same thing for netcdf because the structure of netcdf is a bit like json, but it contains multi-multi-dimension array https://github.com/gama-platform/gama.experimental/blob/master/cict.gaml.extensions.netcdf/src/cict/gaml/extensions/netcdf/file/NetCDFFile.java

[AlexisDrogoul]

The JSON format is a mess (although it is less a mess than the CVS one). There is effectively no way to tell in advance what can be inside as there is no proper metadata passed with it.

That said, JSON files can always be parsed as maps of maps of maps, etc.

So I've wrapped together a quick & dirty GamaJsonFile class to get you started. It reads JSON files and converts them to a GamaMap. Either the direct object denoted by the JSON contents, or (if the file contains a list) a map with only one key ("contents") and a value with this list.

You can use it like all the other GamaFile instances, e.g.:

file fff <- json_file("BRT.json");
map<string, unknown> c <- fff.contents;
// and after that you use it like a normal map
        list<map<string, int>> cells <- c["grid"];
        loop mm over: cells {
            write mm["rot"];
        }
... etc ... 

[AlexisDrogoul]

Regarding the previous commit, I did it in ~10 minutes, so dont expect too much, but it should work for reading JSON files. Later, we can see how to save (save statement) and read (create statement) agents automatically from JSON files, but we'll probably need to rely on the serialize plugin for that.

[AlexisDrogoul]

And, of course, you can install without problems the free JSON editor (http://boothen.github.io/Json-Eclipse-Plugin/) directly in GAMA (I've tested it). Might be worth to give it a look, by the way, as it seems to have a very robust parser.

[AlexisDrogoul]

OK. Thanks to this simple implementation (and because I didn't want to spend too much time on it, also), I have decided to open the way for completely solving #1362 . The save statement now accepts a simple file parameter (e.g. save json_file("copy.json", my_map);) and uses the flushBuffer() method of the corresponding GamaFile.

For other types of files, the corresponding method needs to be written, but I did it for GamaJsonFile and it works well. For instance:

file fff <- json_file("BRT.json");
map<string, unknown> c <- fff.contents;
c["new_key"] <- 0;
c["other_key"] <- [0,0,0];
json_file copy <- json_file("Copy.json", c);
save copy;

of course, any map (not necessarily read from a JSON file) will do the trick.

Consider just that the JSON format is a bit primitive and that complex types (outside of Map and List) will not be converted correctly. So you need to first convert them to either int, float, boolean, string, map or list.

I close this thread -- please reopen if there are any issues related to JSON specifically. For issues related to the saving of the file (like overwriting or not, etc.), which are not resolved yet (it needs to be put in the save statement) but which are more general, please consider completing the thread #1362.

[agrignard]

Perfect. I test it. That's exactly what I need for the reading!!

Gama is at the Gateway of the MediaLab!

[agrignard]

I made a simple use case https://github.com/gama-platform/gama/commit/6207bb40a2082467ccb96021c08bc0a52429004d