web.json - add support for tuples

[dumblob]

I think it's necessary to support tuples as well. For the named tuples, it's straightforward. For other tuples, we could use the index number casted to string (so that it looks and behaves similar in JavaScript and Dao when accessing them).

...
[[Error::Type]] --- Invalid type:
Unsupported value type: tuple
Raised by:  serialize(), from namespace "/usr/lib/dao/modules/web/libdao_json.so::json";
...

[Night-walker]

I wonder how you got that runtime error, since recursive typing is supposed to prevent you from calling json.serialize() with anything containing tuple. If it doesn't, it's a bug and you should report it.

Now, it's not intuitive to map Dao tuples to any single JSON type: unnamed tuples are actually plain lists, while named ones are akin to dictionaries. And it's better when parsing what you've just serialized will bring you the original data in the same form, i.e. the conversion is fully reversible.

It is, of course, possible to add marshaling/unmarshaling as I did for XML (Element.map() and Element.extend()), but I am reluctant to do that as working with JSON is simple enough without that.

[dumblob]

The problem I'm facing is that I need only one-way conversion to JSON for several complex data structures composed of several maps, lists, tuples, strings, ints and time structures variously nested into each other. I don't want to write a routine for those myself if there's already a module :)

Why not to add enum<reversible, ireversible> = $reversible parameter to json.serialize()?

[Night-walker]

Why not to add enum<reversible, ireversible> = $reversible parameter to json.serialize()?

And how to determine whether some JSON object should be parsed as tuple or as map? That's simply impossible. If someone writes e.g. a config file by serializing tuples to JSON, he won't be able to read it like that afterwards.

and time structures

And what are you going to do with those then?

Overall, writing a custom serializer seems a more fitting solution to the problem of converting type X to a JSON type. Or you can instead try to map you structures into json::Data manually, if it's more convenient.

BTW, does recursive type check actually work or not?

[dumblob]

And how to determine whether some JSON object should be parsed as tuple or as map?

No determination needed. Always as map.

If someone writes e.g. a config file by serializing tuples to JSON, he won't be able to read it like that afterwards.

JSON is today often used for exporting data through some JSON-API, which is quite often only "readable", i.e. one-way. There is usually no need to map it back.

And what are you going to do with those then?

Export them as string in ISO UTC (e.g. 2014-10-08T20:27:16+00:00).

BTW, does recursive type check actually work or not?

I don't know what you mean by that in this context.

[Night-walker]

BTW, does recursive type check actually work or not? I don't know what you mean by that in this context.

It must not allow to call json.serialize() with anything being or containing a tuple. "Unsupported value type" is a runtime error which you can only get if recursive typing doesn't really work here.

And what are you going to do with those then? Export them as string in ISO UTC (e.g. 2014-10-08T20:27:16+00:00).

Well, you can just as well convert tuples to maps then.

[dumblob]

It must not allow to call json.serialize() with anything being or containing a tuple. "Unsupported value type" is a runtime error which you can only get if recursive typing doesn't really work here.

If course, it doesn't allow it. I've casted it to json::Data.

Well, you can just as well convert tuples to maps then.

That means writing my own serializer from scratch. I wanted to leverage the efficiency of the code in the web.json module and avoid 98% overlap of code base. The clean approach would be to provide an interface for callback (to handle unknown types) or something similar.

[dumblob]

The callback approach is though not much high-level :(

[Night-walker]

It must not allow to call json.serialize() with anything being or containing a tuple. "Unsupported value type" is a runtime error which you can only get if recursive typing doesn't really work here. If course, it doesn't allow it. I've casted it to json::Data.

Then how did you get that error message? :) The code which generates it is now supposed to never be executed.

The clean approach would be to provide an interface for callback (to handle unknown types) or something similar.

That's too hackish; it will require registering type handlers, matching their input parameters to unknown types, supervising their execution, etc.

The callback approach is though not much high-level :(

Yup. In a language with unified type system (e.g. C#, Scala, Ruby, Rust), it would be possible to associate certain serialization function with an arbitrary type and call it directly as a method of that type. Dao emphasizes simplicity, I don't see a simple and comprehensive way to achieve the same.

[dumblob]

I don't see a simple and comprehensive way to achieve the same.

Me neither, except for something like Go uses in their JSON parser IIRC. They use exceptions when the generator doesn't know how to serialize the type/data and one can catch it, serialize it by himself and then continue execution. This should be quite minimalistic to implement also in Dao and is quite powerful in the end. Even though it might not be the fastest/most_efficient solution, for most use cases it should be more than sufficient.

[dumblob]

Then how did you get that error message? :)

load web.json
type Tup = tuple<x:string>
inst: list<Tup> = { ('abc',) }
io.writeln(json.serialize((list<json.Data>)inst))

[Night-walker]

load web.json type Tup = tuple inst: list = { ('abc',) } io.writeln(json.serialize((list)inst))

Then I don't understand anything. If the tuple was successfully converted to list<json::Data>, you apparently wouldn't get that error message. If not, the code couldn't (shouldn't) be executed, so you still are not supposed to get that message. Something is very wrong with typing here. Time to bring this up to @daokoder.

[Night-walker]

I have an idea. I can provide the following thing for web.json:

serialize(data: any)[item: any => json::Data] => list<json::Data>|map<json::Data>

That is, each separate value in data which does not match json::Data type (its non-recursive part) is passed to the user-specified code section which converts it to something the serializer can handle. You can then add your own support for tuples and etc.:

json.serialize(data){
    switch (X) type {
    case tuple<a,b>: return {'a' => X.a, 'b' => X.b}
    case tuple<c,d>: ...
    case tuple: ...
    }
    return 0 # dummy return
}

[daokoder]

I have an idea. I can provide the following thing for web.json:

serialize(data: any)[item: any => json::Data] => list<json::Data>|map<json::Data>]

Sounds like a nice solution.

[Night-walker]

I actually meant

serialize(invar data: any, style: enum<pretty,compact> = $pretty)[invar item: any => Data] => string

It's done, though I did not test it.

[dumblob]

So I've tried to use it for a complex structure, but I get a weird error:

[[ERROR]] in file "/home/test/src/misc/openalt_conference/participants.dao":
  At line 0 : Invalid function definition --- " __main__() ";
  At line 2964 : Invalid virtual machine instruction --- " MCALL:4432,2050,4435 ";
  ** Invalid call --- " calling normal method with code section ";
     Assuming  : routine serialize(data:invar<map<string,Data>|list<Data>>,style=enum<pretty,compact>)=>string;
     Reference : file "/usr/lib/dao/modules/web/libdao_json.so::json";

[Night-walker]

Some kind of typing/overloading bug. You'd better report it separately.

[dumblob]

It seems to work now.

There is though a problem with nested tuples, maps etc. which means that the code section would need to recursively call serialize(){}. This is a serious issue and the only solution I could think of is serialize(invar data: any, style: enum<pretty,compact> = $pretty)[invar item: any => Data|list<any>|map<string, any>] => string (the serialize() method would examine the result of code section and if needed, recursively call the code section for all any members in list<any>|map<string, any>), but I'm not sure about it.

[Night-walker]

I wasn't sure about that when making the enhancement, but you're right. Essentially, the serializer already worked that way, so I just made the typing for code section result more permissive to allow recursive user-defined conversion.

What needs to be checked is the handling of code section result. I just get a pointer from DaoProcess's stack without any copying or GC interaction -- not confident if the value cannot get overwritten when the section is called recursively on the content of that value. Requires investigation or clarification from @daokoder.

[Night-walker]

I was finally able to try it out. It works. Done.