Allow the Dynamic XML DS to support JSON

[brendo]

A lot, a lot, of API's these days expose their API as JSON, not XML (or in addition too). Symphony needs to keep with the times, so it should be able to read JSON just as it was XML. JSON, thankfully, has a strict syntax and there is a couple of classes floating around the forums (and the net) about transforming JSON to XML, so I don't think this would be too difficult to implement.

To be clear, the JSON would be read in and transformed to XML for use in XSLT.

Things to that need to be decided:

• Do we bundle this into the Dynamic DS and rename it, or do we make a JSON Datasource? External DS, Remote DS spring to mind.
• Does a user have to specify if it's JSON or XML? We should be able to read the Content-Type header and proceed appropriately
• The interface might need a 'ping' ability so that it can determine if the URL is valid, and if it needs to display namespace information.
• The XML Importer 'discover namespaces' feature is useful, and as namespaces scare the hell out of people, I think this feature should be implemented
• Caching can be remain, but it should probably be exposed so you can see how long the cache has left and flush it (aka Cacheable Datasource)

/cc @nickdunn @allen @rowan-lewis @creativedutchmen @eKoEs @nilshoerrmann @czheng

[designermonkey]

Personal opinions are listed:

• We should have separate separate options, to keep it obvious.
• I agree about the 'ping', many a time I've added datasources only to find they don't respond, either due to poor server protocols on my side, or their's.
• Totally agree about the namespaces discovery.
• Caching has always been my biggest annoyance with the current setup. How do we flush it at present? It definitely needs to be in the UI for all cache-able datasource types.

Some other thoughts:

While we're at it, could we add a 'Local File Datasource' too? Thinking on those lines, we should have just two datasource types, JSON or XML (and maybe CSV, the import/export extension is cool, but what about regularly changing files? It has been questioned on the forum recently) and then under these be able to choose the method of aquisition, being External, Local or Direct Input.

[allen]

Functionally, fetching from XML or JSON have two key differences:

XML requires namespace declaration. JSON requires conversion to XML. Aside from that, the method in which data is fetched, cached and delivered to the frontend XML are the same. We could look at something similar to JIT recipes in which fetched data goes through a 'pipe' i.e. XML adds namespace declaration. JSON adds XML conversion. This way more 'pipes' can be added in the future should the need arise.

• Given the aforementioned scenario, I'd choose to have a single DS type. I like the sound of "Remote DS". Dynamic DS always seemed wrong to me. All DSs are dynamic. In fact, Symphony's internal DSs are more dynamic since it doesn't include any caching mechanisms by default.
• I also like the idea of a ping test. Ping test should respond with status code and response time. I've been caught many times fetching from a local Symphony page but forgetting the page is set to maintenance mode so no results were returned.
• At one stage, dynamic DSs had namespace discovery. I wasn't aware this was no longer the case!
• Agree with cache flushing. I've had to add cache purge hooks during development in the past which was tedious.

[allen]

Thinking about it some more, I worry adding a piping mechanism is complicating things a little. Event has filters, JIT has recipes. Adding Remote DS with pipes seem like we've got too many different tertiary hooks to keep track of.

I'm okay with the idea of having two separate DS types, Remote XML DS and Remote JSON DS if that simplify things.

[brendo]

I like the idea of two Datasource types as well, it's more explicit, makes instant sense to a user ('ah ha, JSON, I'll click that ') and it will be easier to manage, test and debug. While the ping and cache functionality is shared (or will be), I think there could be a neat way to make the code reusable instead of duplicating it.

[nilshoerrmann]

If this is planned for 2.3 (I really like the idea of having "Remote Datasources" that are capable to fetch JSON as well), shouldn't we have a more general look at the data source editor? There are already a few other issues posted regarding the DS editor and maybe 2.3 could really push a few things forward?

[simoneeconomo]

I'm okay with the idea of having two separate DS types, Remote XML DS and Remote JSON DS if that simplify things.

Superb! I also like the concept of "Remote DS" :)

Question: will there be a Static JSON DS type?

[designermonkey]

I think there should be. XML and JSON are both widely available data format, and should be equally represented now and in the future for us.

[allen]

What would the use case be for static JSON DS though?

Static XML DS is a really simple type. It simply assigns an XML string that a user defines through the DS's editor into a variable which is injected into the frontend XML. I take it that Static JSON DS would behave the same but with an added processing requirement for JSON -> XML conversion. If that's the case, wouldn't a user simply convert JSON to XML first, then put that into the Static XML DS?

If a user needs a convenient or on-they-fly way to convert JSON to XML, then Remote JSON DS would do this. Say that you have a directory full of JSON files that you want to convert to XML and included in your frontent XML. In this instance Static XML DS wouldn't solve your problem since you need to customise the DS or manually update the DS editor's textarea with your code. This is where the Remote JSON DS would come in.

The Static XML DS was conceived as a way to "easily add some arbitrary XML into the frontend". The idea of a Static JSON DS that needs to be converted into XML then injected into the frontend seem to defeat the purpose.

[klaftertief]

Having two different DS types in the DS editor sounds good to me, too. How it's handled code-wise is another question. And "Remote DS" feels like a very good name.

[designermonkey]

Ok.

[brendo]

There are already a few other issues posted regarding the DS editor and maybe 2.3 could really push a few things forward?

I thought that was the point of the Realign :)

I plan to start work on this ASAP in another branch, at least sorting out the technical side of things anyway

[nilshoerrmann]

I thought that was the point of the Realign :)

Well, we started on the premise "Only cosmetics!" ... When we start working on JSON integration we should also have a look at negation and filter options in general ("Field X is not of the value of {$y}").

[simoneeconomo]

I thought that was the point of the Realign :)

Am I missing something? I thought it was only cosmetics, as Nils said. ;)

[brendo]

Changing the UI of the Datasource Editor is cosmetic, adding something like negation is not IMO. That's more a field's responsibility to implement the filtering, unless your taking about having a better selector instead of 'mode: {value}' in which I guess it's related.

I should clarify what I meant too, I see the realign not as just a 'skin', but a collaborative effort to improve the UI and workflow in general, things like breadcrumbs, active navigation states, save and return, implementing something like EDUI etc.

Applying a new skin only seems superfluous to me as users can do that already with the Admin Override CSS extension, and I feel we need to push Symphony forward to be more intuitive so it's users will love it even more :D

That said, we've gotta be careful not to put too much work on ourselves, we are spread pretty thin as it is and 2.3 must ship before Christmas, so careful planning and scope is the key!

[brendo]

http://goessner.net/articles/JsonPath/ interesting, wonder if we try to support this instead of converting to XML?

edit: looks meh on closer inspection

[cz]

Not sure how supporting JSON selectors could work, but see also JSONSelect if you're interested.

[brendo]

Just the idea of not converting the JSON to XML from the get go. The jsonpath would allow the dev to set the included elements, then the result of that would be converted. You're right though, it doesn't add a whole lot.

[nilshoerrmann]

I don't think something like this should be added until it's really standadised or at least has become an inofficial standard throughout the business.

[brendo]

Summary of where we are at:

• Remote JSON datasource is implemented and works
• XPath is used to select the data you want
• Uses a JSON to XML class to transform the request result into XML before applying the XPath

Few things to note:

• JSON supports 2011-08-20 02:00:00 as a key name, XML does not. If this happens the element will be called key
• All elements are handlised, ie. as_of becomes as-of
• All elements have handle and value attributes.
• JSON supports the notion of null and boolean, should we take this any further? (I think not)

Or if you'd prefer to see it for yourself, this gist shows a sample Twitter feed being consumed.

[michael-e]

@brendo: The json2xml class seems rather advanced, so I will test it in an extension I am building. That's why I thought it might be good idea to put this class in Symphony's toolkit folder (so extensions are "invited" to use it).

What do you think?

[michael-e]

Maybe one might even build two new helper classes, ArrayToXML and JSONToXML (the latter using the first one, since JSON data is converted to an array before being processed).

[brendo]

The above commit:

• Removes the Remote JSON datasource type
• Adds a Format field to the Dynamic XML DS to choose whether the URL is XML/JSON
• Simplifies the logic for invalidating a cached result in the Dynamic XML class.
• Adds Accept: headers for XML/JSON to the Dynamic XML Gateway connectivity.
• Abstracts the JSONToXML functionality to a JSON class, JSON::convertToXML. (@michael-e The General class has an existing array_to_xml method)
• Adds an .offset CSS class to use in conjunction with .group. This makes a 75%/25% split.
• Moves CONTENT/content.resourcespage.php to TOOLKIT/class.resourcespage.php (unrelated sorry).

[creativedutchmen]

@brendo you seem to have undone all your hard work in this commit: 6cc73ab703b1ab4d6cb9fbc6259590f4f6555411. Is this an accident, or wasn't JSON such a great idea after all?

[creativedutchmen]

Aha! The code is moved from the core to an extension. God I hate to be on the cutting edge..

[designermonkey]

:o)