Upgrade to the next data format

[hugoroy]

Hi,

We're working on the v1 for the ToS;DR data format. I'm working currently inside the dataspec branch https://github.com/tosdr/tosdr.org/tree/dataspec and I am documenting the specs in https://github.com/tosdr/tosdr.org/wiki/_pages

I suppose all browser extensions will have to be updated to these changes. We will merge dataspec into master once you're ready. :-)

@AbdullahDiaa @e-user @shybyte @xMartin

[Can you confirm you've seen this, will you have time to do it in June?]

Thanks, Hugo

[xMartin]

The data handling of the Safari extension is heavily "inspired" by the code of the Chrome extension.

Anyway, maybe it's a good chance to create a little library to be shared by the extensions.

[hugoroy]

Hm. I noticed that I forgot to tell you about one change that already made it to master. Sorry for that mistake. Can you already update https://github.com/tosdr/tosdr.org/wiki/Specification:-points#tosdr--point ? Thanks

@xMartin : I agree. This is what the "browser-extensions" repo is for.

[shybyte]

The old and new versions should be online at the same time (in working, updated versions), in order to ease development and have an smooth transition. Or can we already use e.g. https://raw.github.com/tosdr/tosdr.org/dataspec/index/services.json as the new data source?

@xMartin a common little library can easily cause a lot of communication overhead (e.g. hanging around in issue trackers and online chats) which causes in the end more work than just hacking independently and getting "inspired". On the other hand we both live in Berlin (?), so we could meet for a micro hackathon afternoon or so to get this done at least for chrome/safari.

@hugoroy is there anything urgent to do in order to care for your mentioned change in the master?

[shybyte]

Another question: Will there be an index which contains the URLs (to match against) and the rating (or (if there is no rating) to the information, that there are some points to show). This information is important to show the rating icons if the user visit a page. Currently this information must be loaded, by loading all service files, which is not really scalable. Generally the current API is not really made for the need of a plugin which wants to be efficient.

@xMartin @hugoroy Maybe we should do a hackathon about that in order to make the new version better. If the API would be better suited for the plugins, there would be less need for a common library (which would be currently partly a workaround for the cumbersome API)

[xMartin]

Good point. Let's fix the API.

[hugoroy]

AFAIK there should not be any difference between the dataspec and master branches regarding index/services.json

The differences will be in the structure of services/json points/json, and topics/*json files.

But anything to ease your development is welcome indeed, so I guess you could already do that.

Well, the dataspec branch is not ready yet. I'm waiting for some fixes in buildIndexes.js (cc @michielbdejong ). So that's what's needed before we can merge dataspec in master. Otherwise, we just need to make sure the web extensions work fine with the new format before we merge.

One urgent thing though: I already changed in master the "point" value to "good" "bad" "blocker" and "neutral"; so currently the icons do not show up correctly.

[hugoroy]

Le mar. 28/05/13, 22:17, shybyte notifications@github.com:

Another question: Will there be an index which contains the URLs (to match against) and the rating. This information is important to show the rating icons if the user visit a page. Currently this information must be loaded, by loading all service files, which is not really scalable. Generally the current API is not really made for the need of a plugin which wants to be efficient.

Yes, we can do such an index, we “just” have to add it to buildIndexes.js

BTW, do not hesitate to make http://tosdr.org/api.html better :-)

@xMartin @hugoroy Maybe we should do a hackathon about that in order to make the new version better. If the API would be better suited for the plugins, there would be less need for a common library (which would be currently partly a workaround for the cumbersome API)

I'm all for it. But I'm not in Berlin these days.

Hugo Roy, Project Lead Terms of Service; Didn't Read | www.tosdr.org

[hugoroy]

The dataspec branch at https://github.com/tosdr/tosdr.org/ is nearly done. What do you suggest we do to have a smooth transition?

[shybyte]

Old and new version must be online at the same time and for some time on the http://tosdr.org/ domain in order to allow the browsers of existing users to update the extensions.

Proposed steps for API changes:

1. Put the new API version online on http://tosdr.org/ (parallel to the current version) under a new Path ( like /api/v2)
2. Lets update the extensions + discuss and improve the api in the process (maybe in a hackathon)
3. Finalize New API
4. Deprecate Old API
5. x Month later: Remove Old API

I guess these steps are just general best practice for API changes of web services.

[hugoroy]

@michielbdejong is the dataspec branch ready? Do you want to take care of that over the next 2 months?

[hugoroy]

Seems resolved now.