timing data

[MeraX]

Just to keep threads clean, I cut out the timing data question from https://github.com/roelj/inklingreader/issues/7 @xuv https://github.com/roelj/inklingreader/issues/7#issuecomment-40289074 :

Yet another maybe stupid question (and this bug report is turning into a huge thread), but is there "time" data in the WPI file in any way ? I mean, does the file has a record of some sort of "stroke speed" ?

@roelj https://github.com/roelj/inklingreader/issues/7#issuecomment-40296627 :

About timing data.. as far as I know this is not stored. There still is some data left that we don't know what it is or represents. But honestly, I don't think the data represents timestamps or anything related to the speed of drawing. The data is linear in the sense that whatever is written first with the pen, is written first in the file (therefore "playback" of strokes is possible).

I would suggest to investigate, weather the recorded data has a fixed frequency, e.g. 10 records per second. If this is the case we have some speed information. (Note: with my extension to stroke width I added a mechanism to skip records with same (x,y).)

[roelj]

I just tried to figure out whether data is recorded with a fixed frequency by drawing a line slowly and drawing a line quickly. I believe @MeraX is right. There is a fixed frequency and so we do have some timing information. By measuring the distance between points, timing information can be extracted.

[xuv]

That is cool news. I was guessing for something like this also. I think that a way to play back again the strokes with some timing is a great thing. Maybe we would need an open file format for exporting these data, like a JSON or an XML. Any structured text dump of the data would be good I guess.

[MeraX]

I agree, a simple conversion from WPI to some common fileformat like JSON would be handy to further investigate on the data with a programming language of your choice. (e.g. I would play around with it in a python environment)

Personally I prefer JSON and as far as I understand inklinkreader and the WPI format, JSON fits well on the data.

maybe something simple like

{
  "$layer_ID": {
    "$StrokeID": {
      "$pointID": {
        "x": "$x",
        "y": "$y",
        "pressure": "$p",
        "tilt-x": "$tx",
        "tilt-y": "$ty"
      }
    }
  }
}

with x and y already converted to a useful unit like mm. Is there additional information, that could also be interesting to investigate on?

[roelj]

Have a look at the datatypes, they show what data is available to us: master/src/datatypes.

Writing a JSON "converter" wouldn't be too difficult. The SVG "converter" displays pretty much how it can be done.

I'll put it on my TODO list unless someone else is willing to write it :).

[roelj]

Anyone interested in a Python or Guile library? Take a look at Issue #9.

[roelj]

The JSON converter has been added to the code. For further discussion of the JSON format please see Issue #12.

I also have an idea for a document "playback". It'll take some work on the GUI code though.

[xuv]

Thanks for JSON format. Have still to try it out. But great to see this coming out.

For the "playback", I don't really see a purpose for the InklingReader software to have this feature. Although it would make a nice "wow" effect to see a drawing being replayed. (But maybe I didn't understand the "feature" correctly).

In need of a playback, I would maybe use external softwares like http://sourceforge.net/projects/mgcomposer/ (Mouse Gesture Composer) and another one I can't remember the name for now. Those have already build tools to "sample", "edit", "time stretch", etc... that would be suitable for "playback" needs. They also have ways to "send" these information in realtime to other softwares with OSC protocol for example.

Just a point of view.

[roelj]

Well, my idea is to get all the feature from the SketchManager to InklingReader (except for the integration with proprietary software). One of those things is the playback of strokes.

Using MGComposer would be very difficult since it looks like it's written in Java. Thanks for the idea though. The Processing.org language looks pretty cool for this. I don't know how difficult it would be to generate valid Processing.org code.

[xuv]

I didn't know the "official" app had that feature. So it makes sense now that you want to add this. :)

About Processing.org, as a user of it, I don't think it's relevant to export processing code from InklingReader. But we could provide a Processing example project that parses and reads the inkJSON file and turn it into an animation. Processing has already an import JSON lib, so it should be trivial to create this. I can take some time to do this.

The MGComposer link was just an example of an app that does extensive use of "mouse gesture playback". I wasn't suggesting integrating it in InklingReader. Just to show that other apps really expand already on the concept. Maybe exporting to an MGC file format? But I'm not sure it's the most used app around. So it would maybe be a waste of time. There is another similar one with maybe more users. But I can't seem to recall the name of it. Although if you would want to integrate InklingReader with any "mouse/drawing gesture playback (or not)" application, the "online" feature is where to look I guess. ;)

[roelj]

A Processing example project that parses and reads tje InkJSON file and turn it into an animation is a great idea! Is the current JSON output good enough / handy for this?

I'll look into programs like MGComposer (thanks for the link, I didn't know they exist :)).

[xuv]

As a digression or food for thought, I'd like to point to you some open standards that might give perspective to the inkJSON format developed here and might be a suggestion to "timing data".

There is the GML : Graffiti Markup Language which has been developed by a group of artists and is used in quite some artistic projects. In its spec, it has optional "timing data". Unfortunately, no real data structure for "layer".

And then the "official" InkML by the W3C. "This specification describes markup for representing ink entered with an electronic pen or stylus".

I haven't deeply studied both. I know GML because I follow the work of the artists who created and used it. I like the fact that it has a on-line database of drawings/graffiti. There is Processing projects and libraries that read and have used this format. But I think the global project is stalled right now.

And for the InkML, I've not come across any program that uses it. Searching for some, I found a JS library to display InkML drawings in the browser. But it's W3C standard. :)

What's your opinion about this? Should the actual inkJSON be improved regarding these file formats? Should one of these be preferred for future compatibility with other programs?

[roelj]

Not looking into the formats I'd say the name InkJSON implies it's related to InkML. So by that logic a JSON-version for GML would be an GML-JSON.

When looking into the formats InkML looks pretty complex (as always with W3C standards), but it looks complete and easy to use/implement (at least the basics we need).

GML looks straightforward and easy to use/implement as well. Quickly scanning the spec on the first page I have seen a way to color individual strokes, providing X/Y, pressure and (possibly) tilt information.

I think we should definitely follow some existing standard when creating InkJSON. The name implies that it follows the spec of InkML. GML looks like a great format to implement (whether it's in JSON or XML format).

The current JSON output of InklinReader is probably not compatible with either of them, Although making it compatible won't be too difficult.

Now that InklingReader is getting more mature it may be a good time to write more documentation. Are you interested in proposing a JSON format compatible with one (or possibly both) markup standards (InkML and GML)?

[xuv]

I don't really know how you make a JSON compatible with an XML. I mean, it would still need a converter to turn one into the other, anyway. I tend to agree with all of you that JSON is "more easy" than XML in this case. But the 2 "standards" so far are XML based. Making a InkJSON or GML_JSON would create a third "standard" wouldn't it?

So if we go for one, it's one of them XML. The question is, do we favor W3C, which is feature complete but has no (public) real use cases. Or GML, which has an artist community behind it, but with a project that seems stalled (main database of drawings seems down since a couple of days).

I'd tend to favor the "most standard": InkML.

[roelj]

@xuv

I don't really know how you make a JSON compatible with an XML. I mean, it would still need a converter to turn one into the other, anyway. I tend to agree with all of you that JSON is "more easy" than XML in this case. But the 2 "standards" so far are XML based. Making a InkJSON or GML_JSON would create a third "standard" wouldn't it?

I was thinking about something like this: When InkML has a

<ink>

element. The JSON equivalent would be

ink { }

The differences in code to output InkML or InkJSON would be very small this way.

[xuv]

Sorry if this is a stupid question, but isn't there a problem in translating XML attributes and content in a "similar" JSON format?

Take the specification for example http://www.w3.org/TR/InkML/#traces, It has a list of attributes and contents. Their example:

<trace xml:id="id4525abc">
    125 18432,'23'43,"7"-8,3-5,7 -3,6 2,6 8,3 6 T,2 4*T,3 6,3-6 F F
</trace>

I'm not sure how this is going to be translated to JSON. So I'm really wondering why make a JSON version of InkML in the first place. The actual implementation of the JSON is fine, I guess. An XML (InkML) export would then a different feature?

[roelj]

I received an e-mail a while ago from someone named Torben pointing out that there probably is timing data. He figured out how to read the timing data using information from libdpen.

[xuv]

That's great news.

On 16 août 2014 17:58:11 UTC+02:00, Roel Janssen notifications@github.com wrote:

I received an e-mail a while ago from someone named Torben pointing out that there probably is timing data. He figured out how to read the timing data using information from libdpen.

---

Reply to this email directly or view it on GitHub: https://github.com/roelj/inklingreader/issues/8#issuecomment-52396902

@xuv http://xuv.be

[xuv]

Back with questions, as always ;) Are you planning on using libdpen in inklingReader?

I've tried to read the doc your pointing to. Is that timing data usable now? Or is still not enough decoded to use?

[roelj]

I'm not planning to use libdpen in InklingReader as I don't really see why.. I could just implement the timing data bit..

What features would you like to see that could be done with the timing data? I'll see if I can implement it..

[xuv]

In the official app from Wacom, there seems to be a double slider (with an in and an out) that permits to only display (and export) a portion of the drawing. I guess this is based on timing data. Not that I find this useful (yet), but since your intention is to reproduce the features from the official app, it's maybe something to consider.

On the other side, I would love to have this timing data added to the JSON export. This I find useful. Thx.

[roelj]

I've added the clock counter to the WPI parser but now I'm stuck at where I should put the data in the JSON output..

Where would you like to have it? This is the current JSON output format:

{
  "0" : {
    "0" : {
       "0" : {
         "x" : 356.962952,
         "y" : 181.685181,
         "pressure" : 489.000000,
         "tilt" : { "x" : 218, "y" : 242 }
       },
       "1" : {
         "x" : 357.000000,
         "y" : 181.685181,
         "pressure" : 511.000000,
         "tilt" : { "x" : 218, "y" : 242 }
       }
    }
  }
}

I'll publish the changes when it's in the JSON output.

[xuv]

Mmmh. Tricky question.

As I understand the data from the libdpen docs, timing data is not directly related to pen data. I mean, each stroke doesn't have a starting time or end time. Neither has each point. Am I right?

So how does this data comes through? Am I right if I understand it this way: every second, there is a time marker inserted in the stream of points. This gives the "elapsed time" since the device was started. Right?

So I guess each point can have another property called "elapsed_time" (or just "time" for short) that stores this number. Doesn't really matter if two (or more) consecutive points have the same elapsed time, as long as they were drawn during the same second.

Does this makes sense?

[roelj]

Yes, you are right: I just tested and the timing data can show up at any moment (inside a stroke, outside a stroke, after tilt data or after a "new layer"..).

So the proposed changes would look like:

{
  "0" : {
    "0" : {
       "0" : {
         "x" : 356.962952,
         "y" : 181.685181,
         "pressure" : 489.000000,
         "tilt" : { "x" : 218, "y" : 242 },
         "time" : 12
       },
       "1" : {
         "x" : 357.000000,
         "y" : 181.685181,
         "pressure" : 511.000000,
         "tilt" : { "x" : 218, "y" : 242 },
         "time" : 12
       }
    }
  }
}

I think this is good because it doesn't break the old JSON output. The only downside is having a lot of duplication of the "time" field.

I'll implement it and publish the changes in an hour or so.

[xuv]

Great. Thx. And about the JSON, I guess with a format like this, the point is not to be light. :) So no worries about repetition, I 'd say.

[roelj]

It's in 24b0be9bf31728eebde4c8f4df0643de8557aa78. Let me know whether it works for you.

[xuv]

This works for me. Thanks for this implementation.

[roelj]

The original issues have been handled now so I'm going to close this issue now. For the discussions on the XML and JSON formats I'd suggest opening new issues when someone still wants to discuss/enhance these features.

[bakercp]

@roelj @xuv Hi there, this is such a great project -- thanks @xuv for pointing me to it. Just to confirm my hacking experience with timing data in (which is super important for my projects) libdpen, the timing data needs to be propagated from the the primary timestamps to assign a precise time to each sample. This isn't an exact science, as a few assumptions need to be made, but it is possible to get it close enough. Since the sampling rate seems to be fixed, one can propagate timing data forward and backward from each timestamp in order to assign a timestamp to each sample. Thanks for a great project @roelj!

[roelj]

@bakercp Thanks for revealing that there is timing data in the files!

I reopened this issue because timing data can be made more precise as @bakercp explained. I'll look into this further.

[bakercp]

@roelj super! Looking forward to it.

[roelj]

@bakercp: Do you know what the frequency is? I get mixed results:

I'm counting the amount of coordinate data blocks per second and here's a sample output:

Frequency = 0
Frequency = 0
Frequency = 75
Frequency = 153
Frequency = 152
Frequency = 153
Frequency = 152
Frequency = 153
Frequency = 153
Frequency = 152
Frequency = 153
Frequency = 152

This is produced using a drawing where I kept on drawing circles for a couple of seconds at normal speed.

Now the next sample is from a drawing where I kept on drawing circles for a couple of seconds at a high speed:

Frequency = 0
Frequency = 136
Frequency = 156
Frequency = 153
Frequency = 152
Frequency = 153
Frequency = 152
Frequency = 153
Frequency = 153
Frequency = 152
Frequency = 153
Frequency = 152
Frequency = 153
Frequency = 153

What is the frequency? I peeked at your code and I found the base unit to be: 0.0067 sec. However, by my experiment it (taking 152.5 as average) it should be: 0.006557377 sec.

And what's causing the spike of 156 coordinate blocks in a second?

[bakercp]

I'm not sure what the sampling rate is -- for some reason I noted it as 200 Hz, but I'm not sure where I got that number. When I get a minute, I'll dig into my sample files and see what I can find ... unfortunately my Inkling battery corroded recently and I'm in the process of sourcing a new one ... :/

[roelj]

I've added a feature to the GUI to "scroll through time". I hope it performs well on your end too.

[xuv]

Works great for me. Thx.

[roelj]

Timing data is in the JSON and CSV output formats, and in the GUI we now have a slider to go through time.

I think this issue can be closed now.