Excessive CPU Usage of oschema2html.xsl

[fundamental]

Testing with moderate sized inputs (~2.5k line input xml) an excessive amount of time is spent in this stage of processing. xsltproc --profile indicated that most of the time was spent in base64encoder. For http://fundamental-code.com/tmp/test-osc.xml this stage takes about 40 seconds at the moment. After removing the base64encoder calls, this time is reduced to 0.05 seconds. Is there a cheaper way to get a UID though a less expensive hash function or something?

[7890]

Indeed, i can confirm it's absurdly slow with that file (!) The (xsl) base64 is currently used to not care about special characters in a concatenation of several properties of a message. The idea was to not only have a UID per message but to be able to extract infos from it. Since the concatenation can contain urls it seemed near to use base64. A message UID could be used for some drag & drop scenarios not yet fully envisioned. But it's too slow. What do you propose? I'd like to prevent the use of xsl extensions (i.e. xmlstarlet must be "good enough") but maybe using a system base64 program to speed up is an alternative.

[fundamental]

I'm not too sure as I'm really not familiar with the XSLT ecosystem, but there is some information on general hash tables within the transformations though it wasn't entirely clear if this was some odd java extension or something. There is the generate-id() function, but I'm not sure if this would actually be useful outside of the xslt transform itself. There do seem to be projects to do hashing in direct XSLT, but I'm not sure if they will depend on the same costly calls https://github.com/rubinsztajn/xsltmd5 .

My suggestion without knowing much would be to make the IDs something really stupid simple and just establish a reverse map from ID->node if it becomes necessary. Ideally the IDs would either be hashed with a non-cryptographically secure hash function or they would just be incrementally assigned. I'd take any recommendation of mine with a hefty grain of salt, as parsing this code (and any javascript) is slow going for me due to the fact that I don't often use these tools directly.

[7890]

generate-id() is available but won't give us a unique ID that is guaranteed to be unique beyond the scope of transformation, node-set or document. Also generate-id() won't let us extract any information from it (like the tokens it was made up of, that by definition should make it "very" unique).

If we have an UID per message, it can be handed over to any tool that can get the details (just by having the UID as a maybe long but non-strange string). For example to "cherry pick" aspects or tell another program it should add a handler for that message etc.

However the xslt base64 template speed is not acceptable..

[7890]

I've changed the way the id is generated. base64 xslt isn't used anymore. Speed is much better now even with a lot of bash fiddling. For comparison, on PC1 it takes 4 seconds for the above file (684 messages). On a more recent notebook it takes 2.0 seconds, that is including start to end of the oscdoc call.

There is still a lot of potential for speed optimizations once the workflow and handling proofs to be reasonable.

How does this change things on your machine with the largest file you have?

[fundamental]

The largest file still takes many minutes (terminated before it finished), but for moderate sized ones it seemed fine. (2.5 seconds for another version of the 2.5k file) One speedup might be to eliminate the repeated calls to the validation function. If you want to see the stress test file get it here http://fundamental-code.com/tmp/zyn-params.xml

110k lines to show which stages take up most of the time.

[7890]

The test file from you shows oscdoc comes to limits when creating large sets of several 10 thousand messages (this one even hasn't many details per message, just many). Try here (give the browser some time to load and process (~25 MB)): http://lowres.ch/oscdoc/large/index.html Once loaded, it is far from usable. Try “Show All” or “/voice1/” in the search field.. A rough guess is that the current oscdoc output html file is good for say 1 – 5000 messages, it depends on the browser and CPU speed since it's operating flat on all data. Everything greater will probably require a database to reasonably handle it (i.e. search on all messages).

Let's looks at an extreme case. A unit is made of of a component structure that builds a tree with 10 sub-units per level and a depth of 10 levels. This gives +/- 10^10 possible paths. It's obvious that a flat representation of this number of possible methods of the described unit is not suitable for every client (for instance, to parse, have in memory etc.). A rule-based approach running on the client seems better in that case.

The case is not extreme if we think of a unit description that is actually a directory that just subsumes say the descriptions of all existing units (that offer a description). So the need to handle that is not absurd.

For every message in the API design process, making a distinction if it's an “atomic” aspect or part of a repeating pattern makes sense. There can be repeating patterns that scale to high numbers, for instance when the limit depends on a setting (“give me 128 audio channels”, each having a set of plugins with each having 100 more leafs).

Example:

A unit has an internal structure that is at the same time the way to address the components and pass arguments to them:

/a/b/c/d/e (types/args here)

If we take the nested tree case above it could look like

/a4/b7/c2/d3/e (types/args here)

Wishful would be to either define or use for multi-selection something like

/a[0-3]/b[18]/c*/d3/e (types/args here)

(/e would now be processed in multiple contexts)

We can go further and say a component /b([from-to]) can also appear as a leaf in the tree and take arguments.

/before/b[4,6]/b[18] (types/args here)

And wouldn't it be nice to mix it further and say

/a/e[18,(types/args here)], x[!2]/c*/e (types/args here)

etc. making it similar to look like xpath (which can be of interest for working in tree structures).

Generating and parsing such paths for the unit is likely easy since it's in the domain of the unit (i.e. intrinsic to the unit). Another unit interacting must either know the rules that paths can be made up, have a complete representation of the possible paths and/or query one or both of these aspects from the unit in question (semantics not yet considered for now).

Alternatives could be to try having a “path selection” that doesn't result in tens of thousands of possible messages by encapsulating the structure into a bundle or radically abstract away the unit's internal structure from the message paths and make it a string argument.

Let's look at a case

/a[1-3]/e[18,(types/args here)]/c*/e (types/args here)

/a s        //s: selection string, “1 3”, “1,3,2”, “!(3)” .. (or xpath, maybe)
        //this removes having paths /a1, /a2, /a3 etc. and gives more
        //selection freedom (if supported by unit)
/a s*       //s: selection string   *: the (possible) methods of a
        //ex. /a si, /a sif, /a sfi, /a s(arbitrary types if parsing supported by unit)

# (~ bundle)

#a  /a “1 3”
    #e  /e “1 8” 24 “any” .5
        #c  /c  “*”
            #e  /e “” 24 “other” .4
            //empty path implicitely means first or leaf
//extending the bundle
//  #x  /x “9” 33 “here” .1
//      #y  /y “” 0

Or flattened (order matters):

#a  /a “1 3”
    /e “1 8” 24 “any” .5
    /c  “*”
    /e “” 24 “other” .4

A limited interacting unit not handling bundles could send them in a series with a “transaction id” attached (if supported by receiving unit) to make multiple concurrent contexts possible, so messages can be sent to established contexts as regular, small “leaf” messages. The id can be made up by client, given on request by unit and might be used in responses to clients.

The other approach is to put all “non-leaf” paths and selections to a string that is understood by unit and patterns and semantics known to the client.

/e s(types/args here)       //s: selection/context string

/e “/a[1-3]/b[18,(types/args here)]/c*” (types/args here)

For possible large tree structures a unit should be able to tell about all components and/or methods for a given path/context to interactively navigate the 10000000000 messages directory in a way that would (for the initial example) give 10 messages per query.

For example

/a/b/c/x/describe       //possible sub-nodes of /x in context /a/b/c/

would send back to the requester a list of the children (1 level deep, viewed from given/established context) to navigate the domain of methods the unit understands.

(These concepts are not tested)

I hope it can be an inspiration since it's an interesting case (the zyn-subfx synth is known to be great)!

[7890]

When i imagine that these >30k messages are just for /part0/kit0/ .. oscdoc is really not fit for that. It needs another way to serve (nested) oschema instances for large sets.

[fundamental]

On Tue, Jul 22, 2014 at 11:24:46AM -0700, 7890 wrote:

The test file from you shows oscdoc comes to limits when creating large sets of several 10 thousand messages (this one even hasn't many details per message, just many). Try here (give the browser some time to load and process (~25 MB)): [1]http://lowres.ch/oscdoc/large/index.html Once loaded, it is far from usable. Try âShow Allâ or â/voice1/â in the search field.. A rough guess is that the current oscdoc output html file is good for say 1 â 5000 messages, it depends on the browser and CPU speed since it's operating flat on all data. Everything greater will probably require a database to reasonably handle it (i.e. search on all messages).

Yep and I'd imagine it would do worse with the raw set of ~15 million possible unique messages (direction+path+typespec), though it does make for a solid stress test of the basic tools. If it's usable in this case, then everything is likely instantanious for the other cases.

And wouldn't it be nice to mix it further and say /a/e[18,(types/args here)], x[!2]/c*/e (types/args here)

Just to be clear are you proposing this as an addition to the path definitions within the schema?

For possible large tree structures a unit should be able to tell about all components and/or methods for a given path/context to interactively navigate the 10000000000 messages directory in a way that would (for the initial example) give 10 messages per query.

Once again, is this a recomentation for the oschema/oscdoc or is this a general recommendation of API design on absurdly designed software?

For example /a/b/c/x/describe //aspect of a,b,c,x,..

would send back to the requester a list of the children to navigate the domain of methods the unit understands.

If you're just recommending a tree realization of path specs, then I'd support this, otherwise the idea is unclear.

[7890]

If there are 15 million possible unique message signatures, is it so that the unit would understand every of those at any time? Does the unit have this structure available in memory at all times? Is the API is dynamic, for instance do some of the enumerated paths / repeating patterns scale with user input? For instance could the user create a /voice22/? What happens if /voice3/ is being deleted, will /voice3/ be missing or all following move by one?

I'm just thinking from both sides, how could the API be clear, convenient and at the same time scalable for cases like this one. I'm also trying to imagine how a client would use / know about these 15 mio messages, how it could use the API so it's convenient for the client and the unit. I think we agree that having this amount of signatures in memory is maybe not the most convenient variant.

The proposal was that you could think of breaking the path into single messages giving you extendable ways to express variable aspects of the path (i.e. for selection of nodes at that level) and put them again to one via a bundle or transaction id. This leads to a fixed set of message signatures, defined selection criteria (also for unbounded/undefined counts) and gives more power to select and trigger methods of the unit. I'm not sure if the above example was clear, but basically it would say every component that can be a part of the path or a leaf method is defined as a "context-free" message starting with /. Since the message gets it's context only after being part of a path, the message needs good description of possibly different meanings in different contexts. It must be clear in which order these messages can be combined to build a path or "component selector" basically. So if you have /a1/b2/oscillator3/amp it would be /a, /b, /oscillator and /amp that are each described incl. the selection criteria they accept. We could define such an API with oschema and make some tests to see if it can work.

[7890]

"a tree realization of path specs" Something like this

$ cat a.txl | txl2xml
<?xml version="1.0" encoding="UTF-8"?>
<components>
  <u min="1" max="1">
    <a min="1" max="10">
      <b min="1" max="4">
        <x typetag="ii"/>
      </b>
    </a>
    <c min="1" max="1">
      <y typetag="ff"/>
    </c>
    <d min="1" max="8">
      <e typetag="sif"/>
    </d>
  </u>
</components>

$ cat a.txl | txl2xml | xmlstarlet sel -t -m "//*[@typetag]" \
-m "ancestor::*[not(name(.)='components')]" \
-v "concat('/',name(.),'[',@min,'-',@max,']')" -b -v "concat('/',name(.),' ',@typetag)" -n
/u[1-1]/a[1-10]/b[1-4]/x ii
/u[1-1]/c[1-1]/y ff
/u[1-1]/d[1-8]/e sif

[fundamental]

On Wed, Jul 23, 2014 at 08:35:09AM -0700, 7890 wrote:

If there are 15 million possible unique message signatures, is it so that the unit would understand every of those at any time? That would be ideal and the in place dispatch setup understands them all.

Does the unit have this structure available in memory at all times? Some representation of all messages should be available, but there is a massive amount of redundancy which is eliminated with the actual dispatch code which is getting used to generate the xml files.

Is the API is dynamic, for instance do some of the enumerated paths / repeating patterns scale with user input? Right now the API is 100% static. In the case of a dynamic API I think it would most likely make sense to have an approach which effectively says

/foo/bar/

where this class of aspects defines which aspects could appear on that subpath. As long as there is a concise way of defining these aspects or grouped subpaths then the total paths defined should only be something like 10k for every single path in zyn even with absolutely no support for any enumeration or ranged values within a path.

This is likely the best route for creating something that generalizes well considering that the enumerated paths are likely not going to show up quite as much elsewhere, though it would make things somewhat painful for the oscdoc xml writers if future users ended up using enumerated paths.

For instance could the user create a /voice22/? The total number of voices is fixed at compile time. So, there may be voices which are not used, but the space of valid paths is fixed (though it is rather massive).

If an unbounded space was used, then the wildcard path /voice/ would be a better descriptor in my mind, though there currently is not a means of defining that the '' could only be substituted with a well formed base 10 number.

What happens if /voice3/ is being deleted, will /voice3/ be missing or all following move by one? Voice 3 remains voice 3, Voice 4 remains voice 4, etc... I would think that it would be very subpar if paths were changed automatically. This design would break existing OSC controls and it would likely confuse users IMO, so it would make sense to disallow this.

I'm just thinking from both sides, how could the API be clear, convenient and at the same time scalable for cases like this one. I'm also trying to imagine how a client would use / know about these 15 mio messages, how it could use the API so it's convenient for the client and the unit. I think we agree that having this amount of signatures in memory is maybe not the most convenient variant.

I think the fundamental problem here is that large hierarchies form a directory structure populated heavily with symbolic links and ignoring the use of the symbolic links (which you can view through aspects or some similar subpath inclusion method), will result in turning an O(N) problem into an O(e^N) problem. 800-900 base ports are defined which is sanely handleable and 15 million expanded possible messages is not. If the problem is broken up into the subpaths this problem returns to being entirely tractable. ie when I see the path /part0/kit0/padpars/oscil/sapar I can say

• the base has parts
• a part has kits
• a kit has PAD note parameters
• a PAD kit has an oscillator...

To elaborate on this further I'll cite some of my goals of using this project. 1) Provide a more portable means of transmitting the port hierarchy for tools like oscprompt (similar to your oscc) 2) Provide a decent means of extracting the embedded documentation for the use of a user or some UI that needs to explore the possible interactions 3) Use the serialized representation to point out differences between various version 4) Permit the collaboration of various tools with OSC based APIs (mainly realtime based modulation without the extra step of midi learning (which has relatively bad usability without very tight integration IMO)) 5) Provide a better means of automated testing (Think about the power that the qt meta object system provides and how a full OSC API is essentially a decently strong RPC mechanism) 6) Point out any dead bits of the API as there surely are some with all the hacking that has gone on.

The proposal was that you could think of breaking the path into single messages giving you extendable ways to express variable aspects of the path (i.e. for selection of nodes at that level) and put them again to one via a bundle or transaction id.This leads to a fixed set of message signatures, defined selection criteria (also for unbounded/undefined counts) and gives more power to select and trigger methods of the unit. I'm not sure if the above example was clear, but basically it would say every component that can be a part of the path or a leaf method is defined as a "context-free" message starting with /. Since the message gets it's context only after being part of a path, the message needs good description of possibly different meanings in different contexts. It must be clear in which order these messages can be combined to build a path or "component selector" basically. So if you have /a1/b2/oscillator3/amp it would be /a, /b, /oscillator and /amp that are each described incl. the selection criteria they accept. We could define such an API with oschema and make some tests to see if it can work.

Well, yes such an API could work with oschema currently but it fails most of the goals that I have described above. (I am assuming that you mean that zyn should change its API form this) Such a flat API loses the organization that a filesytem+symlink like approach result would have. Secondly it introduces ambiguity as there might be two different ports named "amp" which would refer to different parameters, but when flattened they would collide with this example. This bundle based means of transmission is also not what is reflected in the use of OSC in all of the tools that I have since observed.

So 1) The hierarchy is removed which damages the idiom of one message per command (aka the normal REST like APIs which have become dominate) 2) The ports are disorganized making this setup ill formed for creating documentation that explains separate components 3) Collisions make discerning differences more difficult 4) This convention is atypical which would hinder adoption 5) There needs to be some description of what is acceptable after one of these flat entries to encode a port anyhow, so the flat realization would need to have the same information as the longer paths anyhow 6) see 3

I'm starting to get the feeling that both of us are missing a point of the other at this stage.

[fundamental]

"a tree realization of path specs"

Ah, that makes most of my email a misunderstanding now :p

[fundamental]

Just let me know which format you want to start seeing the test input in and I should be able to get you a steady stream of it (though I may need to rewrite a bit of my port walking code)

[7890]

We could just add ranges to the pattern facet. The patterns then leave the allowed domain of chars defined by the OSC specification (as already done with '*') and will become an abstract path, to be interpreted. Such messages could be marked as abstract.

You could prepare the stream to replace all repeated component paths with ranges like this:

 /part[0]/kit[0]/voice[0,15]/VoiceFilter/Pvowels[0,5]/Pformants[0,11]/freq

(starting numbers at 1 could be considered)

That can then be further simplified with the use of aspects, let's see

[fundamental]

I'll assume the [0-N] format within the paths seemed good enough for the moment. Here's an example file to show the size difference.

http://fundamental-code.com/tmp/compressed-params.xml (down to 7k lines with the inclusion of even more parameters than the last file)

[7890]

I've changed the oschema regular expression so it can support [] ranges in the @pattern and adjusted oscdoc. Now ranges in the form [n,m] denote a min/max inclusive range n-m. Why it's written without the dash is questionable. Looking at math intervals shows using ',' is commonly used too. ',' won't be mixed with decimal separator '.' but '-' could be mixed with negative numbers. We don't have any in that current case though. for [n,m]: 0<=n<=m [n,]: unbounded range -> ~infinite number of messages [n]: simplified writing for [n,n](it's still a range) The program and API must decide if it's using the numbers zero-based or not. For instance 16 voices -> [0,15] or [1,16] ? It's left open, it can both be useful, especially when the paths are made up as human readable hints. Starting to enumerate "things" at 0 is something that (for humans) need an additional interpretation step, to map the number to the nth of something in the "natural" way (i've eaten three apples today (it was the 3rd)).

You might like the tree view in oscdoc. It gives a quick overview on the structure of the unit and is linked to the list filter. It can also be updated to show the active list entry. Also try the typtag filter. Only messages without a typetag: ^$ with an i: i starting with i ^s etc. It needs a lot of testing .. if you experience other missing parts of oscdoc (complete types support for the schema and oscdoc output noted) or performance issues, just let me know.

The speed compared to the first prototype is much better. The 2206 messages of the file above go through in an acceptable time for the usecase (which is to generate pre-rendered but still dynamic HTML page for human use). A message count of a few thousand messages can be handled relatively well. With the introduction of ranges in the paths the count of tokens could be reduced a lot. I allow myself to close this issue and will be curious what you come up with, how the zyn documentation (and hopefully oschema/oscdoc along) will develop.