search

mtiller / recon

Web and network friendly simulation data formats
MIT License
8 stars 4 forks source link

What does x*scale+offset do to booleans or strings? #16

Closed harmanpa closed 10 years ago

harmanpa commented 10 years ago

In meld.py, when fetching data, what happens from the following when the data is a boolean or string?

return map(lambda x: x*scale+offset, data)
xogeny commented 10 years ago

Ha! You asked me this WHILE I was fixing it. That was an early assumption about everything being numeric that I knew I'd need to deal with and I did just that apparently while you were putting this question in.

As it stands, this should be addressed with the latest commit. However, there is still an open issue here which is detecting cases where scale and offset don't make sense. My recent changes make it so that scale and offset are not mandatory. So then you can store strings, booleans, etc.

But what happens if you supply a scale and offset and then provide bool values?

For now, if you don't specify a constraint type then we assume you know what you are doing. If you do provide a constraint value, we check to make sure it is numeric.

I tried forcing users to specify a constraint type if they wanted to use scale or offset. But the problem is that you can have numeric types besides the Python builtins (matrices, complex numbers, numpy.float64, etc) and we cannot check all of those. So not specifying a type is an "out" in that case. Of course, you lose type checking functionality too.

We'll have to see if this is a reasonable compromise. Perhaps we just shouldn't check at all. If they specify a scale and/or an offset, perhaps we should just wait and get a runtime error somewhere.

But in any case, the most recent commits at least removed the need for scale and offset to be mandatory.

harmanpa commented 10 years ago

What about inverse aliases of booleans?

xogeny commented 10 years ago

So you mean something like having scale have a value of True if the alias was exactly equal to another boolean variable and False if it was the inverse of the base signal?

xogeny commented 10 years ago

Is this complexity worth it? I mean boolean signals are not a huge amount of storage. Inverses are then a subset of those. I'm not sure we'd really get any potential savings out of this. For one thing, I suspect people may not even think to use this feature even if we implemented it.

The problem I see with features like this (or even scaling+offset in general) is that they layer additional semantics onto the coding a decoding. I seriously considered just throwing out scale and offset altogether and sticking them in metadata instead. That way, people who extract the signals can do whatever they want. But it doesn't increase the complexity of conforming to the spec (since the spec would only dictate that implementations simply propagate metadata, not interpret it). Do you see my point?

harmanpa commented 10 years ago

I'm not really considering the data saving, just the pre-existing data structures I'd like to fit in. I can interpret it that way if the data is boolean.

xogeny commented 10 years ago

If you don't care about data savings, then don't store as an alias, store as a (base) signal that is boolean. Then you can avoid having to "interpret" it at all.