Transfer units and conversion factors to fluksoapi.py

[saroele]

@Ryton included conversion factors, units and other meta-info in the notebook Demo_WaterGasElekVisualisation. It propose to rewrite it slightly and include it in the fluksoapi module in order to avoid code duplication.

With rewrite slightly, I basically think of using dictionaries instead of lists, and using the utility (gas, water or electricity, later maybe also the variables measured by Kubes) as key.

[Ryton]

Good plan! But: A) Check validity of the used figures first, B) Please also include a (or multiple?) default "time integrated" default unit and conversion factor, as I did. C) And (maybe -though for utility usage sensors, this probably won't be needed-) even include a 'time differentiated' unit/conversion. For other sensed parameters this can be usefull: e.g. Temp measurement: [°C ] => time differentiated: Temp difference [ °C/ hr](for zone temperature or buffer tank evaluation). Or: (e.g. for solar panels electricity meters): [kWh] => differentiated: Average power during a period of time: [W]

Linked to question C): What data is in TMPO? the same data as in the API, or raw data?

[saroele]

Let's use this thread also to discuss unit conventions.

We have the raw data in a certain 'flukso-unit' which is probably not what we want to use and show. So when loading the data, a conversion should automatically happen so the dataframes we obtain are always in our 'base unit'. I think it makes sense to take a 'flux' unit as base unit (like l/min, m³/day, W, kW). We then also have to agree on a unit for 'integrated quantities' like liter, m³, kWh, ...

Proposal based on @Ryton: flukso-units (what we get from the api, to be checked!!):

• water: l/day,
• gas: m^3/day (~ 10 kWh/day)
• electricity: W (this was Ws/day, I think it is W)

base-units (content of our dataframes):

• water: l/s,
• gas: W
• electricity: W

integrated-units (used when integrating over a period, like a day):

• water: m³,
• gas: kWh
• electricity: kWh

[saroele]

Our comments crossed in mid-air... A) and B) ok! C) makes sense indeed for non-consumption data.

For Tmpo we'll have to work a bit more on the data interface, because what we get will most often be counter values (so unit is a quantity). At least, I think so.

@icarus75 can you tell which units tmpo will be in? And check if the flukso-units above are correct?

I will create a branch for this issue with a proposal for implementation. See branch issue24_UnitConventions

[Ryton]

@saroele If tmpo ouptut = "quantity", then definitely a conversion including differentiation should be added (C). Or at least a conversion from the tmpo-unit [a quantity unit] to an integrated unit [qty], & inverted conversion back to a utility base-unit [qty/time] .

[Ryton]

Quote 3 post above: base-units (content of our dataframes): water: l/s, gas: W electricity: W One (major?) point of criticism on the proposed W, l/s and W units: As already discussed at length on the flukso forum, these (near SI-) units are preferred for calculation by engineers, but aren't always ideal for visualisation.E.g. if you make a scatter plot of consumption against time, with 1 minute, 15 minute or hourly data the along te time axis, interpreting the data in units "l/s" or "W" (on Y-axis) can be confusing.

Also note that (at this moment) data is mostly available on minute resolution. shouldn't the default "visualisation unit" have a timebase of one minute, then? [kWh/minute], or [J/min] may seem a bit akward, but [l/min] feels more right to me than [l/s] against a minute timescale...

[saroele]

Yes, that's a good point. Maybe we have to take a step back, and only define clearly which unit the flukso-data is in, and in which unit we use it in our dataframes. From there on, everyone is free to visualize in the most appropriate unit.

Scatterplots with data grouped by eg 15 minutes is indeed a tricky one. Either you present it as a quantity (J, liter, kWh) or as a 'flux' but then it's probably best to use the binning in the unit? Eg: kWh/15min. But that is indeed awkward. Maybe just keeping quantity in that case? Or going back to the base unit, and presenting it as an average. Like W or l/min (avg per 15min). I'll sleep over it :-)

On Sat, Nov 15, 2014 at 10:13 PM, Ryton notifications@github.com wrote:

base-units (content of our dataframes): water: l/s, gas: W electricity: W

One (major?) point of criticism on the proposed W, l/s and W units: As already discussed at length on the flukso forum, these (near SI-) units are preferred for calculation by engineers, but aren't always ideal for visualisation.E.g. if you make a scatter plot of consumption against time, with 1 minute, 15 minute or hourly data the along te time axis, interpreting the data in units "l/s" or "W" (on Y-axis) can be confusing.

Also note that (at this moment) data is mostly available on minute resolution. shouldn't the default "visualisation unit" have a timebase of one minute, then? [kWh/minute], or [J/min] may seem a bit akward, but [l/min] feels more right to me than [l/s] against a minute timescale...

— Reply to this email directly or view it on GitHub https://github.com/opengridcc/opengrid/issues/24#issuecomment-63192878.

[Ryton]

Updated the library to present/convert from/to tmpo units as well (based on comments in commit for issue30_Carpetplot (#32)

Apparently tmpo uses l/hour, whereas the csv files use l/day for gas…and water!

[saroele]

This never made it to the develop branch, I don't know why we 'forgot' it. Nevertheless, units are important!

Taking a step back, I have the impression that we tried to reinvent the wheel. There are good unit libraries in python, and I think pint is one of the most used.

Also, we have a new houseprint class now, and the Sensor object has attributes for 'type', 'quantity' and 'unit'. I dont think quanity is needed if we use pint to define the unit.

So the approach I have in mind:

• use the unit attribute in each sensor to define what unit the raw source data is in
• apply a unit conversion in the get_data() method of the sensor. For now, just simple conversion and differentiation to standard units seems ok as a start.
• later we can add target units to the get_data() methods and use pint to automatically convert.

[Ryton]

Possible Explanation: i never pushed it, as i'm not sure it was solid, both in coding and in value consistency.

I never heard of 'pint' till now, but would love to look into it!

At least, if: a) no-one else wants to prioritize this, and b) if i manage to understand the new structure of this repository, :wink: and where it is installed in windows after !pip install opengrid"... :confused:

Plan of attack:

• Adding conversion (& Diff) towards SI-units first makes most sense imho.
• Later this can be extended (if needed) with other, more tangible units or with economics, such as [€/ time unit] or [€] .

[saroele]

I did a pull request yesterday, I think I made big step forward. Basically, unit conversion is now in the hear of the Sensor object, and that's a logic place. I documented and demo'd everything in the notebook Demo_Units_and_Conversion.ipynb in the issue24 branch. (see here: https://github.com/opengridcc/opengrid/blob/issue24_UnitConventions/notebooks/Demo_Units_and_Conversions.ipynb)

I think that the only thing missing compared to what you did @Ryton is costs and tariffs. For fixed tariffs, its just a multiplication, for day/night or variable tariffs we need to define Timeseries with the cost profile and do an element-wise multiplication. Maybe this is not a priority now? I would also do this in a different issue.

Also, the units are not incorporated in the data itself. It would be an interesting idea to extend the DataFrame and Series objects with a unit attribute? That's a new development though, and not high priiority now.

If you have a look this weekend, start with the Demo notebook and see if that is sufficient for now. Curious to hear your feedback!

[Ryton]

Great, it looks like you made lots of progress already! Will take a look at it.

PS: I still disagree about the low priority of adding costs & tarrifs though. ;-) At least adding a fixed tarif is quick to implement & can generate very tangible insights! ;-) Though the calculated results may be a bit of compared to the real bill, with the uncertainties on:

• some measurements (missed pulses, scaling error, sensor uncertainty)
• caloric value of high vs low caloric gas
• (measured) apperent electrical power instead of real (accounted) power.

[saroele]

I think the best way is to associate a tariff to a Site. It will then be available to all Devices and Sensors on this site. To define a tariff we need:

• the quantity or type (like electricity, gas, water, fuel)
• unit (€/m**3, c€/kWh, €/MWh, ...)
• a cost profile. There are different options for the cost profile:
• flat
• weekly profile based on weekday and hour
• any other profile The first two can be configured with just a few parameters, the latter requires a pandas Series or DataFrame with the profile. I have a method lying around to create a weekly profile, eg for day/night tariff.

Maybe we can define a small class that defines a tariff. We can then subclass it for the different options. A tariff object should have a method to return a dataframe with the cost of a given profile. Obtaining the price would then be something like:

tariff = TariffWeekly(type, unit, cost_low, cost_high, days_high, hours_high)
power = Sensor.get_data(head=..., tail=...) # returns a df
cost = tariff.get_cost(power) # a dataframe with cost instead of power

This seems something very generic, and we can then also check the units and raise an exception if the unit of the power dataframe and of the tariff are not compatible. At this moment however, the power dataframe does not contain info about its own cost. I'd like to add the unit as attribute to the dataframe.

[saroele]

Discussion and results from meeting:

• unit can be added to series as attribute and will be preserved when concateing a series to dataframe
• check if slicing the dataframe keeps the unit ==> NO
• so using this attribute is not really a solution: the attribute gets lost when doing .copy() or other operations

Conclusion : fixing the default units of dataframes and series as follows:

The default resulting unit for counter values (diff=False) are:

sensortype unit

water ==> liter gas ==> kWh electricity ==> kWh

The default resulting unit for flux values (diff=True) are:

sensortype unit

water ==> liter/min gas ==> W electricity ==> W

About costs and tariffs: lets make another issue and keep it on the list. Not top priority now.

[saroele]

I have made one more useful change: all get_data() methods will return a dataframe with unit attribute set for each column. As we discovered, this is not persistent: pandas will remove this attribute when doing operations with the dataframe. But for a first check it is extremely useful.

Also, each sensor now has a new attribute: cumulative. For fluksosensors, if it is '' or None, it is determined based on the sensortype, and for electricity, gas or water it will be set to True. This attribute has an important implication: get_data will use it to determine if the data has to be differentiated or not. You can override this behaviour by setting diff=True or diff=False in the get_data methods.

[saroele]

We're done with the units. About costs and tariffs: lets make another issue and close this one.