P110.getEnergyUsage() no longer returns historical data

[bacheshrew]

I think historical data is cleared by the firmware update, but should there be empty fields (or zeros) in the data returned? Using the proposed change https://github.com/fishbigger/TapoP100/pull/77 and calling getEnergyUsage() I get two different results, depending on the firmware version.

For devices running the older firmware (v1.0.16 Build 220624 Rel.171733), I get 'result' which shows historical data populated as: {'today_runtime': 1213, 'month_runtime': 42966, 'today_energy': 1414, 'month_energy': 56071, 'local_time': '2022-10-22 20:13:00', 'past24h': [72, 42, 167, 134, 54, 37, 72, 88, 37, 38, 85, 77, 40, 62, 98, 105, 65, 87, 62, 36, 88, 74, 46, 29], 'past30d': [1916, 1766, 1825, 1788, 2050, 2037, 1919, 2081, 1858, 1783, 2044, 1922, 1852, 1907, 1790, 1762, 1851, 1896, 2072, 2046, 1796, 1848, 1941, 1940, 1839, 1894, 1792, 1765, 1677, 1414], 'past1y': [0, 0, 0, 0, 0, 0, 0, 0, 0, 29890, 56387, 40689], 'past7d': [[75, 60, 65, 54, ....<snipped to save space>....]], 'current_power': 129899}, 'error_code': 0}

For devices running the newer firmware (v1.1.2 Build 220930 Rel.144500), I get 'result' populated with no historical data fields as: 'result': {'today_runtime': 165, 'month_runtime': 9314, 'today_energy': 33, 'month_energy': 1891, 'local_time': '2022-10-22 20:13:01', 'electricity_charge': [0, 0, 0], 'current_power': 12173}, 'error_code': 0}

[CameronInLondon]

I can confirm V1.1.2 is not returning the historical data when calling getEnergyUsage(). I get:

{'result': {'today_runtime': 78, 'month_runtime': 666, 'today_energy': 14, 'month_energy': 174, 'local_time': '2022-10-23 10:14:31', 'electricity_charge': [0, 0, 0], 'current_power': 12071}, 'error_code': 0}

It is worth noting that the Tapo app has moved from 'Energy Usage' and 'Runtime' to 'Energy Usage' and 'Power'. I will add that I have found the new feature to add price per unit buggy in the app.

[CameronInLondon]

Thanks to @tnmendes for comment here

With some small changes to the getEnergyUsage() method

{ "method": "get_energy_data", "params": { "end_timestamp": 1667260800, "interval": 43200, "start_timestamp": 1640995200 } }

My testing shows that it works and working with an internal of 43200. {'local_time': '2022-10-23 15:00:51', 'data': [315, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'start_timestamp': 1666488410, 'end_timestamp': 1667260800, 'interval': 43200}

But when I test with other internals like 3600 or 7200 I am getting {'error_code': -1001}

[tnmendes]

Glad to see the solution helped. So the Tapo app is using for intervals: 43200 = 1 month 1440 = 1 day 60 = 1 hour But I'm detecting strange behavior in start_timestamp you need to set the correct value which is the first day of the month at zero hours to work. The tapo app is asking for the last 3 months so I am assuming that new firmware will just store 3 months of daily information.

[CameronInLondon]

Yes as you said before minutes, I see now. I can get 7 days of hourly data, 90 days of daily data, and 12 months of monthly data.

It is returning all the data in one list none of the past7d from the older firmware.

I note that if requesting to start mid month via start_timestamp it in fact just returns from the start of the month. Like below (only got this plug two days ago hence the 21 zeros at the start) 'data': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 160, 537, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

A strange thing is that when using looking at hourly data the plug seems to be a day ahead. By that I mean I can look at data for 24 Oct. I seem to have the correct location selected in the app. For example start datetime(2022, 10, 23) and end datetime(2022, 10, 25). Hence catching data for 23rd to end of 24th (today is the 23rd). I get:

'data': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 23, 29, 41, 20, 5, 0, 0, 15, 27, 0,
 0, 0, 0, 0, 0, 0, 0, 0, 0, 11, 12, 12, 11, 33, 75, 41, 50, 64, 58, 33, 82, 75, 30, 0], 
'start_timestamp': 1666479600, 'end_timestamp': 1666652400, 'interval': 60}

How can it be returning data for tomorrow? Crystal ball, Flux capacitor... Bug.

[bobo3k]

"A strange thing is that when using looking at hourly data the plug seems to be a day ahead." It looks like it's the same on the Android APP... at the time of the screenshot, it was 9:15 AM, Oct 24th on my computer. The graph on the mobile app is showing data up to oct 24th 23:00-24:00 and it contains data! I am ot sure how we are meant to treat the data.

hoping this might help, B

[lemoustachiste]

Generally speaking,

are we able to rollback the firmware version?

I have blocked requests to tplink servers with my pihole so hopefully I maintain a fixed version from now on.

[lemoustachiste]

The daily read is indeed off by one day (reading the data from Oct 25th 00:00 to Oct 26th 00:00 is giving my current day consumption (Oct 24).

The monthly read is also not intuitive. If I put the end/start date as for the daily read, the first index is the first day of October. If I put the time range from Oct 1st to Oct 31st, then this is what I get:

[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1561, 9059, 8366, 9350, 9943, 7758, 7284, 6865, 9349, 6953, 6498, 5901, 11572, 5680, 5855, 9431, 6924, 6134, 10837, 7471, 6608, 7289, 6859, 5560, 11716, 6858, 3274, 17125, 4754, 2861, 2257, 4640, 1601, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

1601 being today's entry, and it's off by 21 entries in the list 🤷 ?

[bacheshrew]

It looks like some good detective work has taken place and there is a workaround at least for this. Is anyone able to share what needs tweaking and where to get my code running again and returning historical data?

[bobo3k]

I still can't map the data datetimes correctly, but I found an interesting function in the disassembled tapo app (I don't speak java, I'm a php dev), maybe this can help understanding how the exchange is made... I can see a reference to an interval of 5, which wasn't expected. Anyway, I think the app isn't displaying the correct datetimes either, and maybe it would be worth it to wait for another update before doing anything... not sure

/ renamed from: q / public final void mo47832q() { long j = (long) 1000; Calendar instance = Calendar.getInstance(C5132x0.m15953d(this.f35770i)); C18563j.m60883e(instance, "it"); long currentTimeMillis = (System.currentTimeMillis() / j) j; instance.setTimeInMillis(currentTimeMillis); instance.set(13, 0); instance.set(12, (instance.get(12) / 5) 5); C18563j.m60883e(instance, "calendar"); long timeInMillis = instance.getTimeInMillis() / j; long j2 = timeInMillis - 86400; EnergyPowerDataResult value = mo47833r("day_power").getValue(); Long l = null; mo47821B(timeInMillis, 5, value != null ? Long.valueOf(value.getEndTimestamp()) : null); m40793x(new EnergyPowerDataParams(j2, timeInMillis, 5)); instance.set(12, 0); instance.set(13, 0); long timeInMillis2 = instance.getTimeInMillis() / j; long j3 = timeInMillis2 - 604800; EnergyPowerDataResult value2 = mo47833r("week_power").getValue(); if (value2 != null) { l = Long.valueOf(value2.getEndTimestamp()); } mo47821B(timeInMillis2, 60, l); m40793x(new EnergyPowerDataParams(j3, timeInMillis2, 60)); instance.setTimeInMillis(currentTimeMillis); instance.set(13, 0); instance.set(12, 0); instance.set(11, 0); long timeInMillis3 = (instance.getTimeInMillis() / j) + ((long) 86400); long j4 = timeInMillis3; m40790p(new EnergyPowerDataParams(timeInMillis3 - ((long) 604800), j4, 60)); instance.set(13, 0); instance.set(12, 0); instance.set(11, 0); instance.set(5, 1); instance.add(2, -2); m40790p(new EnergyPowerDataParams(instance.getTimeInMillis() / j, j4, 1440)); instance.setTimeInMillis(currentTimeMillis); instance.set(13, 0); instance.set(12, 0); instance.set(11, 0); instance.set(5, 1); instance.add(2, 1); instance.set(2, 0); m40790p(new EnergyPowerDataParams(instance.getTimeInMillis() / j, instance.getTimeInMillis() / j, 43200)); }

ps: I removed the "code" tag, because it was showing the code as 1 long string

[bacheshrew]

@bobo3k The timestamps appear to be in unix format

Perhaps PyP110.py could be amended to add a copy of getEnergyUsage(), renamed as getEnergyData()

    def getEnergyData(self, startT, endT, intervalT):
        URL = f"http://{self.ipAddress}/app?token={self.token}"
        Payload = {
            "method": "get_energy_data",
            "params":{ "end_timestamp": startT, "interval": intervalT, "start_timestamp": endT},
            "requestTimeMils": int(round(time.time() * 1000)),
        }
....snip....

Other folks on here would know a better way of doing this

[CameronInLondon]

There are a number of posts on the Tapo forum about firmware v1.1.2 and the date shift issue. https://community.tp-link.com/en/smart-home/forum/topic/587026 https://community.tp-link.com/en/smart-home/forum/topic/586720 https://community.tp-link.com/en/smart-home/forum/topic/586664

Not sure if it is worth coding a fix for this date shift issue or to wait for a bug fix from Tapo. Not sure how long we might wait.

[bobo3k]

@bacheshrew I know about the timestamps, but the problem is with the fact that the data returned is off compared to the timestamp... for example if you ask today's info from 2022-10-26 00:00 to 2022-10-27 00:00 it will show data from 27 to 28th... and we're not there yet. (look at the screenshot from the APP, it's faulty too). I'm pretty sure it's a bug they'll have to fix via a new firmware. My guess: we can only wait.

[bacheshrew]

Thanks @bob3k. Let's wait and see.

[tnmendes]

Clearly, there are several issues with the dates of the firmware update. I'm developing the app Watt for Smart Devices and the solution I have at the moment that matches android Tapo's app is: startTime = 1st day of the month of the previous month at zero hours EndDate= today with today's time

I think it is also necessary to pay attention to the GMT

I am also ignoring all the values in the array that are higher than today

[fishbigger]

Thanks all for your detective work on this. I feel it would be better to wait for a firmware update than to code in a temporary fix so I'm going to mark this as won't fix.

Let me know if there are any further comments on this :)

[CameronInLondon]

@fishbigger it seems that there is more than one issue raised in this thread. I see two clear issues and there might be more.

1) P110.getEnergyUsage() no longer returns historical data - can be fixed adding a new method to PyP110.py using @bobo3k code. 2) The timeshift bug, where the plug is returning today's data but with tomorrows date.

While I would agree that point (2) should be left for a Tapo bug fix. I am not sure that point (1) will be 'fixed' by Tapo, as it seems like that is the new way they are delivering the data.

[fishbigger]

Thanks @CameronInLondon. I don't have a P110 plug to test this with but if you could open a pull request with the suggested changes and test it I'd be happy to merge it into the module.