Question: How to access historic power data?

[gaiking-uk]

Hi @jasonacox,

First of all I just want to say thanks for creating this project -- while the iPhone / Android apps are "pretty good" and let you perform all the standard user tasks (turning plugs on/off, dimming lights, etc) I have found the power monitor interface pretty limited (with only the ability to see how much power I am using right now and then a daily totals for previous usage), so am really excited to start using tuyapower to start collecting much more useful and detailed info! 👍🏼

---

STAGE 1: Setup + Initial Access --✅ Complete

• Full disclosure: While I dabble in several languages I haven't actually written python before, so apologies in advance for any dumb questions or obvious misses that I might make!
• That being said, am happy to report that I was able to install phyton, create the Cloud project on the Tuya IoT platform, authorise tuyapower to access my account, scan + detect my devices and run a successful poll command! 🎉

  tuyapower.devicePrint(PLUGID,PLUGIP,PLUGKEY,PLUGVERS)
  > Switch On: True
  > Power (W): 57.800000
  > Current (mA): 373.000000
  > Voltage (V): 246.700000
  > Projected usage (kWh):  Day: 1.387200 Week: 9.710400  Month: 42.078400

---

STAGE 2: Exporting detailed usage data -- ❓ In progress / guidance requested

• I noticed on the IoT platform, the device logs are pretty frequent, with the device publishing it's live power usage approx. every minute (or 1440 times per day)

• I don't have a RaspberryPi or other device that will online and able to poll my smart plugs in real time 24/7
• So, am wondering how instead I can query the device's previous usage reports and then bulk download these [e.g. my laptop runs a scheduled task once per day, polls the last 24 hours of usage and downloads the ~1440x minute-by-minute usage numbers (ideally!)
• Would you be able to give me any pointers on which command(s) I need to use please (and a super-quick summary of how I can export the returned data as a CSV 🙏🏼)

[jasonacox]

Hi @gaiking-uk ! Thanks for the note.

To compute energy, you need to perform samples of current power (wattage) over time and determine the "area under the curve" (ie. mathematical integration). So let's say you measure the power p(x), and you do that n times over a period of t hours, you would compute the energy using an equation like this:

$$\int_{1}^{n} p(t) dt$$

or

$$\sum (p{1}...p{n}) / t$$

Practically, if you had a load that ran for 5 hours and you tested it every hour like this:

p(1) = 100W p(2) = 200W p(3) = 300W p(4) = 400W p(5) = 500W

Using the above:

Energy = (100W + 200W + 300W + 400W + 500w) / 5 Energy = 1500W / 5 = 300Wh = 0.3kWh

As you can see, you need to do accumulation of power samples over time to determine the energy. All the "energy" plugs I have seen (I own several) do not have a memory feature to record that accumulation on board, so they use the Cloud to record the power (W) samples. The Cloud is then able to perform the math to produce the energy consumption.

The tuyapower project doesn't provide any functions to access the cloud. It simply reads the current state of the plug. To get the data recorded in the cloud, you can use the tinytuya library. See this https://github.com/jasonacox/tinytuya#tuya-cloud-access - I don't know the exact commands you would need to send to the cloud to get the power history. Perhaps it shows up in the logs and you can pull those to extract the data. Let us know if you figure it out!

Keep in mind, API calls to the cloud are very limited and even once a day may put you over the free threshold. Alternatively, as you say, you could set up a Raspberry Pi or Linux box to record the power throughout the day (e.g. using telegraf, influxdb and grafana for graphing).

[gaiking-uk]

Hi @jasonacox, thanks for the quick reply and explanation + additional info. Having read through the info you kindly provided and doing some thinking about my own aims and how to achieve this I wanted to reply and re-cap your info (check my understanding) and share my thoughts on ways forward...

Context / Use Case

Thanks for the info about how to project future/average loads, while I am sure this will be really useful in general and/or for future readers, for me specifically I am keen to retrieve the raw/actual historic power usage as one of the main things I want to do right now is sort out my PC power-saving/sleep config (e.g. how much power it draws when active but idle, in S3 sleep, S0 sleep, hibernating (+boot), etc and as the PC is plugged into the smart plug I want to measure, this creates two problems for me specifically:

1. While using multi-sampling to project an average power usage is 100% valid, it will give a blended answer (whereas I really want to gather data about different situations, separately)
2. Using the host machine to collect this data means I can only collect data while the system is active

---

Option 1: ❌ Use tuyapower to extract historic data from device [Not possible]

• Basically this is not possible as the tuyapower commands that are available are only able to poll live power usage (i.e. the power being used right at this moment)
• More fundamentally though [and to make clear this is not a tuyapower issue] the smart plugs themselves do not store any historic usage data, so it won't be possible to extend tuyapower and add this functionality later
• Any data shown in the Smart Life app for example is from the Tuya cloud, not from the device itself

Option 2: 💲 Use a Raspberry Pi (or other device) to live-sample data over-time

• TBF, I have considered getting a Raspberry Pi on and off for years (and am not necessarily ruling out that option) however two the main things that have generally put me off are...
  1. price (A Raspberry Pi 4B is £150 on Amazon and yes I realise other/cheaper models are available so would need to spend some time checking which model(s) I could order based on my likely intended uses
  2. unfamiliarity / learning curve: as I mentioned earlier I program across several languages ([cmd] Shell, PowerShell, VBA, CSS, JavaScript, AHK) but have no knowledge of python or C/C++

Option 3: 💲Access the historic information in Tuya's cloud via API

• Again, thanks for the heads up on the potential cost warning for this option
• I had an inkling that the free API usage would be limited and so going down this road might requiring paid model a but hadn't thought the limits would be as severe as a single call using up all my free allowance
• Ultimately this project is a "nice to have" and in this day and age (when it already feels like everyone is moving to the subscription model / asking for a monthly fee) I doubt I could justify the cost of (guessing) £5/month or £60/year just to get access a log of how much power my 4 smart plugs use

Option 4: 🐌 Access the historic information in Tuya's cloud via alternative methods

• Falling back on my existing skillset, I could write a script in PowerAutomate Desktop to manually scrape this data from the web
• Provided Tuya didn't detect and block the relatively high http traffic, I could go the page mentioned in my initial post (showing the last ~15 mins of usage data), extract the html table, click the "next" button and repeat
• It looks like you can access this fine-level usage info for the previous 7 days via the web interface (so I couldn't access all the historic data in full, but I could start building my own database of this info)
• The process is quite slow though, capturing a page may take ~15 seconds which may sound good (capturing 15 mins of data at 60x speed) but if you extrapolate this out, it means it would take 24 mins to extract a day's worth of info
• It's also worth bearing in mind, that you computer is essentially "unusable" while the script is running, yes background tasks are possible but the UI will be constantly in use so you won't be able to interact with the PC while the script is running / extracting info

---

Summary

• Giving what I want to achieve, probably options (2) Raspberry Pi and (4) Manually scraping data make the most sense, so I may look into those options further
• tuyapower can only access live power usage info but as said above, this is due to a limitation of the technology / because the smart plugs themselves do not store any data
• If I have mis-understood any of the above (or you want to respond to any points / add any additional thoughts), please feel free to comment and add any extra info or questions -- otherwise I hope this serves as a useful discussion/guide for anyone in the future that may want to do something similar along these lines and thanks once again for your help!

[jasonacox]

Well written @gaiking-uk ! Thanks for taking the time to post this. It will be helpful for others.

A few thoughts for consideration:

• Don't underestimate yourself. With your existing coding knowledge, I suspect you could learn python in less than a day.
• Yes, RPIs are way to expensive these days. I sure hope they come down in price. However, keep in mind, you don't need the latest. You can often find used older unit for less than $30. I have an old Raspberry Pi B that runs most of my automation.
• Have fun!

[gaiking-uk]

✅ Main discussion ends here

---

NB: The above note is intended for the benefit of any future readers, to advise that the main content of the thread likely stops here, however there may be additional comments/discussion below...

[gaiking-uk]

👍🏼Thanks @jasonacox - I appreciate the vote of confidence!

I often label myself as a "dabbler developer", i.e. I have a decent background in programming (studied it at uni) but for better or worse have ended up using using a mini-plethora of languages, with knowing/using a little of everything from AHK, to PowerShell, to writing my own registry edits and cmd scripts [being a perfectionist and wanting to tweak how my computer config has definitely not helped, lol] 😀

My point is, I often find myself having to get to grasps with a new language and while it doesn't phase I also know first-hand that it can be a bit frustrating when you just begin, i.e. when you know very little! I know this isn't exactly the place, but just wanted to share some anecdotes since your last comment...

---

Early experiments with python

I did some experimenting with trying to build upon my initial "do one off poll" script... Again, as a complete python beginner, I had some stumbling blocks -- trying to do a simple loop seemed to suggest I needed to install one of several module options....

... the turtle module was an interesting one to me, as when I googled how to export as csv I read answers on stackoverflow that said I needed to install the pandas module -- am sensing a bit of a running, animal-based, but also "everything is a module" theme 🙂.

---

Update on my tuyapower / python script

As a quick(er) and dirty solution, falling back on my existing knowledge to export the data to a CSV, I wrapped my script in a shell/batch cmd which appends the .py output to a CSV for me...

# Get-SmartPlugPower-v2.py
# Prints the live power usage (watts) of the device specified every X seconds
# > Laptop,17/04/2023 13:03:10,30,0.0001534

import time
from decimal import *
from datetime import datetime
import tuyapower
SECS = 30

DEVICENAME  = '*************'
PLUGID      = '********************'
PLUGIP      = '192.168.1.***'
PLUGKEY     = '*************'
PLUGVERS    = '3.3'

i = 0
while i < 1:           # i never increased so always true 
    time.sleep(SECS)   # Query device every X seconds
    (on, w, mA, V, err) = tuyapower.deviceInfo(PLUGID,PLUGIP,PLUGKEY,PLUGVERS)
    now = datetime.now()                                # captures datetime
    currDateTime = now.strftime("%d/%m/%Y %H:%M:%S")    # formats as string
    KWH = "{:.10f}".format(w / 60 / (60 / SECS) / 1000)
    w = "{:.1f}".format(w)
    print(currDateTime + ',' + DEVICENAME + ',' + w + ',' + KWH)

:: 1-line cmd command to run python script and capture output in CSV
python Get-SmartPlugPower-v2.py >> path-to-csv

Finally I plug all the data into Excel, pivot it, and (thanks to your help / with the use of the tuyapower module) am now able to start generating and building up a more useful report on my power usage...

The above is still very much a work-in-progress but I have some other things I need to work on so won't be able to play any more on this for a little while so thought I would share my progress so far (in case any newbie coders want to build upon my script). Please note though at some point I would ideally extend / improve the above by:

• Direct CSV export
• Polling/exporting multiple devices (via a object array / loop rather than just copy + pasting code
• Turning into a function
• Error handling if device can't be reached, etc

---

Initial thoughts on python

Literally just a rapid-fire brain dump on my initial experimentations with the language, will be interesting to see how my own thoughts of python evolve over the next few weeks/months, but for now...

• I miss the variable symbols that languages like PowerShell use
• (related) it is a bit interesting not knowing necessarily if I am referencing a variable, function or module... e.g. in the code above, not sure what would've happened if I called my current time variable time instead of now
• I REALLY miss brackets and braces, on that basis python seems most like AHK (AutoHotKey) where things like if statements are loops are controlled more by indentation than by start and end ( ) { } (and just from experience, AHK can be a real nightmare to debug because of that sometimes)
• More... "fragmented" (modular) than most languages I've used, which I guess makes sense assuming that this is to make the "core" python as lightweight as possible and rely on the developer to import whatever they need, but again (from a newbie perspective) it was a little odd / frustrating having to import a module just to get the current time, or be able to use a decimal number, etc.... don't get me wrong though, I definitely see the logic it is just a matter of getting used to that design approach

---

(Lastly) A quick follow-up on RPis

Our discussion re-ignited my curiosity for getting a raspberry pi. My main intended uses for it would be....

• Running tuyapower to poll devices
• Running Home assistant
• Plex server (with h264 and h265 content)

I was leaning towards the significantly cheaper RPi 3, but as part of my research I stumbled upon this comparison which kinda depressed me a little, lol...

Image credit: magpi.raspberrypi.com

So I would prefer a RPi 4 for sure (especially after seeing the above) but managed to find a RPi 3 for about £45 (so 1/3 of the price of a RPi 4!). Checking if that would let me do the above was a little less straight-forward when looking into the Plex (and especially h265) side, as depending on which article I read online, comments ranged from "RPi 3 definitely won't be able to handle h265 video" to "RPi 3 can handle h265 1080p@30fps fine!"

I think I am going to have a punt on a RPi 3B+ (hopefully it won't be a lemon). The back order time is 2-4 weeks but hopefully I can start playing about with moving my tuyapower stuff over to a RPi in a month or so!

[jasonacox]

Nice! Decent first start and don't disagree with your observations about python. My first languages were basic, C and C++ so I can't help but think {this way;} but I have really grown to love python. It is powerful and the indentation becomes a visual construct that allows me to easily navigate the code. I have some pointers, in case it is interesting...

i = 0
while i < 1:

this is the same as:

while True:

I've never used decimal. It basically converts a binary floating point into a string and does manipulation on that to get more exact floating point calculations out to crazy big precision. Is that what you were needing? Python can do floating point math without doing that. You can cast most strings as floats using float():

w = "55"
print(float(w)/33)
1.6666666666666667

I have been playing around a lot with ChatGPT to find quick answers to "how to do this in python" (and many other languages). I also use it to convert from one language to the next. I also have it help me tune my code at times to get insights. I pasted your code into ChatGPT and asked it to remove decimal, it came up with this:

import time
from datetime import datetime
import tuyapower

SECS = 30

DEVICENAME = '*************'
PLUGID = '********************'
PLUGIP = '192.168.1.***'
PLUGKEY = '*************'
PLUGVERS = '3.3'

while True:
    # Query device every X seconds
    on, w, mA, V, err = tuyapower.deviceInfo(PLUGID, PLUGIP, PLUGKEY, PLUGVERS)
    now = datetime.now()  # capture current datetime
    currDateTime = now.strftime("%d/%m/%Y %H:%M:%S")  # format as string
    kwh = round(w / 60 / (60 / SECS) / 1000, 10)
    w = "{:.1f}".format(w)
    print(f"{currDateTime},{DEVICENAME},{w},{kwh}")
    time.sleep(SECS)

You might play with ChatGPT a bit too. You can ask things like "how do you do a loop in python?". Lots of fun and always a good learning exercise for me. But in any case, GREAT JOB on plowing in to python. Have fun!!! 😄

[gaiking-uk]

Hi Jason, thanks for your message!

i = 0
while i < 1:

Lol, you are of course totally right - and in hindsight I guess maybe that made me seem a bit more amateur than I am (or at least how I think I am, lol) 🤣... Just for super quick context / trying to make my coding skills not look quite so laughable, this did initially start life out as:

SECS = 30

i = 0
while i < 20160:    # Loop will run for 7 days max (i.e. 2 * 60 * 24 * 7)
    [..]
    i+=1

... but mid-way through I decided the 7-day arbitrary max life was a bit pointless, and so just deleted the i+=1 and changed 20160 --> 0, mainly just as a quick hack to make it an infinite loop 😇

---

Yeah so the decimal thing came about because of wanting to include KwH, plus I was sampling tiny usage numbers -- after all, by literal definition a KwH is "1000's of watts per hour" and I'm measuring "watts per 30 seconds" 🙂 I found initially that when the numbers (as floats) got small, python started writing them out as 8.54167e-05 (i.e. scientific notation) and the stackoverflow.com said the answer was to convert them to decimal to fix this - I don't think I'd looked at string/print formatting by this point, but may be worth re-visiting so thanks!

---

ChatGPT

Hmmm lol, I don't know if you watch LTT | Linus Tech Tips at all? but for the past month or two there have been so many amazing announcements about ChatGPT (and AI in general) that I now kind of feel like Luke (from LTT) does, as in I am simultaneously all of...

• Insanely excited about the recent GIANT leaps that have been made
• Extremely concerned about the morale implications, such as it explicitly choosing to knowingly lie to humans in order to achieve it's goal
• Annoyed with people already milking/exploiting it (to be clear, not you)
• Feeling a bit overwhelmed/fatigued with just how many announcements there are about ChatGPT on a daily basis now
• Concerned that films like "I am mother" and "iRobot" (where AI basically uprises, tries to kill everyone and create a new harmonious society, because it calculates that's the "optimum" thing to do) have gone from "complete science fiction" to "this could happen any day now" 🙂
• But still, super intrigued how I could better use it as an info source!

Joking aside (although, partly wasn't even joking / we're definitely all gonna die! lol) -- Yeah, I did play around with chatGPT for coding a month or 2 ago but wasn't super-impressed (the PowerShell it gave me didn't quite work right and it made some minor mistakes, etc). Interestingly though, you have prompted me to re-assess again today and...

• At the time: 🤔 MEDIOCRE -- My initial thoughts were that this would be "GREAT for getting 95% of the way there" but wasn't sure how much it would help in practice, as the stuff I do professionally involves doing more complex coding for O365 / SharePoint - and it's generally the last 5% that takes most of my time anyway
• In hindsight: 😍 AMAZING -- You make a REALLY good point given my current python learning / extremely limited knowledge in there, when generating 95% of the code for me, would literally save me hours! -- good thinking bat man! 🤩

---

Specific ChatGPT reflections

1. Interesting that it took out the ( and ) as didn't know you could do that! 👍🏼 on, w, mA, V, err = tuyapower.deviceInfo(PLUGID, PLUGIP, PLUGKEY, PLUGVERS)

2. It did a round( ) on the KwH number? EDIT: Ah, I see it used 10 as a second parameter / rounded to 10dp, that makes a lot more sense! 🤣 kwh = round(w / 60 / (60 / SECS) / 1000, 10)

3. Oooh, I like what it did with the print line... print(f"{currDateTime},{DEVICENAME},{w},{kwh}")

... in my defence, I did do something similar to this initially but python padded the , with spaces and I wanted to get rid of these. So, without knowing any better after a bit of answer-searching on S/O, went down the clunky concatenation route, however I like this way a lot better!

[jasonacox]

I'm right there with you!

I find it rather interesting how "human" ChatGPT can be (including lying like a human, being over confident, or even hallucinating). But how much of that is the training set? If we load all of human knowledge into the LLM, does that mean it also takes in all of our evils too? Likely... hum... what could possibly go wrong? 😂

I'm incurably optimistic. There are definitely questions about AI and concerns as you mention, but my spidey sense is picking up a force multiplier. Using it in the right way can amplify me, not diminish me. I can use it to learn, to explore, to brainstorm, to enhance and to augment what I do. So much potential and usefulness is here and it is already delivering. Worth exploring IMHO. 😉

Regardless... keep learning! 👍

[gaiking-uk]

I do like the notion of being "incurably optimistic"! And while I certainly don't consider myself a pesimist, my current view on ChatGPT is something like:

"This is...

• Definitely going to be game-changing, potentially seeing as a revolutionary as say mobile phones, within that...
• 85% going to really beneficial helping people work better, find info more easily, save time... But also even in more directly beneficial ways, like better cancer screening, disease detection, etc
• 13% risk from corporate greed/overreach, companies using AI to mine their insanely immense mountains of data and come up with freakishly accurate models for how to market to every person, knowing every persons wants, fears, etc
• 2% chance that (very likely by accident) it all goes HORRIBLY WRONG -- Like I say, the plot of "I Am Mother" is an AI that is tasked with improving human quality of life... She saves millions of embryos in cryo-storage, then kills everyone on the planet, does planetary-scale repair and rejuvenation, reclaiming deserts and regrowing the rainforest, etc and then when complete restarts the human race" -- to the humans that are born, a level of heinous genocide utterly unthinkable... to the AI, "yes, billions of people died but trillions will be born in the new world and have a better quality of life than any human ever has"... My point being, all it takes is for someone to expand it's thinking just that bit too far, give it a little too much power and the world ends before ever realising the genie you just let out of the bottle!"

Gaiking, 2023 🙂

BESIDES, in the history of mankind, it's not like we've ever mis-used technology to try and attack / destroy each other! 😀 ... I mean years ago the world was choking with all the smoke from coal power stations. Then we learnt how to split the atom and thus enabling clean, green, nuclear energy! -- I mean it's not like we ever used nuclear technology to attac.... oh... wait... nevermind!

But other than that though, yes, going to be sooo good! 🙂

---

PS - I will say though, again in all seriousness, that I think AI is the answer to one of my longest-running problems I've never managed to solve namely...

• As I tend to spread myself fairly thin across a load of different technologies, languages and problems-to-be-solved.. I come in contact with a huge collection of useful information on different topics (everything from key cmd line switches for ffmpeg, to how to build custom context menus for different types of files, from how to restore all the .ps1 files I accidentally deleted from my OneDrive to the specific fix I need to do to my laptop when it gets this obscure bug every 12 months or so where it completely fails to detect the bluetooth hardware)
• However, when it comes to accessing the previously really useful information again, 12 months later it can be incredibly hard to track down... I've tried OneNote, custom databases, Excel repositories, online knowledge systems but it's really hard to get right to that golden nugget of info you know exists, but can't remember where it is
• That's where I think AI / ChatGPT could come in, and like I say be an absolute game-changer as it's basically like a person with intelligence, that knows everything (kind of), and has the ability to tailor it's responses to exactly what you want to know by remembering everything that important to you!

[gaiking-uk]

Thanks again for your help Jason. Closing the ticket as the core issue is resolved (and to help keep the number of open/active to a minimum)... Am sure you will get another tome or two from me in the future though! 😀

Good chatting with you! Martin