DiedB / Homey-SolarPanels

Connects production statistics of a wide range of PV inverters to Homey
https://apps.athom.com/app/it.diederik.solar
GNU General Public License v3.0
52 stars 37 forks source link

Growatt integration not working #105

Closed EricBorland closed 1 year ago

EricBorland commented 4 years ago

Hi there,

I just got a Growatt Inverter and successfully setup the WIFI module so I can review all data in server.growatt.com. I also could successfully login and pair my inverter into Homey. However, the power indicators are always "-" without showing any result. There is no error, just no metrics shown (see attached picture).

I have tried to do the requests your library does using postman and both the login and the getPlant work great (even in this last one I'm already able to get the data I'd like Homey to show) but the getInverter shows "{ msg: "501", success: "false" }"...

I am doing something wrong? could it be that the API has changed? I have reviewed the API used by the website server.growatt.com and it's slightly different... If that's the case and many users are like me, I'd be happy to contribute to integrate with the new Growatt API.

Let me know your thoughts and thank you very much for the app!

IMG_3099

DiedB commented 4 years ago

@pro-sumer Could you have a look?

pro-sumer commented 4 years ago

Maybe this weekend.

For me the integration is still working (but I'm using the Homey 5.0 TestFlight beta App, if that matters)

DiedB commented 4 years ago

I don't think the app version matters, in rare cases maybe the Homey software version could make a difference. But I think it is something else at play here.

@EricBorland could you submit an error report through the app settings screen?

DiedB commented 4 years ago

Schermafbeelding 2020-05-16 om 20 52 24

@pro-sumer Here is the app output

pro-sumer commented 4 years ago

@EricBorland Can you please post the (redacted) results of the two succeeding calls, so I can have a look whether those are similar to mine?

(make sure you at least redact your account name, password, email, plant ID, and device serial number)

EricBorland commented 4 years ago

I don't know what you mean by two succeeding calls, but the point is that the call GET /newInverterAPI.do?op=getInverterData&type=1&date=2020-05-18&id=######## return always: { "msg": "501", "success": "false" }.

I believe it's because of the type of datalogger or inverter I have... If it's okay for you, when I have time, I'll try to develop the api.js endpoints for mine... Moreover, I'd like to retrieve the battery level, the savings, the current power consumption, etc.

pro-sumer commented 4 years ago

I meant these two calls:

And then finally:

Can you please verify that the id you use is a deviceSn in the deviceList returned by the newPlantAPI call?

Maybe having a different datalogger/inverter indeed requires different API calls/parameters. Please let us know what you discover. If you use their mobile App a proxy acting as a MITM might help figuring out what exact API call is needed.

PS: I just had such a 501 when trying to log in, but the second time it worked with the same credentials...

DiedB commented 4 years ago

@EricBorland Do you have an update?

Frastabyte commented 4 years ago

Good day all,

I have the same issue (Growatt inverter and Homey only showing - symbol instead of values) and noticed that Zonnepanelen is just updated to 4.0.13. Does this also include a fix for this bug?

DiedB commented 4 years ago

No, this has not been addressed yet. We don't really know what would be causing it, but I think it's only a small subset of users having this problem.

pro-sumer commented 4 years ago

@Frastabyte Would you be able to help debug this using a REST client?

Frastabyte commented 4 years ago

@pro-sumer I am new to Homey and Github sure would be willing to participate.

pro-sumer commented 4 years ago

@Frastabyte Are you familiar with JavaScript code and can you perform the requests mentioned in this post with a REST client?

https://github.com/DiedB/Homey-SolarPanels/issues/105#issuecomment-630248638

EricBorland commented 4 years ago

Hi there,

Sorry for the late response. I've been really busy lately. I think the issue is related to the version of Datalogger or inverter, maybe I have a newer or older version and that's why the API works differently.

Here are the result of the calls you requested:

https://server.growatt.com/newLoginAPI.do?userName=USERNAME&password=PASSWORD

{ "back": { "data": [ { "plantName": "Home", "plantId": "######" } ], "service": "1", "quality": "0", "isOpenSmartFamily": 0, "totalData": {}, "success": true, "user": { "uid": "", "userLanguage": "en", "inverterGroup": [], "timeZone": 8, "lng": "", "dataAcqList": [], "type": 0, "password": "######################", "isValiPhone": 0, "kind": 0, "mailNotice": true, "id": ######, "lastLoginIp": "########", "phoneNum": "", "approved": false, "area": "Europe", "smsNotice": false, "isAgent": 0, "token": "", "nickName": "", "parentUserId": 0, "customerCode": "", "counrty": "Spain", "isPhoneNumReg": 0, "createDate": "2020-05-07 03:13:05", "rightlevel": 1, "appType": "c", "serverUrl": "", "lat": "", "lastLoginTime": "2020-07-02 18:11:41", "roleId": 0, "enabled": true, "agentCode": "#####", "inverterList": [], "isValiEmail": 0, "accountName": "#######", "email": "#################", "company": "", "activeName": "", "codeIndex": 1, "appAlias": "Borland", "isBigCustomer": 0, "noticeType": "" }, "msg": "", "app_code": "0" } }

https://server.growatt.com/newPlantAPI.do?op=getAllDeviceListThree&pageNum=1&pageSize=15&plantId=PLANTID

{ "plantMoneyText": "0.4/€", "optimizerType": 0, "ammeterType": "0", "storagePgrid": "0", "todayEnergy": "2.5", "storageTodayPpv": "1192", "invTodayPpv": "0", "totalEnergy": "268.1", "nominalPower": 0, "todayDischarge": "0", "Co2Reduction": "0", "isHaveOptimizer": 0, "storagePuser": "1192", "useEnergy": "0", "totalMoneyText": "37.53", "nominal_Power": 1620, "deviceList": [ { "lost": false, "location": "", "datalogSn": "#########", "deviceSn": "#########", "deviceStatus": 12, "pCharge": "0", "activePower": 212, "deviceAilas": "Growatt 5KW", "deviceType": "storage", "storageType": "3", "eChargeToday": "0", "apparentPower": 282, "capacity": "63 %", "energy": "268.1" } ] }

https://server.growatt.com/newInverterAPI.do?op=getInverterData&type=1&date=2020-07-02&id=DEVICESN

{ "msg": "501", "success": false }

As I told, everything looks normal until that last request...


In the meantime, I've modified your app to work with the api used by server.growatt's site and added some extra info that I require for my personal needs, this is the result:

IMG_3331 IMG_3332

Let me know if you need anything else.

Cheers.

pro-sumer commented 4 years ago

@EricBorland Interestingly you get "deviceType": "storage" while I get "deviceType": "inverter".

What kind of changes did you have to make to support a storage device?

(Since I don't have such a device I cannot investigate the API for it and the Growatt API is not published anywhere as far as I know)

pro-sumer commented 4 years ago

@Frastabyte What kind of device do you have? Inverter/Battery/...?

EricBorland commented 4 years ago

I have an Off-Grid Inverter and Pylontech Battery connected to it.

I actually had to change it quite a bit since the API (urls, auth, response schemas) are completely different... Maybe we should add another driver for those inverters? Moreover, I needed the info about the battery level, storage charging/production, etc so I can automate things depending on that so I've added few capabilities more. If that's okay with you, I can cleanup the code to make a PR (can't promise a due date, though).

pro-sumer commented 4 years ago

That indeed seems to be a completely different device compared to my simple inverter.

@DiedB What would be the best way to support multiple different devices from the same brand?

EricBorland commented 4 years ago

We could have two api.js files and use one or another depending on the device type, however, capabilities should be different as well and I'm not sure if it is possible within the same driver

pro-sumer commented 4 years ago

What API endpoints did you have to use?

Maybe I can try those in a REST client to see whether they support my simple inverter as well.

DiedB commented 4 years ago

We could have two api.js files and use one or another depending on the device type, however, capabilities should be different as well and I'm not sure if it is possible within the same driver

If we implement their off-grid branch, we would need a different driver for sure.

Frastabyte commented 4 years ago

@pro-sumer Since about one month I have a Growatt 3000 MTL-S inverter that connects my solar panels. It came with a USB dongle connected to the bottom of the inverter that connects wireless to a ShineLanBox that connects wired to my home internet router. I use one single subnet at home. I have no issues connecting from any device (Android app or Windows via a web browser) or any locations to my Growatt dashboard to see the information of my solar setup. Just the plugin in Homey seems to have issues connecting. I have the Homey Pro (Early 2019) now for like 2 months and it is running on version 4.2.0.

Unfortunately I have no experience with Java and/or the POST or GET commands. I did a quick google search but I am not sure how to proceed

EricBorland commented 4 years ago

If we implement their off-grid branch, we would need a different driver for sure.

I could make PR for that if you'd like to include it. Just I can't promise a deadline though...

pro-sumer commented 4 years ago

@DiedB Can we get logging from @Frastabyte somehow?

Frastabyte commented 4 years ago

Would it help if we do a quick MS Teams meeting somewhere in the next few days? Just let me know if and when would suit you.

pro-sumer commented 4 years ago

Would it help if we do a quick MS Teams meeting

Sorry, I'm not comfortable with that.

Let me think whether there's an alternative option to investigate your issue (preferably one where you don't have to share your Growatt credentials).

pro-sumer commented 4 years ago

@Frastabyte Let's first check whether your login works:

  1. Please install the HomeyScript App and calculate the MD5 hash of your password as described in issue 65.
  2. Run this script in the online editor (replace USERNAME and HASH with your real credentials) and post the result here (after redacting sensitive info like your username, password/hash, and email address).
const username = "USERNAME"
const password = "HASH"
const result = await fetch('https://server.growatt.com/newLoginAPI.do', 
  { method: 'POST', 
    headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
    body: `userName=${username}&password=${password}`
  }
)
const body = await result.json()
return body
Frastabyte commented 4 years ago

Good evening Rob,

So what I did:

The outcome of this was:

Script returned: { back: { success: false, msg: '502' } }

Please let me know if there is anything else I can do.

pro-sumer commented 4 years ago

success: false, msg: '502'

So the login fails.

Please let me know if there is anything else I can do.

  1. Please temporarily change your password to hapsnurk on the Growatt website
  2. Run this script again with the hash 64d4832469b0d59d81e73c17e78232b4
  3. Check what the Solar Panels App reports with these credentials
pro-sumer commented 4 years ago

PS:

  • MD5-ed the password and replaced the 0's with c's,

Note that you should not replace every zero by a 'c':

  1. Split the regular MD5 hash up in pairs of 2 characters
  2. For each pair only replace the first character by a c if it is a 0
  3. Join the pairs again
Frastabyte commented 4 years ago

Sorry, my mistake. I replaced the "first zero's" of every pair and now it succeeds. The return answer is

Script returned: { back: { data: [ [Object] ], service: '1', quality: '0', isOpenSmartFamily: 0, totalData: {}, success: true, user: { uid: '', userLanguage: 'ho', inverterGroup: [], timeZone: 1, lng: '', dataAcqList: [], type: 0, password: '25d55ad283aa40caf464c76d713cc7ad', isValiPhone: 1, kind: 0, mailNotice: true, id: 480978, lastLoginIp: '100.120.96.9', phoneNum: '0612345678', approved: false, area: 'Europe', smsNotice: false, isAgent: 0, token: '', nickName: '', parentUserId: 0, customerCode: '', counrty: 'Netherlands', isPhoneNumReg: 0, createDate: '2020-05-20 18:17:01', rightlevel: 1, appType: 'c', serverUrl: '', lat: '', lastLoginTime: '2020-07-05 23:08:22', roleId: 0, enabled: true, agentCode: '', inverterList: [], isValiEmail: 0, accountName: 'MYREALUSERNAME', email: 'INFO@MYREALUSERNAME.nl', company: '', activeName: 'MY FIRST AND LAST NAME', codeIndex: 1, appAlias: 'MYREALUSERNAME', isBigCustomer: 0, noticeType: '' }, msg: '', app_code: '0' } }

Does this help?

pro-sumer commented 4 years ago

Does this help?

Unfortunately not (yet).

Can you please try again with this return statement instead?

return body.back.data[0]

(this should show your plantId which we're going to use in the next call)

Frastabyte commented 4 years ago

Yes that works, I see my plant name and plant ID. Would you need the full data?


Script returned: { plantName: 'Familie NAME', plantId: '3095XX' }

pro-sumer commented 4 years ago

Nice, that means that step 1 (of 3), log in, works!

Time for step 2... (getting plant details)

Please try this script now (with instructions as before):

const username = "USERNAME"
const password = "PASSWORD"

const loginResult = await fetch('https://server.growatt.com/newLoginAPI.do', 
  { 
    method: 'POST', 
    headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
    body: `userName=${username}&password=${password}`
  }
)
const loginResultJSON = await loginResult.json()
const cookies = loginResult.headers.raw()['set-cookie'].map(directive => directive.split(';')[0]).join(';')
const plantId = loginResultJSON.back.data[0].plantId
const getPlantDataResult = await fetch(`https://server.growatt.com/newPlantAPI.do?op=getAllDeviceListThree&pageNum=1&pageSize=15&plantId=${plantId}`,
  {
    method: 'GET',
    headers: { 'Content-Type': 'application/x-www-form-urlencoded', Cookie: cookies }
  }
)
const getPlantDataResultJSON = await getPlantDataResult.json()

return getPlantDataResultJSON

Please post the result (but redact the serial number, which you will need in step 3)

Frastabyte commented 4 years ago

This is the output I got after adjusting the username and password. It is not a sunny day today over here ;-)


Script returned: { plantMoneyText: '2/€', optimizerType: 0, ammeterType: '0', storagePgrid: '0', todayEnergy: '10.2', storageTodayPpv: '0', invTodayPpv: '888.8', totalEnergy: '778.7', nominalPower: 3000, todayDischarge: '0', Co2Reduction: '0', isHaveOptimizer: 0, storagePuser: '0', useEnergy: '0', totalMoneyText: '155.74', nominal_Power: 3465, deviceList: [ { lost: false, location: '', eToday: '10.2', datalogSn: 'NAC49XXXXX', deviceSn: 'JFE09XXXXX', deviceStatus: 1, type: 0, powerStr: '0.89kW', bdc1Soc: 0, eTodayStr: '10.2kWh', deviceAilas: 'JFE09XXXXX', deviceType: 'tlx', bdc2Soc: 0, type2: 1, power: '888.8', energy: '778.7' } ] }

pro-sumer commented 4 years ago

This is interesting:

deviceType: 'tlx'

For me it's:

deviceType: 'inverter'

For @EricBorland it was:

deviceType: 'storage'

Looks like we found a third category... (and a second one that's not supported?)

pro-sumer commented 4 years ago

@Frastabyte Let's try the third step anyway, though I expect it to fail in your case.

Please try this script (after filling in the correct values; SERIALNO should be the value of deviceSn from the previous script):

const username = "USERNAME"
const password = "PASSWORD"
const serialNo = "SERIALNO"
const date = "2020-07-06"

const loginResult = await fetch('https://server.growatt.com/newLoginAPI.do', 
  { 
    method: 'POST', 
    headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
    body: `userName=${username}&password=${password}`
  }
)
const cookies = loginResult.headers.raw()['set-cookie'].map(directive => directive.split(';')[0]).join(';')
const getProductionData = await fetch(`https://server.growatt.com/newInverterAPI.do?op=getInverterData&id=${serialNo}&type=1&date=${date}`,
  {
    method: 'GET',
    headers: { 'Content-Type': 'application/x-www-form-urlencoded', Cookie: cookies }
  }
)
const getProductionDataJSON = await getProductionData.json()

return getProductionDataJSON
Frastabyte commented 4 years ago

Yes, that is true, it failed with the deviceSn, datalogSn and deviceAilas as the value set in const serialNo


Script returned: { msg: '501', success: false }

pro-sumer commented 4 years ago

@Frastabyte Since I don't have such an inverter I'm afraid you will have to take it from here...

Two things you could try:

  1. Try different values for the type parameter in the newInverterAPI.do endpoint call
  2. Install a proxy (which captures network requests) to find out which endpoint is used
Frastabyte commented 4 years ago

Hi Rob,

Holiday season is closing in and I have very limited time. I will setup a proxy soon but at this moment I lack time. Will come back soon on this.

Didiervu91 commented 4 years ago

I’m having the same problem with a new Growatt installation, the TL-XE. No data is being shown in the Homey app. Is there anything I can do to help? Is there a fix yet?

pro-sumer commented 4 years ago

Had a second look at the API using @Didiervu91's post on the Homey Community Forum.

My Growatt plug-in is using:

The Growatt ShinePhone App now uses this for my inverter:

As you can see their App is still using the "old" newInverterAPI and not something like newTwoInverterAPI for my inverter. I tried that in a REST client anyway, but got a 404 with a Chinese error message.

@Didiervu91 Can you please do this?

  1. Play a bit more with the ShinePhone App with HTTP Catcher active? Try to get some calls that contain the word "inverter" in the URL's path or op=getInverterData in the URL parameters.
  2. Check the value of field deviceType in the result of the newTwoPlantAPI.do?op=getAllDeviceList call?

For me the value is "inverter", but based on "TL-XE" it might be "tlx" for you like for @Frastabyte?

Didiervu91 commented 4 years ago

When I get direct info from the inverter through ShinePhone it goes to http://server-api.growatt.com/newTlxApi.do?op=getTlxParams&id=INVERTER-ID

The response from http://server-api.growatt.com/newPlantAPI.do?action=getUserCenterEnertyDataTwo

Is { "plantNumber": 1, "todayProfitStr": "€2.4", "isHaveFormulaMoney": true, "nominalPowerStr": "2.8kWp", "eventMessBeanList": [], "todayValue": "9.8", "formulaCo2Str": "12.2kg", "powerValue": "0.0", "monthValue": "12.2", "powerValueStr": "0kW", "todayStr": "9.8kWh", "monthStr": "12.2kWh", "alarmValue": 0, "totalProfitStr": "€2.9", "yearStr": "0kWh", "deviceDetail": {}, "treeValue": "1.0", "monthProfitStr": "€2.9", "treeStr": "1", "formulaCo2Vlue": "12.2", "yearValue": "0.0", "totalStr": "12.2kWh", "deviceTypeMap": { "STORAGE": 0, "TLXH": 0, "MIX": 0, "SPA": 0 }, "plantType": 1, "todayUnit": "kWh", "totalValue": "12.2", "statusMap": { "faultNum": 0, "onlineNum": 0, "offline": 1 }, "nominalPowerValue": 2835.0, "formulaCoalStr": "4.9kg", "plantId": ######, "formulaCoalValue": "4.9", "plantStatus": 0 }

Which has all the required data I believe.

When going to

http://server-api.growatt.com/newTwoPlantAPI.do?op=getAllDeviceList&language=5&plantId=PLANT-ID I get

{ "plantMoneyText": "2.4/€", "optimizerType": 0, "ammeterType": "0", "storagePgrid": "0", "todayEnergy": "9.8", "storageTodayPpv": "0", "invTodayPpv": "0", "totalEnergy": "12.2", "nominalPower": 2500, "todayDischarge": "0", "Co2Reduction": "12.16", "isHaveOptimizer": 0, "storagePuser": "0", "useEnergy": "0", "totalMoneyText": "2.93", "nominal_Power": 2835, "deviceList": [{ "lost": true, "eToday": "9.8", "location": "", "datalogSn": "NAC5A2426H", "deviceSn": "#INVERTER-ID", "deviceStatus": "-1", "type": 0, "powerStr": "0kW", "eTodayStr": "9.8kWh", "bdc1Soc": 0, "deviceAilas": "#INVERTER-ID", "deviceType": "tlx", "power": "0", "type2": "1", "bdc2Soc": 0, "energy": "12.2" }] }

So yes, tlx indeed

Didiervu91 commented 4 years ago

Sorry, forgot the response from newTlxApi:

{ "newBean": { "trakerModel": 0, "optimezerList": [], "wselectBaudrate": 0, "fwVersion": "AL1.0", "restartTime": 60, "batTempLowerLimitD": 0, "batTempLowerLimitC": 0, "location": "", "treeID": "ST_#INVERTER-ID", "addr": 81, "bmsCommunicationType": 0, "priorityChoose": 0, "startTime": 60, "bagingTestStep": 0, "bdc1Version": " -0", "timezone": 8, "plantname": "", "batParallelNum": 0, "level": 4, "modbusVersion": 305, "imgPath": "./css/img/status_gray.gif", "userName": "", "power": 0, "serialNum": "#INVERTER-ID", "pDischarge": 0, "dtc": 5100, "portName": "port_name", "energyMonth": 0, "tlxSetbean": null, "powerMax": "", "modelText": "S01B00D00T00P0FU01M0019", "status": -1, "eToday": 9.800000190734863, "vnormal": 1000, "vbatStartForDischarge": 0, "bdc1Sn": "XXXXXXXXXX", "sysTime": "", "record": null, "batSeriesNum": 0, "comAddress": 1, "plantId": 0, "manufacturer": " PV Inverter ", "communicationVersion": "ZABA-0001", "model": 72057594289651737, "treeName": "#INVERTER-ID", "lost": true, "vbatWarnClr": 0, "vbatStopForDischarge": 0, "powerMaxTime": "", "lastUpdateTime": { "time": 1599653735000, "minutes": 15, "seconds": 35, "hours": 20, "month": 8, "year": 120, "timezoneOffset": -480, "day": 3, "date": 9 }, "children": [], "statusText": "tlx.status.lost", "liBatteryFwVersion": "0", "energyMonthText": "0", "updating": false, "countrySelected": 0, "id": 0, "bdc1Model": "0", "vbatWarning": 0, "liBatteryManufacturers": "0", "deviceType": 1, "bmsSoftwareVersion": "0", "dataLogSn": "NAC5A2426H", "innerVersion": "ALAA0101", "energyDayMap": {}, "strNum": 0, "alias": "#INVERTER-ID", "batTempUpperLimitC": 0, "batTempUpperLimitD": 0, "batteryType": 0, "powerMaxText": "", "eTotal": 12.199999809265137, "tcpServerIp": "8.211.39.169", "vbatStopForCharge": 0, "groupId": -1, "lastUpdateTimeText": "2020-09-09 20:15:35", "mppt": 513, "pCharge": 0, "parentID": "LIST_NAC5A2426H_22", "bctMode": 0, "bctAdjust": 0, "pmax": 2500 }, "inverterType": "MIN 2500TL-XE" }

pro-sumer commented 4 years ago

Thank you for posting the results for your inverter!

My Growatt plug-in is using:

It looks like both this API and the newer "Two" API now provide the required data "eToday" and "power" in the newPlantAPI.do endpoint. So maybe the plug-in can use this instead of the third endpoint to get production data. It should support both my "inverter" and your "tlx" inverter then, I think.

I'm planning to experiment with this at a future time, but don't know when.

Didiervu91 commented 4 years ago

Okay I tested it out and I've changed the url from getInverterProductionData to ${plantURL}?action=getUserCenterEnertyDataTwo and data to { currentPower: Number(response.body.powerValue), energyToday: Number(response.body.todayValue) } and it worked immediately. Perhaps you can test these changes for your inverter to see if it works?

pro-sumer commented 4 years ago

I think by using that call you will get the total values for your "plant" (home) if you have more than 1 inverter.

While that will be fine for most users, I'm currently experimenting with using newPlantAPI.do filtered for a specific inverter.

pro-sumer commented 4 years ago

@Didiervu91 Can you please try this updated API implementation and let me know whether it works for your "tlx" inverter?

pro-sumer commented 4 years ago

@Didiervu91 When do you have time to check this?

Didiervu91 commented 4 years ago

I’ll try to do it this afternoon!

Op 19 sep. 2020 om 15:47 heeft Rob notifications@github.com het volgende geschreven:

 @Didiervu91 When do you have time to check this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.