Time error

[harryverkooijen]

Johan,

Every day I god a time error. Bad request 400 After restart van Gott the problem is (for 1 day) over.

wat can i do?

see:

2020-11-02 10:12:58,stdout, - Bad request 400: Invalid Time [39:77]

2020-11-02 10:12:58,stdout, - Grott PVOutput response:

2020-11-02 10:12:57,stdout," - {'d': '20330000', 't': '39:77', 'v1': 800, 'v2': 1848.1, 'v6': 412.3} " 2020-11-02 10:12:57,stdout, - Grott send data to PVOutput :

2020-11-02 10:12:57,stdout," - No MQTT message sent, MQTT disabled " 2020-11-02 10:12:57,stdout, "pvipmtemperature": 349}}

2020-11-02 10:12:57,stdout," 3673, \"pv1current\": 36, \"pv2current\": 35, \"pvtemperature\": 323, " 2020-11-02 10:12:57,stdout," \"pvenergytoday\": 8, \"pvenergytotal\": 2659, \"pv1voltage\": 3001, \"pv2voltage\": " 2020-11-02 10:12:57,stdout," 23658, \"pvpowerout\": 18481, \"pvfrequentie\": 5001, \"pvgridvoltage\": 4123, " 2020-11-02 10:12:57,stdout," \"values\": {\"pvstatus\": 1, \"pv1watt\": 10803, \"pv2watt\": 12855, \"pvpowerin\": " 2020-11-02 10:12:57,stdout," {\"device\": \"HVE3A21041\", \"time\": \"2033-00-00T39:77:77\", \"buffered\": \"no\", " 2020-11-02 10:12:57,stdout, - MQTT jsonmsg:

2020-11-02 10:12:57,stdout, - pvipmtemperature: 34.9

2020-11-02 10:12:57,stdout, - pvtemperature: 32.3

2020-11-02 10:12:57,stdout, - pv2current: 3.5

2020-11-02 10:12:57,stdout, - pv2voltage: 367.3

2020-11-02 10:12:57,stdout, - pv1current: 3.6

2020-11-02 10:12:57,stdout, - pv1voltage: 300.1

2020-11-02 10:12:57,stdout, - pvgridvoltage: 412.3

2020-11-02 10:12:57,stdout, - pvfrequentie: 50.01

2020-11-02 10:12:57,stdout, - pv2watt: 1285.5

2020-11-02 10:12:57,stdout, - pv1watt: 1080.3

2020-11-02 10:12:57,stdout, - pvenergytotal: 265.9

2020-11-02 10:12:57,stdout, - pvenergytoday: 0.8

2020-11-02 10:12:57,stdout, - pvpowerout: 1848.1

2020-11-02 10:12:57,stdout, - pvpowerin: 2365.8

2020-11-02 10:12:57,stdout, - pvstatus: 1

2020-11-02 10:12:57,stdout, - pvserial: HVE3A21041

2020-11-02 10:12:57,stdout, - Grott values retrieved:

2020-11-02 10:12:57,stdout, - 2033-00-00T39:77:77

2020-11-02 10:12:57,stdout, - Grott data record date/time processing started

2020-11-02 10:12:57,stdout,

2020-11-02 10:12:57,stdout, - record layout : T060104

2020-11-02 10:12:57,stdout, - offset : 6

2020-11-02 10:12:57,stdout, - decrypt : True

2020-11-02 10:12:57,stdout, - Growatt new layout processing

2020-11-02 10:12:57,stdout, 000000000000000000000000000000000000000000000000000000000000000071d4

2020-11-02 10:12:57,stdout, 0051600000b220000000000000000000000000000000000230001117000000000000000000000

2020-11-02 10:12:57,stdout, 00000000000000015d0c280bff0000002d00594e2000000000000000040000060c00000005000

2020-11-02 10:12:57,stdout, 043002000001dfb1015002100001e930000000800000a63001e5e2b0143000000000000000000

2020-11-02 10:12:57,stdout, 002c000100005c6a0bb9002400002a330e59002300003237000048311389101b002200001f8c1

2020-11-02 10:12:57,stdout, 85645334132313034310000000000000000000000000000000000000000210000274d4d020000

2020-11-02 10:12:57,stdout, 005d0006010101044a50433641323731583200000000000000000000000000000000000000004

2020-11-02 10:12:57,stdout, - Growatt plain data:

2020-11-02 10:12:57,stdout, - Growatt data decrypted V2

2020-11-02 10:12:57,stdout, - decrypt : True

2020-11-02 10:12:57,stdout, - layout : T060104

2020-11-02 10:12:57,stdout, - Grott data record length 265

2020-11-02 10:12:57,stdout, - Grott automatic protocol detection

2020-11-02 10:12:57,stdout, \x72\x6f\x77\x61\x74\x74\x47\x72\x6f\x77\x61\x74\x74\x47\x72\x6f\x06\xb5

2020-11-02 10:12:57,stdout, \x77\x61\x74\x74\x47\x72\x6f\x77\x61\x74\x74\x47\x72\x6f\x77\x61\x74\x74\x47

2020-11-02 10:12:57,stdout, \x74\x74\x47\x72\x6f\x77\x61\x57\x74\x46\x63\x1f\x77\x61\x74\x74\x47\x72\x6f

2020-11-02 10:12:57,stdout, \x42\x72\x6f\x72\x77\x74\x74\x4c\x50\x6f\x77\x61\x74\x74\x47\x72\x6f\x77\x61

2020-11-02 10:12:57,stdout, \x6f\x2e\x2f\x54\x74\x47\x72\x6f\x77\x61\x74\x70\x47\x72\x69\x7b\x61\x74\x74

2020-11-02 10:12:57,stdout, \x61\x74\x74\x47\x72\x6f\x77\x61\x74\x75\x1a\x7e\x47\x7c\x9e\x74\x74\x47\x5f

2020-11-02 10:12:57,stdout, \x74\x4f\x72\x6f\x7d\x02\x74\x6a\x19\x59\x6e\x34\x61\x74\x74\x47\x72\x6f\x77

2020-11-02 10:12:57,stdout, \xfe\x7f\x34\x61\x54\x74\x47\x6f\x94\x67\x74\x74\x55\x47\x72\x71\xe4\x61\x74

2020-11-02 10:12:57,stdout, \x77\x42\x74\x74\x75\x45\x6f\x77\x29\x45\x67\xce\x62\x74\x77\x43\x74\x74\x58

2020-11-02 10:12:57,stdout, \x74\x74\x6b\x72\x6e\x77\x61\x28\x1e\x4c\xcb\x6f\x53\x61\x74\x5e\x74\x7c\x36

2020-11-02 10:12:57,stdout, \x47\x72\x6f\x77\x61\x74\x74\x47\x72\x6f\x77\x40\x74\x74\x60\x3f\x22\x75\x61

2020-11-02 10:12:57,stdout, \x27\x21\x24\x47\x35\x75\x43\x5f\x43\x50\x74\x74\x47\x72\x6f\x77\x61\x74\x74

2020-11-02 10:12:57,stdout, \x61\x74\x74\x47\x72\x6f\x77\x61\x74\x74\x47\x72\x6f\x77\x61\x74\x74\x47\x72

2020-11-02 10:12:57,stdout, \x00\x5d\x00\x06\x01\x01\x01\x04\x0d\x22\x2c\x41\x20\x46\x43\x76\x2a\x5d\x77

2020-11-02 10:12:57,stdout, - Growatt original Data:

2020-11-02 10:12:57,stdout," <socket.socket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('172.17.0.3', 56008), raddr=('47.91.67.66', 5279)> " 2020-11-02 10:12:57,stdout, - Growatt packet received:

[johanmeijer]

Looks like error in the time decoding. I am going to look at it. Thank you for the data that should be enough. PVOutput does not like the time value of 39.77 (I would do either I think)

After this error Grott stops processing? You have to restart (or auto restart will do this)?

There are no errors in processing the rest of the day? Time values are normal after this failure?

[harryverkooijen]

Johan,

Grott keeps working, only ever update later has de same error (and same fault in converting the time). Server.growatt.com works fine, only PVoutput refuse update. I think MTQQ date is also wrong, I don’t use it. All values looks fine, only the time part is wrong. Total stop Grott and restart, fix the problem, 1 day later it reappears.

I have TL3-s inverter, 1 month old.

Thanks,

Harry

Van: Johan Meijer notifications@github.com Verzonden: maandag 2 november 2020 14:39 Aan: johanmeijer/grott grott@noreply.github.com CC: Harry Verkooijen Harry@harryverkooijen.nl; Author author@noreply.github.com Onderwerp: Re: [johanmeijer/grott] Time error (#22)

Looks like error in the time decoding. I am going to look at it. Thank you for the data that should be enough. PVOutput does not like the time value of 39.77 (I would do either I think)

After this error Grott stops processing? You have to restart (or auto restart will do this)?

There are no errors in processing the rest of the day? Time values are normal after this failure?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/johanmeijer/grott/issues/22#issuecomment-720477726, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARRQEJCZG5FZUNOY46Y4GK3SN2ZA5ANCNFSM4THI4DYQ.

[johanmeijer]

@harryverkooijen the data/Time value in the record looks crippled. Seeing the fact the other values are ok it seems comming from the inverter.

I know that after te start of the communication (after grott starts orin the morning when the inverter starts sending) a lot of information is exchanged between the inverter and the growatt server. One of them is a time set command beging sent from growatt to your inverter.

If you specified blockcmd = True Grott will block all commands except the time set command.

Did you use blockcmd = True? If you do can you run it without that? Maybe there are other time sync commands I block that I should not block.

If this is the case I like to have some more information from the startup of the inverter in the morning (I supose this is one of the first records). If this is not the first record after the start I like to have more information from the communciation before this error. We can discuss later how and what.

What I can do in the coding is detect the Time (or data information is wrong) and replace that with the server time (That is also used if here is no time available in the record that is the case for some inverters). This does not solve the problem it just by-pass it. But testing for invalid datae/time values is not bad I think.

Do you have a shineWifi, shineLan or shineLink interface?

[harryverkooijen]

Johan,

I have shineWifi

The corrupt time/date value spontaneously comes after 2 hours of normal function. Initial after starting it is always correct. I can solve this issue by restarting Grott.

I use blockcmd = True , I shall remove this and look of the problem did not accrue anymore. I let you know.

Harry

Van: Johan Meijer notifications@github.com Verzonden: maandag 2 november 2020 15:43 Aan: johanmeijer/grott grott@noreply.github.com CC: Harry Verkooijen Harry@harryverkooijen.nl; Mention mention@noreply.github.com Onderwerp: Re: [johanmeijer/grott] Time error (#22)

@harryverkooijenhttps://github.com/harryverkooijen the data/Time value in the record looks crippled. Seeing the fact the other values are ok it seems comming from the inverter.

I know that after te start of the communication (after grott start of in the morning when the inverter starts sendin) a lot of information is exchanged between the inverter and the growatt server. One of them is that there is a time set command is sended from growatt to your inverter.

If you specified blockcmd = True Grott will block all commands except the time set command.

Did you use blockcmd = True? If you do can you run it without that? Maybe there are other time sync command I block that I should not block.

If this is the case I like to have some more information from the startup of the inverter in the morning (I supose this is one of the first records). If this is not the first record after the start I like to have more information from the communciation before this error. We can discuss later how and what.

What I can do in the coding is detect the Time (or data information is wrong) and replace that with the server time (That is also used if here is no time available in the record that is the case for some inverters). This does not solve the problem it just by-pass it. But testing for invalid datae/time values is not bad I think.

Do you have a shineWifi, shineLan or shineLink interface?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/johanmeijer/grott/issues/22#issuecomment-720513649, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARRQEJDRWXDS3RLY6BWMLNTSN3AOFANCNFSM4THI4DYQ.

[johanmeijer]

Hi Harry,

Thanks I am very curious if this helps.

I also have uploaded a t060104.json file in the example directory on github. If you copy this file into your main directory the time processing from the data record will be bypassed and the time of the system on which grott is running will be used. You have to restart grott to read the new processing values.

See this as a temporary fix.

I like to know more about the real issue but can understand if you do not want to put a lot of effort in this. In the next updated I will bypas the time processing if a no valid time occurs (this is also a bypass but does make the code more reliable).

Running with blockcmd = False also allows Growatt to update the firmware of your shineWife. Maybe that helps with resolving errors as well (if it is a known error). This can also be forced via de Growatt website. I run firmware: 1.7.7.7

From the growatt website:

[harryverkooijen]

Johan,

I already had firmware 1.7.7.7. today same problem, after 1 hour of correct logging the time get corrupted again. Stop Grott and restart solves (temporary) this issue. I have blockcmd =false. I cannot use your .json file because I use the docker Grott 2.2.2. version on Synology.

thanks for your assistance

Harry

[johanmeijer]

Ok. I didn't realised youre using the docker version. It is possible to copy the json file in the docker container but lets not go that route.

I will create a new grott version and container with the proposed change above.

I really do not understand why the time field is corrupted. It seems to be done by the growatt server / wifi stick (the print of the original data is what is comming from the network buffer and what is sent through by grott to the growatt server). I use that (de-crypt and decode it). I do not see any other data/time related data in the record.

The only thing we can do is ignore the data field if it contains no valid data. I think the growatt server is not using this date/time information but uses the of receipt. What will be strange if it concerns buffered records (records saved by shiniWifi if there is no connections).

[johanmeijer]

@harryverkooijen I have just published Grott 2.2.3 image on docker hub (ledidobe/grott:2.2.3). Can you test with this one?

I made the changes as described above. There is now a parameter time = auto/server in the .ini file (environmental to be used with docker container: -e gtime="server" in the docker run command ) .

The default is auto. This means that Grott tries to use the date/time from the inverter data record. If this is not availble or invalid then it will use the time of the server where grott is hosted. I extended the validation of the date/time and thinks it will always detect wrong values (I tested it with your record and that one is detected and handled ok now).

I think you can run with the default but if it still gives problems you can force Grott to use the server time by adding the gtime environmental at start up.

I also checked your data record was crippled during transmission or by Grott. The records are sent with a CRC code to check if it is not modified. This is not the case. It is sent with the strange date/time by the wifiShine module. So internally in the wifi stick (or inverter) something happens that invalidates the date/time.

As described above with restarting Grott a handshake between the growatt server and inverter (shinewifi) is started. One of the things that is beeing done during this handshake is setting the date/time. So every time you restart Grott the date/time will be set. But it will also be modified after a while apparently.

[harryverkooijen]

still time error. Timestamp is not ivallid annymore, but impossible. PV output refuse it.

I use the gtime=server command to see if this fix ik permanent.

2020-11-06 10:12:18	stdout	-  Bad request 400: Moon Powered
2020-11-06 10:12:18	stdout	- Grott PVOutput response:
2020-11-06 10:12:17	stdout	-  {'d': '20201106', 't': '01:12', 'v1': 6000, 'v2': 3576.7, 'v6': 414.6}
2020-11-06 10:12:17	stdout	- Grott send data to PVOutput :
2020-11-06 10:12:17	stdout	- No MQTT message sent, MQTT disabled
2020-11-06 10:12:17	stdout	383, "pvipmtemperature": 369}}
2020-11-06 10:12:17	stdout	"pv2voltage": 3685, "pv1current": 56, "pv2current": 54, "pvtemperature":
2020-11-06 10:12:17	stdout	"pvenergytoday": 60, "pvenergytotal": 3344, "pv1voltage": 3296,
2020-11-06 10:12:17	stdout	38356, "pvpowerout": 35767, "pvfrequentie": 4998, "pvgridvoltage": 4146,
2020-11-06 10:12:17	stdout	"values": {"pvstatus": 1, "pv1watt": 18457, "pv2watt": 19899, "pvpowerin":
2020-11-06 10:12:17	stdout	{"device": "HVE3A21041", "time": "2020-11-06T01:12:16", "buffered": "no",
2020-11-06 10:12:17	stdout	- MQTT jsonmsg:
2020-11-06 10:12:17	stdout	- pvipmtemperature:  36.9
2020-11-06 10:12:17	stdout	- pvtemperature:     38.3
2020-11-06 10:12:17	stdout	- pv2current:        5.4
2020-11-06 10:12:17	stdout	- pv2voltage:        368.5
2020-11-06 10:12:17	stdout	- pv1current:        5.6
2020-11-06 10:12:17	stdout	- pv1voltage:        329.6
2020-11-06 10:12:17	stdout	- pvgridvoltage:     414.6
2020-11-06 10:12:17	stdout	- pvfrequentie:      49.98
2020-11-06 10:12:17	stdout	- pv2watt:           1989.9
2020-11-06 10:12:17	stdout	- pv1watt:           1845.7
2020-11-06 10:12:17	stdout	- pvenergytotal:     334.4
2020-11-06 10:12:17	stdout	- pvenergytoday:     6.0
2020-11-06 10:12:17	stdout	- pvpowerout:        3576.7
2020-11-06 10:12:17	stdout	- pvpowerin:         3835.6
2020-11-06 10:12:17	stdout	- pvstatus:          1
2020-11-06 10:12:17	stdout	- pvserial:          HVE3A21041
2020-11-06 10:12:17	stdout	- Grott values retrieved:
2020-11-06 10:12:17	stdout	- date-time:  2020-11-06T01:12:16
2020-11-06 10:12:17	stdout	- Grott data record date/time processing started

[johanmeijer]

It stays strange what the inverter is doing with the time. Does gtime=server solved the problem (I expect that).

[johanmeijer]

@harryverkooijen any news?

[harryverkooijen]

Johan,

Grott is now running 3 days in a row without any error. I checked log file, but i think i can not detect if the inverter dispatched a wring time because of de use of the system time.

I did a little test with buffering, and here Grott also uses system time for bufferd records. It is maybe an improvement that Grott, for buffered records, uses the time in the package (corrupt or not). This makes more sense for the data in PVoutput, I think.

thanks for the update.

[harryverkooijen]

John,

I mentioned a little side affect after my buffer test. Where PVoutput received all buffered records with system time, The server.growatt received the buffered records with a time stamp 4 hours early. I assume grott does not alter record before rerouting them to the server.growatt so this must be a server.growatt issue.

[johanmeijer]

Indeed grott doesn't change the record content. So the time stamp 4 hours earlier in growatt server is probably a setting at the website. Did you set your location correct (at the plant)? I set it at plant definition time. I can not find at the website where this can be changed afterwards.

I do find a possibility at the inverter setting to change date/time maybe that helps (you might need to disabled blockcmd to use this).

You are right it does not make sence to send buffered records to pvoutput (and even to MQTT I think) if the server time is used. I will look for a way to handle that. I am now thinking about sent it with the time in the data record (if it has a valid date/time format) and make it possible, by adding an option, to disable sending buffers (at least for PVOutput, in de MQTT JSON Message there is a field that indicates it is a buffered record allready).

Ideas are welcome!

[harryverkooijen]

I have one wish for extending Grott.

I don’t want to use MQTT for data acquisition, but I would like to have a backup of my data, locally stored. I run Grott on a Synology NAS and I have plenty of storage. I would like a option to write de MQTT records to a file. 1 file every day, txt, data coma separate. to a folder (in Docker you can share a folder in a container with a folder on the NAS). preferably sorted in monthly folders. This is than a backup of my data for further use or to process in third party apps.

I think when I use an additional APP that het MQTT record's collects and put in a file I can accomplice the same, but I prefer a 1 app solution.

Harry

[johanmeijer]

Hmm I going to think about this. What I do think that is complex is the file handling (especially if you want to do some kind of management, like collect per week / month etc). But it also a challenge to expand my programming skills.

I do understand you do not want an additional tool but have you thought about a time managed database like influxdb. I write to influx via nodered and create graphics with grafana. That is very easy to implement. I can give grott a interface to write directly to influx.

Why I use mqtt and nodered is the possibilty to combine information. By example I write (with an own developed P1 smart meter interface) the energy usage values to mqtt and combine this with the solar information (I have 2 seperate sets of solar panels and inverters) and write it to PVOutput.org.

[harryverkooijen]

I dived into the matter and I agree that an influx interface is preferable. As Grott get an interface to influx I will go with that. There is a docker container for influx, I shall look for that.

Harry

[johanmeijer]

Nice. I will try to add that! We can also create a container with both influx and grott in it (but lets take it step by step).

[johanmeijer]

I changed te time processing and the buffer records are only send to PVOoutut.org (and/or MQTT) when:

• sendbuf = True (in .ini or gsendbuf = True environment variable @ docker startup)

and

• date/time format from the data record is valid.

Do you think we can close this Issue?

It is more clear we open a new issue with a request for INFLUXDB support.

[johanmeijer]

@harryverkooijen I created Issue #29 for creating an influxdb interface.

I think this one can be closed.

[harryverkooijen]

Yes, This one can be closed.

Thank you.