larsks / envoy-tools

Tools for interacting with the Enphase Envoy.
17 stars 3 forks source link

Proxy occasionally accepting xml files with partially duplicated data #1

Closed mddvul22 closed 8 years ago

mddvul22 commented 8 years ago

Hi Lars,

I’ve noticed that if the HTTP post to reports.enphaseenergy.com has any difficulties (every once in a while, I get errors (non-200 errors) in my proxy log:

Various errors I’ve seen before: error: [Errno 32] Broken pipe HTTPError: HTTP Error 503: Service Unavailable HTTPError: HTTP Error 400: Bad Request URLError: <urlopen error [Errno 111] Connection refused> URLError: <urlopen error [Errno 104] Connection reset by peer> URLError: <urlopen error [Errno 110] Connection timed out>

When this happens, the python proxy still gets a copy of the xml file deposited in the spool-dir. But the envoy fails to send it on to reports.enphaseenergy.com.

The result of this is that, 5 minutes later, the envoy retries, with a new XML file that contains the current data, as well as the data from the previous report.

Given that the envoy is going to resend data, it would be nice if the proxy intelligently handled those xml files that were not successfully handed off to Enphase by either removing them or moving them out of the way.

mddvul22 commented 8 years ago

Hi Lars, as I give this more thought, I'm not sure if the proxy should be responsible for dealing with this. When these errors do occur, there are successive xml files created that contain duplicate (or repeated sets of data). And as a result, I do end up with duplicated rows of data in my mysql database. However, on further reflection, I think moving the xml out of the way, and/or deleting it might be more drastic that leaving it like it is. To delete or move the xml file out of the way, would cause my mrtg graphing to show incorrect information for intervals where there wasn't an xml file to import. Since my mrtg graphing is based upon the most recent timestamped data in the mysql table, the duplicates actually aren't impacting my graphs, as things are currently designed. So, I think that the better option is for me to simply run some database queries occasionally, to retroactively remove duplicate rows from my tables.