Innaccurate database entries - also uploaded to pvoutput website

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Using the software with MSAccess database
2.
3.

What is the expected output? What do you see instead?
Data contained in database does not match that of csv file.  Database contains 
entries all day long, even during the night, where csv file only has data from 
0600-2000.  Extra data is being live uploaded to PVoutput website.  I see 
repeated data on PVoutput website after 1200. 

What version of the product are you using? On what operating system?
1.3.6, windows XP

Please provide any additional information below.

Original issue reported on code.google.com by Pbart...@gmail.com on 31 Mar 2011 at 1:06

[GoogleCodeExporter]

Some additional information, There seems to be a problem with the time.  The 
live plots on pvoutput are taking 2x as much time (ie. plots start at 1200 
intead of 0600).  I have my account on pvoutput set to 5 minute intervals and 
also the pv bean counter software set to 5 minutes.  

thanks,

Pete B

Original comment by Pbart...@gmail.com on 31 Mar 2011 at 1:36

[GoogleCodeExporter]

Forgot to add a link to my pvoutput pg, see this for an example:

http://www.pvoutput.org/intraday.jsp?id=1885&sid=1421&dt=20110331

thanks,

Pete B

Original comment by Pbart...@gmail.com on 31 Mar 2011 at 1:40

[GoogleCodeExporter]

Can you please upload the csv file for 31/03 and the pvoutput.log file produced 
on this day.

Original comment by DennisMackayFisher on 31 Mar 2011 at 8:48

• Changed state: Started

[GoogleCodeExporter]

Also the settings.xml file please

Original comment by DennisMackayFisher on 31 Mar 2011 at 8:49

[GoogleCodeExporter]

Here is the settings.xml file

Original comment by Pbart...@gmail.com on 31 Mar 2011 at 10:19

[GoogleCodeExporter]

Here is the log file.  There are quite a few errors.

Thanks a lot, I really appreciate the software you are developing!

Pete B

Original comment by Pbart...@gmail.com on 31 Mar 2011 at 10:22

[GoogleCodeExporter]

Here is the csv file.

Original comment by Pbart...@gmail.com on 31 Mar 2011 at 10:45

[GoogleCodeExporter]

The data on pvoutput for 31-3-2011 is now correct.  Earlier in the day it was 
incorrect, but later it started displaying it correctly.  The live data for 
30-3-2011 is still incorrect, you can see a repeating pattern from 1200-1800 
and 1800-2400.

Thanks for your help,

Pete B

Original comment by Pbart...@gmail.com on 1 Apr 2011 at 1:10

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

The error messages in the log file are pvoutput.org upload rejections. pvoutput 
prevents more than 60 updates per hour. If you exceed this the updates are 
rejected until an hou has passed. Another 60 updates are then accepted.

The incorrect dates would appear to be a timezone change perhaps at around 
6:30pm on 29/03. The shift appears to be around 6 hours. The database stores 
dates in GMT and converts to local time for display. If timestamps are written 
to the database in one time zone and the timezone is then changed. The display 
time will be different as one GMT time in one timezone appears as a different 
time in another timezone.

To correct the problem you need to recreate the entries in the database for 
29/03 and 30/03. To do this try the following:

Stop the service.

Set "First Full Day" to the oldest date for which data is available in the 
inverter.

Check "Reset First Full Day"

Uncheck "Enable Upload" in pvoutput.org settings.

Save the settings and restart the service.

This should cause all data available in the inverter to be reloaded into the 
database. This should fix the timezone change issue.

It may take 10 or 15 minutes to complete the updates. If you watch the 
"Archive" directory you should see the csv files being reloaded one at a time. 
When this is complete do the following:

Stop the service.

Uncheck "Reset First Full Day" (you don't want to reload every time the service 
starts.

Check "Enable Upload" and "Force Live Upload" in pvoutput.org settings.

Set "Live Upload Days" to 1.

Save the settings and start the service.

This should refresh the data for TODAY. If you encounter the 60 record update 
limit, let it run until the errors stop (no more than 1 hour).

Stop the service.

Set "Live Upload Days" to 2.

Save the settings and start the service.

This should refresh the data for YESTERDAY.

Stop the service.

Set "Live Upload Days" to 3.

Save the settings and start the service.

This should refresh the data for 2 days ago.

Continue until all available data hase been uploaded. 7 days is the limit for 
upload to pvoutput.org.

When complete stop the service, uncheck "Force Live Upload" and set "Live 
Upload Days" to the recommended setting of 2 days. Save the settings and 
restart the service.

"Force Live Upload" only works on one day at a time. This was done to avoid 
excessive 60 update limit errors while forcing data up to pvoutput.org.

Let me know how you go with this.

Dennis M-F

Original comment by DennisMackayFisher on 2 Apr 2011 at 12:32

[GoogleCodeExporter]

Dennis,

I'm giving your suggestions a try right now.  A couple thing I noticed:

1. Is it normal for PVBC to download all the data from the inverter from first 
full day to the current day  every time it runs, even when Reset First Full Day 
is not checked?  It seems to be dowloading the entire month everytime it 
connects to the inverter.

2. Why are there entries in the MS Access database during the night? These 
entries even show output from the inverter when there was none. (ie. my entries 
run from 0645-2300 but the CSV file only has ouptput data from 0645-1830)

Thanks,

Pete B

Original comment by Pbart...@gmail.com on 2 Apr 2011 at 4:37

[GoogleCodeExporter]

Dennis,

I had changed my timezone on pvoutput.org because I noticed it didn't have my 
correct location.  Does your software read that info from the pvoutput's 
website?

I reloaded the entire month back into the database to correct the timezone 
change.

The live update function would not work.  I had to delete the days on 
pvoutput.org and then check the enable auto backload to get the data reloaded.

The data is still messed up for days 29032011 and 30032011.  You can see the 
repeating pattern in the data files:

http://www.pvoutput.org/intraday.jsp?id=1885&sid=1421&dt=20110329

Now I am wondering if Pvoutput.org remembers that timezone change and is 
reapplying it even after reuploading those day's data?

Thanks,

Pete B

Original comment by Pbart...@gmail.com on 2 Apr 2011 at 5:11

[GoogleCodeExporter]

The timezone change I was reffering to was on the capture PC not pvoutput. I do 
not believe the "Country" setting makes any difference at pvoutput.

The csv file you uploaded was 31/03. I need to see the csv files for the 
corrupt days. Can you please upload the csv files for 29/03 and 30/03.

Also upload a zipped copy of the access DB.

I forgot to mention that you need to delete the days at pvoutput before 
reloading the data.

PVBC should not reload all data from the inverter unless "Reset First Full Day" 
is checked. I will check this out when I have a copy of the database.

Dennis M-F 

Original comment by DennisMackayFisher on 2 Apr 2011 at 7:37

[GoogleCodeExporter]

I did a little more investigating.  I replaced the access DB with the original 
empty one and reloaded all the data from the inverter.  The file size went from 
60MB to 1.4MB.  All the stored data during night hours is gone now.  Also PVBC 
is now only downloading today's data from the inverter, instead of the whole 
month.  I guess the DB was corrupt somehow.

The timezone change is a mystery to me, it hasn't been changed on the capture 
PC recently.

So, right now, the only mystery is why those two days are corrupt.  I'm 
attaching the DB and csv files to this msg.

Thanks,

Pete B

Original comment by Pbart...@gmail.com on 2 Apr 2011 at 8:09

[GoogleCodeExporter]

The files produced by your inverter are a little different to mine. To my 
knowledge until now all PVBC users were connecting to SB3000TL-20, SB4000TL-20 
or SB5000TL-20 inverters. These inverters output entries for the full 24 hours. 
Yous only outputs data for the active part of the day. 

The PVBC full day detection will not work correctly with your files as it is 
based on finding 24 hours of data for each day. It will see your data as 
"Incomplete".

I will find a fix , until then you will need to manually modify "First Full 
Day" to prevent constant inverter download of old days.

Dennis M-F

Original comment by DennisMackayFisher on 3 Apr 2011 at 7:56

[GoogleCodeExporter]

Dennis,

I have a SunnyBoy 3000-US inverter. 

Thanks,

Pete B 

Original comment by Pbart...@gmail.com on 3 Apr 2011 at 1:18

[GoogleCodeExporter]

Dennis,

If it will help, I've attached a whole month worth of inverter output files.

Thanks,

Pete B

Original comment by Pbart...@gmail.com on 3 Apr 2011 at 3:00

[GoogleCodeExporter]

The required fix will be in v1.4.1.3

This will be released in about 1 week.

Dennis M-F

Original comment by DennisMackayFisher on 11 Apr 2011 at 10:34

[GoogleCodeExporter]

Thanks Dennis,

I'll be looking for it.

Pete B

Original comment by Pbart...@gmail.com on 11 Apr 2011 at 11:41

[GoogleCodeExporter]

The change you require is included in 1.4.1.2. It is available now on the 
downloads page.

Original comment by DennisMackayFisher on 13 Apr 2011 at 9:03

[GoogleCodeExporter]

Dennis,

I installed the new version.  I noticed some errors modifying the Access 
Database.  Are they normal?  I've attached the log file.

Thanks,

Pete B

Original comment by Pbart...@gmail.com on 13 Apr 2011 at 11:24

[GoogleCodeExporter]

Some are normal however yours seems excessive. I suspect that PVBC will run 
correctly on the result, however I would like to inspect the database structure.

Can you please zip and upload your access database. I will check the structure 
for issues.

Is PVBC running correctly (capturing data and uploading to pvoutput)?

I expect you will get these errors whenever you restart PVBC, until you make a 
manual entry in the "version" table. This table should have 1 row (entry). The 
columns "Major", "Minor", "Release" and "Patch" should have values 1,4,1,2.

Can you open this table using Microsoft Access and change the existing values 
to these? This will prevent PVBC from constantly attempting to modify the DB 
structure.

Dennis M-F

Original comment by DennisMackayFisher on 14 Apr 2011 at 9:15

[GoogleCodeExporter]

I first received the errors with my current database.  I then tried using one 
of the backup empty databases and still got the errors.  I forced all archived 
data to be loaded into the new database by moving it out of the archive folder, 
into the PVRecords folder.  PVBC recognized it as new data and loaded it into 
the empty database.  PVBC ran today just fine, uploading to the pvoutput 
website.

I'll look at modifying the access database.  I'm not very familiar with Access, 
but I'll give it a try.

One other thing I noticed, my old database had a file size of 82.2MB after 
running for approx 6 wks.  When I reloaded all the old data into the new 
database, the file size was only 1.7MB.  After 24 hours the database is now 
8.7MB.  The file size seems to be growing very fast and will grow to over a GB 
less than a year.

The zip file attached to this msg contains both databases.  PVRecords_jet1 is 
the old database.  PVRecords_jet2 is the new one.

Thanks,

Pete B

Original comment by Pbart...@gmail.com on 15 Apr 2011 at 12:31

[GoogleCodeExporter]

The new database has the correct structure but as I suspected it needed the 
version table entry changed. It has been changed in the attached copy of your 
database.

You can reduce the database size using the access "Compact" utility. Turn off 
PVBC. Open the database using Access. Open the main drop down menu and choose 
"Manage" - "Compact and Repair". I use this on your old database and it reduced 
to less than 1MB.

Demmis M-F

Original comment by DennisMackayFisher on 16 Apr 2011 at 6:51

[GoogleCodeExporter]

Dennis,

Yes, that did the trick.

Everything seems to be working now.  I'll periodically compact the DB so it
doesn't take up as much space.

Thanks for all the hard work, I really like the software.

Pete B

Original comment by Pbart...@gmail.com on 16 Apr 2011 at 7:06

[GoogleCodeExporter]

Original comment by DennisMackayFisher on 17 Apr 2011 at 4:57

• Changed state: Verified