search

jasonacox / Powerwall-Dashboard

Grafana Monitoring Dashboard for Tesla Solar and Powerwall Systems
MIT License
270 stars 57 forks source link

How to Back Up Other Tables in the powerwall Database #420

Closed aesculus closed 5 months ago

aesculus commented 6 months ago

I have added some other tables to the powerwall database for monitoring my heat pumps. When I use the backup routine that is part of the Powerwall-Dashboard these tables are not backed up.

Can you tell me how I can alter the backup.sh or other file(s) to also include my special tables into the backup?

aesculus commented 5 months ago

Jason: Just a bump in case you did not see this request.

youzer-name commented 5 months ago

Pressed for time at the moment, but I also have extra databases in Influx and I'm pretty sure they're all being backed up by the regular backup process. I'll confirm and get back to you later tonight or some time tomorrow. (US EST).

jasonacox commented 5 months ago

@youzer-name is correct, the default script backs up everything in the influxdb folder. If you are using the powerwall influxdb instance for your data, your data is being backed up.

This is the tar command:

 tar -Jcvf ${BACKUP_FOLDER}/Powerwall-Dashboard.$STAMP.tar.xz influxdb

Backup script here:

https://github.com/jasonacox/Powerwall-Dashboard/blob/a59e11fcaa2395ff569d83d4cfa4f47adff3f90b/backups/backup.sh.sample#L13-L41

aesculus commented 5 months ago

Hi Jason and @youzer-name : Maybe I don't recognize what the backup structure looks like. To confirm I added three additional "tables" to the powerwall database. Maybe these are the 12,13 and 14 shards in the raw items below? I was sort of expecting a sql like table listing.

Here is the output from the influx backup command:

database powerwall /var/lib/influxdb/backups 2024/01/11 18:26:54 backing up metastore to /var/lib/influxdb/backups/meta.00 2024/01/11 18:26:54 backing up db=powerwall 2024/01/11 18:26:54 backing up db=powerwall rp=autogen shard=1 to /var/lib/influ xdb/backups/powerwall.autogen.00001.00 since 0001-01-01T00:00:00Z 2024/01/11 18:26:55 backing up db=powerwall rp=raw shard=2 to /var/lib/influxdb/ backups/powerwall.raw.00002.00 since 0001-01-01T00:00:00Z 2024/01/11 18:26:55 backing up db=powerwall rp=raw shard=12 to /var/lib/influxdb /backups/powerwall.raw.00012.00 since 0001-01-01T00:00:00Z 2024/01/11 18:26:56 backing up db=powerwall rp=raw shard=13 to /var/lib/influxdb /backups/powerwall.raw.00013.00 since 0001-01-01T00:00:00Z 2024/01/11 18:26:57 backing up db=powerwall rp=raw shard=14 to /var/lib/influxdb /backups/powerwall.raw.00014.00 since 0001-01-01T00:00:00Z 2024/01/11 18:26:58 backing up db=powerwall rp=pwtemps shard=5 to /var/lib/influxdb/backups/powerwall.pwtemps.00005.00 since 0001-01-01T00:00:00Z 2024/01/11 18:26:58 backing up db=powerwall rp=vitals shard=3 to /var/lib/influxdb/backups/powerwall.vitals.00003.00 since 0001-01-01T00:00:00Z 2024/01/11 18:26:58 backing up db=powerwall rp=kwh shard=4 to /var/lib/influxdb/backups/powerwall.kwh.00004.00 since 0001-01-01T00:00:00Z 2024/01/11 18:26:58 backing up db=powerwall rp=daily shard=9 to /var/lib/influxdb/backups/powerwall.daily.00009.00 since 0001-01-01T00:00:00Z 2024/01/11 18:26:58 backing up db=powerwall rp=daily shard=11 to /var/lib/influxdb/backups/powerwall.daily.00011.00 since 0001-01-01T00:00:00Z 2024/01/11 18:26:59 backing up db=powerwall rp=monthly shard=10 to /var/lib/influxdb/backups/powerwall.monthly.00010.00 since 0001-01-01T00:00:00Z 2024/01/11 18:26:59 backing up db=powerwall rp=grid shard=6 to /var/lib/influxdb/backups/powerwall.grid.00006.00 since 0001-01-01T00:00:00Z 2024/01/11 18:26:59 backing up db=powerwall rp=pod shard=7 to /var/lib/influxdb/backups/powerwall.pod.00007.00 since 0001-01-01T00:00:00Z 2024/01/11 18:26:59 backing up db=powerwall rp=alerts shard=8 to /var/lib/influxdb/backups/powerwall.alerts.00008.00 since 0001-01-01T00:00:00Z 2024/01/11 18:26:59 backup complete: 2024/01/11 18:26:59 /var/lib/influxdb/backups/var/lib/influxdb/backups/meta.00 2024/01/11 18:26:59 /var/lib/influxdb/backups/var/lib/influxdb/backups/powerwall.autogen.00001.00 2024/01/11 18:26:59 /var/lib/influxdb/backups/var/lib/influxdb/backups/powerwall.raw.00002.00 2024/01/11 18:26:59 /var/lib/influxdb/backups/var/lib/influxdb/backups/powerwall.raw.00012.00 2024/01/11 18:26:59 /var/lib/influxdb/backups/var/lib/influxdb/backups/powerwall.raw.00013.00 2024/01/11 18:26:59 /var/lib/influxdb/backups/var/lib/influxdb/backups/powerwall.raw.00014.00 2024/01/11 18:26:59 /var/lib/influxdb/backups/var/lib/influxdb/backups/powerwall.pwtemps.00005.00 2024/01/11 18:26:59 /var/lib/influxdb/backups/var/lib/influxdb/backups/powerwall.vitals.00003.00 2024/01/11 18:26:59 /var/lib/influxdb/backups/var/lib/influxdb/backups/powerwall.kwh.00004.00 2024/01/11 18:26:59 /var/lib/influxdb/backups/var/lib/influxdb/backups/powerwall.daily.00009.00 2024/01/11 18:26:59 /var/lib/influxdb/backups/var/lib/influxdb/backups/powerwall.daily.00011.00 2024/01/11 18:26:59 /var/lib/influxdb/backups/var/lib/influxdb/backups/powerwall.monthly.00010.00 2024/01/11 18:26:59 /var/lib/influxdb/backups/var/lib/influxdb/backups/powerwall.grid.00006.00 2024/01/11 18:26:59 /var/lib/influxdb/backups/var/lib/influxdb/backups/powerwall.pod.00007.00 2024/01/11 18:26:59 /var/lib/influxdb/backups/var/lib/influxdb/backups/powerwall.alerts.00008.00

youzer-name commented 5 months ago

Look in the backup tar file. You'll see it backs up the data folder from Powerwall dashboard. Inside that would be folders for any other databases in that instance of InfluxDB.

image

Everything you show above is for the powerwall database however, so I assume you actually added measurements to the Powerwall database rather than creating a new database. InfluxDB doesn't really have tables, so you're not going to find a table structure in the export. The backup process simply makes a copy of the files that InfluxDB is using, so if you restore those files and spin up InfluxDB, you've restored the data.

This is not the same as for example running a MySQLdump command on a MySQL database and then running that SQL file to recreate the database, tables, and data.

Also, not really my thing, but for those who appreciate it, this is #420...

aesculus commented 5 months ago

OK. Thanks for the clarification. Yes I just created 3 new "measurements" and write them into the pypowerwall db every minute.

So measurements are not something that is "exposed" during backup but is embedded in the database? And the only influxdb database that I have is the one Powerwall-Dashboard created and I was addending.

jasonacox commented 5 months ago

I asked ChatGPT to describe the file system of InfluxDB. It might be helpful to you: https://chat.openai.com/share/cab2e62c-9d38-4e70-8aab-98e5c3b6f917

The official docs from Influxdb https://docs.influxdata.com/influxdb/v2/reference/internals/file-system-layout

aesculus commented 5 months ago

Jason: Thanks for the added information but my feeble brain was not able to resolve my issue. The ChatGPT even made it worse because it alluded to "measurements" which is the nearest equal to tables that I can come up with.

Here is the structure of my powerwall database after I added my stuff. Note the first 3 are mine and the last two are the standard ones provided by you.

name: HD21E10576_0_T name: HD21E10576_1_T name: HD22G10192_0_T name: http name: weather

Anyway I poked around the tar file for awhile and was finally able to find some of my stuff embedded in the binaries so I feel better now.