Backup creation failed with alternat DB_PORT

[sanderplug]

Hi, getting Backup creation failed, please check the dsmr_backend logfile. with DB_PORT set to 55432, instead of the normal 5432 value. My postgres database is running on my synology where port 5432 was already occupied. I had changed the port in the docker configuration to 55432 and dsmr-reader is working fine.

However I can not make backups, the following message is shown in the UI of dsmr reader: "Backup creation failed, please check the dsmr_backend logfile. "

In the dsmr_backend logfile the following is shown:

[2019-11-17 11:03:46,275] CRITICAL  - Unexpected exit code (1) for backup: b'could not find a "pg_dump" to execute\npg_dump: [archiver (db)] connection to
 database "dsmrreader" failed: could not connect to server: Connection refused\n\tIs the server running on host "10.0.0.240" and accepting\n\tTCP/IP conne
ctions on port 5432?\n'                                                                                                                                   
[2019-11-17 11:03:46,308] ERROR    b'could not find a "pg_dump" to execute\npg_dump: [archiver (db)] connection to database "dsmrreader" failed: could not
 connect to server: Connection refused\n\tIs the server running on host "10.0.0.240" and accepting\n\tTCP/IP connections on port 5432?\n'                 
NoneType: None '  

Could it be that the DB_PORT value is not populated to the backup routines and how can this be fixed?

[evtk]

checking in to report some sort of the same issue. I did not change the port value, but I do have the same error in my dashboard.

Where did you find this in the logs exactly?

[sanderplug]

I think I found the logs in the /tmp directory.

[xirixiz]

I`ll verify the backup strategy this weekend

[xirixiz]

Do you have the backups folder mounted with the correct security settings? --volume ${SOME_LOCAL_FOLDER}:/dsmr/backups:rw

Can you run: docker exec -t dsmrdb pg_dumpall -c -U dsmrreader > dsmrreader.sql Can you see the data in the exported file then?

I don't have any errors using the default port, so I think the issue you two have are different to eachother. I`ll look into the port issue first. Within the docker container, you're still using 5432?

[evtk]

In the end my problem turned out having picked 'latest' for the postgres container image, while is should have picked 11.6 alpine. Version 12 is not yet compatible.

I found a way to get debugging information by tweaking the debug settings in the environment settings. Though I was never able to build up a log file in dsmr_backed. I watched the output receiving in the terminal in synology docker view.

[Xitro01]

Still an issue, when will there be support voor V12 of postgres?

[xirixiz]

There is, not PG13. Be more clear please. What is it you're trying to achieve.

[Xitro01]

There is, not PG13. Be more clear please. What is it you're trying to achieve.

I would like to have my daily back-ups, but they're not working. Got the exact same issue as above.

[rogierlommers]

Same here; started about 10/15 days ago. It used to run fine (I'm using postgres 12:latest).

I'm getting above message in the GUI. In fact, I have backups disabled:

So not sure what process is triggering these backup failures?

[xirixiz]

Ok, that's something I can look at. If you're using PG12 and a recent DSMR Docker image, the backup should be running fine. I`ll reopen this issue and verify some things.

[Xitro01]

I'm running PG_VERSION | 13.0-1.pgdg100+1 (which is postgres:latest). I probably have to downgrade to 12, to make it work, right? Any idea when 13 will be supported?

[xirixiz]

Yes, indeed. PG13 isn't supported in Alpine. Not sure when it will be supported.

You can verify it here (edge branch selected in the url below): https://pkgs.alpinelinux.org/packages?name=postgres*&branch=edge

[xirixiz]

@rogierlommers it must be something else, I just tested with PG12 and the backup is working fine. I also see backups in the mounted folder.

Do you mount a volume/folder with the DSMR container? I have something like this configured:

--volume ${DOCKER_DATA}/dsmr/backups:/dsmr/backups:rw \

@rogierlommers did you also check the mentioned logfile? It might be a permission issue.

I also know some people run a scheduled backup outside of the DSMR container, so on the Docker host itself (example based on docker run, but to stop and start you can also use docker-compose ofcourse):

#!/bin/bash
docker stop dsmr
docker exec -t dsmrdb pg_dumpall -c -U dsmrreader | /bin/gzip > <somepath>/dsmr/backups/dsmrreader.$(date +%F).sql.gz
docker start dsmr

## Only keep the latest 3 backups
find <somepath>/dsmr/backups -mtime +3 -delete

[rogierlommers]

@xirixiz , stupid question, but when I get this: Backup creation failed, please check the dsmr_backend logfile. Where exactly (in the container) can I see these logfiles? because the tailing the docker container (dockter tail -f dsmr) does not privde any additional info.

[xirixiz]

It shows the location of the logfiles on container startup.

I think the initial issue has been resolved by Dennis, running DSMRDB on a different port as the ENV vars are now populated to set the settings in Python.

[rogierlommers]

Damn; must have overseen that. Sorry. So, the backend logs shows a bunch of these lines:

[2020-11-02 19:58:21,694] CRITICAL  - Unexpected exit code (1) for backup: b'could not find a "pg_dump" to execute\npg_dump: error: no matching tables were found\n'
[2020-11-02 19:58:21,697] ERROR    (OSError) dsmr_backup.services.backup.run errored: b'could not find a "pg_dump" to execute\npg_dump: error: no matching tables were found\n'
[2020-11-02 19:58:43,048] CRITICAL  - Unexpected exit code (1) for backup: b'could not find a "pg_dump" to execute\npg_dump: error: no matching tables were found\n'
[2020-11-02 19:58:43,051] ERROR    (OSError) dsmr_backup.services.email.run errored: b'could not find a "pg_dump" to execute\npg_dump: error: no matching tables were found\n'
[2020-11-02 19:58:52,222] CRITICAL  - Unexpected exit code (1) for backup: b'could not find a "pg_dump" to execute\npg_dump: error: no matching tables were found\n'
[2020-11-02 19:58:52,225] ERROR    (OSError) dsmr_backup.services.backup.run errored: b'could not find a "pg_dump" to execute\npg_dump: error: no matching tables were found\n'

[rogierlommers]

How exactly does the backup mechanism work?

Are you shelling out to use pg_dump? That command is inside the postgres container, right? How do you get from the dsmr container to the postgres container? I don't really get that part.

Reason I'm asking: I have setup dsmr to use a dediated schema in the db. I think that's the cult pit here.

[xirixiz]

Ahh, ofcourse, that is most probably because of the different schema you are using which is currently not supported with DSMR Reader 😖.

[rogierlommers]

I indeed remembered a few seconds ago. You know what... I'll switch back to dedicated postgres containers for all my stuff.

[rogierlommers]

fuck these resources... :).. gives me reasons to buy a heavier server in my meterkast ;).

[xirixiz]

Or you just use the script I mentioned earlier, stop dsmr, take a backup, start dsmr

[rogierlommers]

Or you just use the script I mentioned earlier, stop dsmr, take a backup, start dsmr

Well, for some reason I can't disable the backup mechanism. If you "uncheck" it in the settings page; then I still get these errors. fyi: I don't need backups. I'll fix backups myself (of all schema's, in a generic way)

[xirixiz]

postgres, not much of a resource, but I agree it would be nicer to merge these rather small services/datbases.

[xirixiz]

Or you just use the script I mentioned earlier, stop dsmr, take a backup, start dsmr

Well, for some reason I can't disable the backup mechanism. If you "uncheck" it in the settings page; then I still get these errors.

Ah, that's something for Dennis to fix 👍