S3: No connection to backend

[vialcollet]

Hi. The backup works fine in local mode. The collection switch returns the correct information.

However when I try to use s3, files seem to be correctly loaded in s3 but then the collection switch returns BackendException: No connection to backend.

Here is my configuration file:

# Copyright (c) 2013 Toni de la Fuente.                                                                        
#                                                                                                              
# This program is free software: you can redistribute it and/or modify it under                                
# the terms of the Apache License as published by the Apache Foundation.                                       
#                                                                                                              
# This program is distributed in the hope that it will be useful, but WITHOUT                                  
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS                                
# FOR A PARTICULAR PURPOSE.                                                                                    
#                                                                                                              
# Most recent information about this tool is available in:                                                     
# http://blyx.com/alfresco-bart                                                                                
#                                                                                                              
# Latest code available at:                                                                                    
# http://blyx.com/alfresco-bart                                                                                
#                                                                                                              
#########################################################################################                      
# alfresco-bart: ALFRESCO BACKUP AND RECOVERY TOOL                                                             
# Version 0.2                                                                                                  
#########################################################################################                      
# ACTION REQUIRED:                                                                                             
# this file should contains passwords, please, set this file as read only                                      
# for root (chmod 400 alfresco-bart.properties)                                                                
# Copy this file in you ALFRESCO_INSTALLATION_PATH/scripts.                                                    
#########################################################################################                      

## Logging                                                                                                     
# Log configuration                                                                                            
LOG_DATE_FILES=`date +%F`                                                                                      
LOG_DATE_LOG=`date +%F-%X`                                                                                     
# Directory to left all Alfresco BART tasks logs with reports.                                                 
ALFBRT_LOG_DIR=/opt/alfresco/logs/bart                                                                         
ALFBRT_LOG_FILE=${ALFBRT_LOG_DIR}/alfresco-bart-${LOG_DATE_FILES}.log                                          

## Duplicity Setup ##                                                                                          
# If yes, please make sure you specify either PASSPHRASE, see INSTALL file.                                    
ENCRYPTION_ENABLED=false                                                                                       
PASSPHRASE=xxx                                                                                    
DUPLICITYBIN=`which duplicity`                                                                                 
GZIP=`which gzip`                                                                                              

## Backup policies                                                                                             
# Number of days of every full backup (if not backup found it does a full)                                     
FULLDAYS=30D                                                                                                   
# Backup policies to apply all backups collections (retention and cleanup)                                     
BACKUP_POLICIES_ENABLED=true                                                                                   
# Number of moths to remove all backups older than or backup retention period                                  
CLEAN_TIME=3M                                                                                                  
# After MAXFULL counter, all incrementals will be deleted and all full will be kept until CLEAN_TIME applies,  
MAXFULL=6                                                                                                      
# If you want to keep full backups of last 12 months but only with incremental in last 6 months                
# you must set CLEAN_TIME=12M and MAXFULL=6                                                                    
# Volume size in MB, default is 25MB per backup volume, consider reduce or increase it if you                  
# are doing tape backup (if a backup takes 60MB you will get 3 volumes, 25+25+10)                              
VOLUME_SIZE=25                                                                                                 

# Alfresco root installation path. It has to be a real directory not a symlink                                 
ALF_INSTALLATION_DIR=/opt/alfresco

# Alfresco alf_data path                                                                               
ALF_DIRROOT=${ALF_INSTALLATION_DIR}/alf_data                                                           

# Duplicity log vervosity 0 Error, 2 Warning, 4 Notice, 8 Info, 9 Debug (noisiest)                     
#0 recommended for production                                                                         
DUPLICITY_LOG_VERBOSITY=4                                                                              
BART_LOG_TAG=[BART-Loglevel$DUPLICITY_LOG_VERBOSITY]                                                   
GLOBAL_DUPLICITY_PARMS="-v${DUPLICITY_LOG_VERBOSITY} --volsize=${VOLUME_SIZE} --log-file=${ALFBRT_LOG_F

## Indexes Configuration                                                                               
BACKUP_INDEX_ENABLED=false                                                                             
# Alfresco index type, use solr or lucene                                                              
INDEXTYPE=lucene                                                                                       
# If Solr is selected, all Solr installation and config files will be included in your backup          
INDEXES_DIR=${ALF_DIRROOT}/lucene                                                                      
# If Lucene change to ${ALF_DIRROOT}/lucene                                                            
# SOLR or Lucene backup location                                                                       
INDEXES_BACKUP_DIR=${ALF_DIRROOT}/backup-lucene-indexes                                                
# If lucene add the apropiate location like ${ALF_DIRROOT}/backup-lucene-indexes                       
# If solr, please go to Alfresco Share Admin Console - Search Solr and set "Backups to                 
# keep to 1" in both Main and Archive                                                                  
# Other that do a backup of the "backed up" indexes, to easy restore of your Solr                      
# indexes is recommendable to backup all configuration files but the live files,                       
# To enable Solr installation and configuration backup set next property                               

## DB Configuration ##                                                                                 
BACKUP_DB_ENABLED=true                                                                                 
# use mysql, postgresql or oracle                                                                      
DBTYPE=mysql                                                                                           
LOCAL_BACKUP_DB_DIR=${ALF_DIRROOT}/alfresco-db-backup                                                  
# if you are using the bundle posgresql db, with next value it will include pg live files              
# for easy restore. For remote, different or external db leave it as is.                               
LOCAL_DB_DIR=${ALF_DIRROOT}/postgresql                                                                 
DATE_FILE=`date +%Y-%m-%d_%H%M%S`                                                                      

# Global DB parameters                                                                                 
DBNAME=alfresco                                                                                        
DBUSER=alfresco                                                                                        
DBPASS=xxxxxxxxxxxxxx                                                                                      
DBHOST=localhost                                                                                       

# MySQL - Alfresco DB Configuration                                                                    
MYSQL_BINDIR=/usr/bin                                                                                  
MYSQLDUMP_BIN=mysqldump                                                                                

# PostgreSQL - Alfresco DB Configuration                                                               
PGSQL_BINDIR=${ALF_INSTALLATION_DIR}/postgresql/bin                                                    
PGSQLDUMP_BIN=pg_dump                                                                                  
# Add PGUSER and PGPASSWORD if required for your installation and you are not doing                    
# running BART as root.                                                                                
PGUSER=                                                                                                
PGPASSWORD=${DBPASS}                                                                                   
# If you found problems to dump your Postgres DB try creating a file .pgpass with                      
# content "::::alfresco" without quotes, while alfresco is the password of your alfresco db            
PGPASSFILE=${ALF_DIRROOT}/.pgpass      
# Oracle - Alfresco DB Configuration                                            
ORACLE_BINDIR=                                                                  
ORASQLDUMP_BIN=exp                                                              

## ContentStore Configuration                                                   
BACKUP_CONTENTSTORE_ENABLED=true                                                
ALF_CONTENTSTORE=${ALF_DIRROOT}/contentstore                                    
# To exclude contentstore.deleted leave next value empty                        
ALF_CONTENSTORE_DELETED=                                                        
ALF_CACHED_CONTENTSTORE=                                                        
ALF_CONTENTSTORE2=                                                              
ALF_CONTENTSTORE3=                                                              
ALF_CONTENTSTORE4=                                                              
ALF_CONTENTSTORE5=                                                              

## Alfresco configuration and application files backup                          
# This backup includes deployments and configuration files excludes             
# the ContentStore, Indexes and DB as above                                     
BACKUP_FILES_ENABLED=true                                                       

## Backup type ##                                                               
# use s3, ftp, scp, local                                                       
BACKUPTYPE=s3                                                                   

## LOCAL backup destination folder ##                                           
# absolute path starting with /                                                 
LOCAL_BACKUP_FOLDER=/home/bastien/backup                                        

## Amazon S3 information ##                                                     
AWS_ACCESS_KEY_ID=xxxxxxxxxxxxxxxxxxxxxxxxxx                                       
AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx                 
# Upper case bucket name is not allow                                           
S3FILESYSLOCATION="s3+http://my_bucket"                                             
S3OPTIONS="--s3-use-new-style --s3-use-rrs"                                     
S3_EUROPEAN_BUCKET=false # not used yet                                         

## FTP Information ##                                                           
FTP_SERVER=                                                                     
FTP_USER=                                                                       
FTP_PASSWORD=                                                                   
FTP_FOLDER=                                                                     
FTP_PORT=21                                                                     
FTPS_ENABLE=false                                                               

## SCP Information ##                                                           
# SSH shared keys required                                                      
SCP_SERVER=                                                                     
SCP_USER=                                                                       
SCP_FOLDER=                                                                     

## Backup temp folder definition ##                                             
TEMPFOLDERNAME="alfresco-backup-`date +%F-%s`"                                  

## Recovery configuration
ENABLE_SINGLE_FILE_RECOVERY=false

# Temporary DB for recovery and get the phisical path of a single content.
REC_DBTYPE=mysql

# Recovery database must be already created but EMPTY! Alfresco-BART will recover your
# backup database and find the files for you, after this task the recovery database
# tables will be deleted.

# MySQL - Recovery DB Configuration ##
REC_MYDBNAME=alfresco_rec
REC_MYUSER=alfresco
REC_MYPASS=xxxxxxxxxxxxxxxxxxxx
REC_MYHOST=localhost
REC_MYSQL_BIN=/usr/bin/mysql
REC_MYSQLDUMP_BIN=/usr/bin/mysqldump

# PgSQL - Recovery DB Configuration ##
REC_PGDBNAME=alfresco_rec
REC_PGUSER=alfresco
REC_PGPASS=alfresco
REC_PGHOST=localhost
REC_PGSQL_BIN=/opt/alfresco/postgresql/bin/psql
REC_PGSQLDUMP_BIN=/opt/alfresco/postgresql/bin/pg_dump

# Oracle - Recovery DB Configuration ##
REC_ORADBNAME=alfresco_rec
REC_ORAUSER=alfresco
REC_ORAPASS=alfresco
REC_ORAHOST=localhost
REC_ORASQL_BIN=
REC_ORASQLDUMP_BIN=exp                                                         

[toniblyx]

Hi, this looks like a Duplicity configuration issue, see comment #6 here https://bugs.launchpad.net/duplicity/+bug/1278529

Does it work?

[vialcollet]

Hi thanks for the answer. I don't really understand how to implement this comment in my case... I tried gsettings set org.gnome.DejaDup.S3 bucket cav78, but it returns: No such schema 'org.gnome.DejaDup.S3'...

Could you help me to understand? This is an new area to me :(

[toniblyx]

What version of Duplicity do you have installed? I have seen this bug is fixed in v0.7.01 (2015/01/11). YOu also can try with s3+http://my_bucket/somedir instead. If not, try updating Boto, is the python library that allows Duplicity to talk to S3.

[vialcollet]

Yes I saw that there is a more recent version. I am using the available package for ubuntu: Installed: 0.6.23-1ubuntu4.1 I don't know how to upgrade... and if this is safe. What do you think?

[vialcollet]

Oh and I forgot to mention that I tried with setting up a directory and this didn't help.

[toniblyx]

I haven't upgrade it yet but it looks like there are stable packages for ubuntu here https://launchpad.net/~duplicity-team/+archive/ubuntu/ppa

[vialcollet]

Thank you so much for your help... ! I have succesfully updated duplicity. Now the message is similar but not 100%:

======================= BACKUP COLLECTION FOR INDEXES lucene ======================
BackendException:
./alfresco-bart.sh: line 741:  : command not found
=========================== BACKUP COLLECTION FOR DB mysql ==========================
BackendException:
========================== BACKUP COLLECTION FOR CONTENTSTORE =========================
BackendException:
============================== BACKUP COLLECTION FOR FILES ============================
BackendException:

[toniblyx]

can you post the command you type? you have disabled lucene backup in config...

[paulmaunders]

I have also been getting the BackendException: No connection to backend error with duplicity.

I noticed that s3cmd had stopped working for me too with the following error...

# s3cmd ls ERROR: S3 error: 403 (RequestTimeTooSkewed): The difference between the request time and the current time is too large.

It turns out the system time on my vagrant box was 3 days in the past. Forcing a time update seemed to fix both s3cmd and duplicity!

[douglascrp]

@vialcollet @paulmaunders Please, take a look at the pull request number #17 S3 bucket config sample improvement. I hope it is useful

[douglascrp]

@toniblyx I believe this is not a problem with the script it self, but with the format used to setup the connection to S3. Please, review my pull request, and if you accept it, I believe this issue can be closed.

[douglascrp]

Issued solved with the merge of pull request #17