Error with S3 backend : No module named 'boto'

badwulfy commented 3 years ago

Hello,

Thanks for your work with volumerize. :tada:

I have an issue with S3 during backup (boto module missing). If i attach to container and install boto manually, next run will work.

> docker exec -it volumerize backup             
BackendException: Could not initialize backend: No module named 'boto'
running /postexecute/backup/0-removeoldbackup.sh
Checking if old backups should be removed

running /postexecute/backup/1-replicate.sh

> docker exec -it volumerize pip3 install boto  
Collecting boto
  Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB)
     |████████████████████████████████| 1.4 MB 8.1 MB/s 
Installing collected packages: boto
Successfully installed boto-2.49.0

> docker exec -it volumerize backup           
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Mar 10 23:30:21 2021
--------------[ Backup Statistics ]--------------
StartTime 1615415569.40 (Wed Mar 10 23:32:49 2021)
EndTime 1615415569.56 (Wed Mar 10 23:32:49 2021)
ElapsedTime 0.17 (0.17 seconds)
SourceFiles 298
SourceFileSize 13114843 (12.5 MB)
NewFiles 1
NewFileSize 4096 (4.00 KB)
DeletedFiles 2
ChangedFiles 1
ChangedFileSize 573440 (560 KB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 4
RawDeltaSize 2153 (2.10 KB)
TotalDestinationSizeChange 959 (959 bytes)
Errors 0
-------------------------------------------------

running /postexecute/backup/0-removeoldbackup.sh
Checking if old backups should be removed

running /postexecute/backup/1-replicate.sh

My volumerize is setup with a docker-compose :

version: "3"
services:
  # other services ...

  volumerize:
    container_name: volumerize
    volumes:
    - volumerize-cache:/volumerize-cache
    - ./tobackup:/source
    environment:
    - VOLUMERIZE_SOURCE=/source
    - VOLUMERIZE_TARGET=s3://s3.fr-par.scw.cloud/xxxx
    - AWS_ACCESS_KEY_ID=xxxxxx
    - AWS_SECRET_ACCESS_KEY=xxxxx
    - PASSPHRASE=xxxxxxxxx
    - 'VOLUMERIZE_JOBBER_TIME=0 0 3 * * *'
    - VOLUMERIZE_FULL_IF_OLDER_THAN=14D
    - TZ=Europe/London
    image: fekide/volumerize:latest

Is it a misconfiguration on my side or a problem in volumerize?

Thanks in advance for your help :smile:

sargreal commented 3 years ago

Thanks for your bug report @kiralex !

I have never worked with S3, so i didn't realize that there were multiple versions used.

From the duplicity Readme, I can see that different versions of the boto package are required for different versions of S3. Do you know if the latest version also works with the older ones?

badwulfy commented 3 years ago

Thanks for your answer.

I just researched clues in the man page of duplicity and I realized there is a specific URL which use boto3 (boto3+s3://), and the legacy one whihch use boto2 (s3://). As the doc says :

S3 storage (Amazon)

s3://host[:port]/bucket_name[/prefix]
s3+http://bucket_name[/prefix]
defaults to the legacy boto backend based on boto v2 (last update 2018/07)
alternatively try the newer boto3+s3://bucket_name[/prefix]
For details see A NOTE ON AMAZON S3 and see also A NOTE ON EUROPEAN S3 BUCKETS below.

I think it would be awesome if we can have the choice. So boto in version 2 and boto3 would be installed and we could use both s3:// and boto3+s3:// URL format.

I don't know if boto3+s3:// is currently supported in volumerize

If not, boto in version 2 should be installed (and maybe consider to support this syntax in a near/far futur, but I don't want to put any pressure on you, as s3:// suits me perfectly)
if yes, both version 2 and 3 should be installed.

sargreal commented 3 years ago

Thanks for your investigation!

Yes boto3 is already installed, so i will just add the boto package.

Fekide / volumerize

Error with S3 backend : No module named 'boto' #13