No backups shown in the admin interface and failing manually since new duplicity version has been installed

[haslinger]

I have seen a new duplity version has been installed lately:

duplicity/jammy,now 3.0.1-ppa202408051423~ubuntu22.04.1 amd64 [installed]

And from that moment on I don't see any backups on the backup status page any more. Running them manually results in in an error message:

stefan@box:~/mailinabox/management$ sudo python3 backup.py
[sudo] password for stefan: 
Traceback (most recent call last):
  File "/home/stefan/mailinabox/management/backup.py", line 12, in <module>
    import rtyaml
ModuleNotFoundError: No module named 'rtyaml'

[kiekerjan]

With duplicity 3.0.1, I don't see the list of backups anymore either. However, the backup command should be run like:

stefan@box:~/mailinabox$ sudo management/backup.py

Which works as far as I can see.

[haslinger]

@kiekerjan Thank you for your follow up and the correction to my attempt to run the backup manually. I can confirm, that

stefan@box:~/mailinabox$ sudo management/backup.py

works fine and I am a happy :rabbit: to have a manually created backup.

[amedee]

I also have the problem, I posted on the forum: https://discourse.mailinabox.email/t/backup-to-s3-files-are-in-s3-bucket-but-miab-says-no-backups-have-been-made-yet/12232/2

I replaced the apt version of duplicity 3.0.1 with the snap version of duplicity 3.0.0:

apt remove duplicity
wget https://downloads.sourceforge.net/project/duplicity/snaps/duplicity_3.0.0_amd64.snap
snap install duplicity_3.0.0_amd64.snap --dangerous --classic
ln -s /snap/bin/duplicity /usr/bin/duplicity
/root/mailinabox/management/backup.py

Then I refreshed the Backup Status page and it looks ok now.

As the snap command clearly says itself: this is DANGEROUS and I will restore the original situation as soon as the bug is fixed. Only use this workaround if you know what you are doing!

I used the snap package because I couldn't find a 3.0.0 apt package, otherwise I would have used apt pinning. That might have been a tiny bit cleaner.

[amedee]

I have seen a new duplity version has been installed lately:

duplicity/jammy,now 3.0.1-ppa202408051423~ubuntu22.04.1 amd64 [installed]

And from that moment on I don't see any backups on the backup status page any more. Running them manually results in in an error message:

stefan@box:~/mailinabox/management$ sudo python3 backup.py
[sudo] password for stefan: 
Traceback (most recent call last):
  File "/home/stefan/mailinabox/management/backup.py", line 12, in <module>
    import rtyaml
ModuleNotFoundError: No module named 'rtyaml'

If you want to run the backup in the "bad way" with the command python3 backup.py then you first have to run

pip3 install rtyaml
pip3 install exclusiveprocess

I did get this warning:

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

so there may be side effects, use at your own risk.

But it may also indicate that the base installation of MIAB requires two extra Python packages?

[kiekerjan]

If you want to run the backup in the "bad way" with the command python3 backup.py then you first have to run

I recommend against this. MailinaBox has it's own Python virtual environment, see the first line of backup.py There might be unexpected side effects.

[amedee]

Thank you, I just noticed the shebang.

@haslinger don't do that! You're barking up the wrong tree!

[kiekerjan]

Looks like (again) it's recommended not to upgrade to the latest duplicity. As @amedee shows, there is a regression in version 3.0.1 I think it's this bug Once fixed, we might also consider pinning the Duplicity package to prevent thiis.

[amedee]

Looks like (again) it's recommended not to upgrade to the latest duplicity.

What is the recommended strategy for those of us who have already upgraded, as the PPA of 3.0.0 is no longer available? Is it ok to replace it with the Snap version, as I did? That version will not auto-upgrade.

[amedee]

I use Ansible for all my MiaB customizations, which then run with GitHub Actions. I made this customization for this bug, it should be self explanatory.

- name: Configure duplicity
  become: true
  block:
    - name: Uninstall apt version of duplicity
      ansible.builtin.apt:
        name: duplicity
        state: absent

    - name: Download duplicity snap package
      ansible.builtin.get_url:
        url: https://downloads.sourceforge.net/project/duplicity/snaps/duplicity_3.0.0_amd64.snap
        dest: /root/downloads/duplicity_3.0.0_amd64.snap
        mode: u=rw,go=r
        owner: root

    - name: Install snap version of duplicity
      community.general.snap:
        name: /root/downloads/duplicity_3.0.0_amd64.snap
        classic: true
        dangerous: true

    - name: Create backup config from template
      ansible.builtin.template:
        src: home/user-data/backup/custom.yaml.j2
        dest: /home/user-data/backup/custom.yaml
        owner: root
        mode: u=rw,go=r
      notify: Perform backup

and the handler:

- name: Perform backup
  become: true
  ansible.builtin.command:
    cmd: /root/mailinabox/management/backup.py
  changed_when: true

and the template file:

min_age_in_days: 7
target: {{ mailinabox_backup_target }}
target_user: {{ common_aws_access_key }}
target_pass: {{ common_aws_secret_key }}

{{ mailinabox_backup_target }} is the s3://s3.<region>.amazonaws.com/<bucket>/<directory> URL to the S3 bucket.

[kiekerjan]

I see duplicity 3.0.2 is released, which seems to resolve the issue. For now I'll hold the package like apt-mark hold duplicity I'll look into changes needed for pinning the Duplicity package. Unless someone beats me to it 😉

[amedee]

This is my customization in Ansible now:

- name: Configure duplicity
  become: true
  block:
    - name: Uninstall snap version of duplicity
      community.general.snap:
        name: duplicity
        state: absent

    - name: Install apt version of duplicity
      ansible.builtin.apt:
        name: duplicity=3.0.2-ppa202408091359~ubuntu22.04.1

    - name: Prevent duplicity from being upgraded
      ansible.builtin.dpkg_selections:
        name: duplicity
        selection: hold

[haslinger]

The automatic update of duplicity to version 3.0.2 has resolved the issue.

[amedee]

Great! I changed my customization thusly:

In the vars:

apt_preferences__list:
  - package: 'duplicity'
    reason: |-
      Duplicity version 3.0.1 has a bug and I don't want it installed
    version: '3.0.1'
    priority: '99'

And in meta/main.yaml:

dependencies:
  - role: debops.debops.apt_preferences

This way, version 3.0.1 will always be skipped. And if in the future we get prior warning of another package that breaks stuff, then I can just add that package with it's version to apt_preferences__list.