Feature Request - "proxmox backup server" as storage option

[n1ete]

low hanging feature fruit for proxmox backup server

in /etc/pve/storage.cfg a definition looks like this:

pbs: pbs-storage
        datastore $DATASTORENAME
        server $PBS_HOST
        content backup
        fingerprint $PBS_FINGERPRINT
        prune-backups keep-all=1
        username $USERNAME

are zfspools supported btw?

[lae]

zfspools in #114

[lae]

documentation for this feature https://pve.proxmox.com/wiki/Storage:_Proxmox_Backup_Server

[lae]

Is there a specific JSON schema for encryption-key that we have to follow? I think it should be the contents of /etc/pve/priv/storage/<STORAGE-ID>.enc but I'm not sure, and I can't test because I have no PBS setup.

[lae]

@n1ete Do you think you could test out that PR when you get a chance?

[Andre15711]

What ist the status about this feature? is it tested? When will it the merged into the release?

[lae]

@Andre15711 Is this something you can help test? If I can get a confirmation that it works properly for someone, I guess I can go ahead and merge it.

[Andre15711]

@Andre15711 Is this something you can help test? If I can get a confirmation that it works properly for someone, I guess I can go ahead and merge it.

Yes, why not. Let me try it.....

Any advice? I never tested a role before....

[lae]

@Andre15711 Normally, you can install a specific branch (like this PR) using a URL like the following:

ansible-galaxy install https://github.com/lae/ansible-role-proxmox/archive/feature/pbs-storage.tar.gz,pr-169,lae.proxmox --force

This grabs the archive from the branch feature/pbs-storage and gives it a version number of "pr-169" (usually it's a version number like 1.5.0 but it could really be anything) and installs it as lae.proxmox into your roles directory.

Only problem however is that this is a pretty old PR, and the role has received updates since. It's possible the above may be too outdated to work as-is. In that case, you'd need to check out the main branch somewhere temporarily, merge the feature branch (feature/pbs-storage), and either install that with ansible-galaxy or symlink it in your ~/.ansible/roles/ directory. But that may be considerably more work depending on your familiarity with git...

[Andre15711]

@lae I just tried out the merge the feature branch local na dliked the new role via symlink into my '~/.ansible/roles/' and run my current playbook, but without luck. The "Configure Proxmox Storage" task failed with tracebacks.

The full traceback is:
Traceback (most recent call last):
  File "/home/ansible/.ansible/tmp/ansible-tmp-1685094439.91814-357048-38741695215623/AnsiballZ_proxmox_storage.py", line 107, in <module>
    _ansiballz_main()
  File "/home/ansible/.ansible/tmp/ansible-tmp-1685094439.91814-357048-38741695215623/AnsiballZ_proxmox_storage.py", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/ansible/.ansible/tmp/ansible-tmp-1685094439.91814-357048-38741695215623/AnsiballZ_proxmox_storage.py", line 47, in invoke_module
    runpy.run_module(mod_name='ansible.modules.proxmox_storage', init_globals=dict(_module_fqn='ansible.modules.proxmox_storage', _modlib_path=modlib_path),
  File "/usr/lib/python3.9/runpy.py", line 210, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_proxmox_storage_payload_3vsafbr6/ansible_proxmox_storage_payload.zip/ansible/modules/proxmox_storage.py", line 450, in <module>
  File "/tmp/ansible_proxmox_storage_payload_3vsafbr6/ansible_proxmox_storage_payload.zip/ansible/modules/proxmox_storage.py", line 415, in main
  File "/tmp/ansible_proxmox_storage_payload_3vsafbr6/ansible_proxmox_storage_payload.zip/ansible/modules/proxmox_storage.py", line 219, in __init__
TypeError: expected string or bytes-like object
failed: [vm-host2] (item={'name': 'local-zfs', 'type': 'zfspool', 'content': ['images'], 'pool': 'local-zfs/local-zfs', 'sparse': True}) => changed=false 
  ansible_loop_var: item
  item:
    content:
    - images
    name: local-zfs
    pool: local-zfs/local-zfs
    sparse: true
    type: zfspool
  module_stderr: |-
    Shared connection to xxx.xxx.xxx.xxx closed.
  module_stdout: |-
    Traceback (most recent call last):
      File "/home/ansible/.ansible/tmp/ansible-tmp-1685094439.91814-357048-38741695215623/AnsiballZ_proxmox_storage.py", line 107, in <module>
        _ansiballz_main()
      File "/home/ansible/.ansible/tmp/ansible-tmp-1685094439.91814-357048-38741695215623/AnsiballZ_proxmox_storage.py", line 99, in _ansiballz_main
        invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
      File "/home/ansible/.ansible/tmp/ansible-tmp-1685094439.91814-357048-38741695215623/AnsiballZ_proxmox_storage.py", line 47, in invoke_module
        runpy.run_module(mod_name='ansible.modules.proxmox_storage', init_globals=dict(_module_fqn='ansible.modules.proxmox_storage', _modlib_path=modlib_path),
      File "/usr/lib/python3.9/runpy.py", line 210, in run_module
        return _run_module_code(code, init_globals, run_name, mod_spec)
      File "/usr/lib/python3.9/runpy.py", line 97, in _run_module_code
        _run_code(code, mod_globals, init_globals,
      File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/tmp/ansible_proxmox_storage_payload_3vsafbr6/ansible_proxmox_storage_payload.zip/ansible/modules/proxmox_storage.py", line 450, in <module>
      File "/tmp/ansible_proxmox_storage_payload_3vsafbr6/ansible_proxmox_storage_payload.zip/ansible/modules/proxmox_storage.py", line 415, in main
      File "/tmp/ansible_proxmox_storage_payload_3vsafbr6/ansible_proxmox_storage_payload.zip/ansible/modules/proxmox_storage.py", line 219, in __init__
    TypeError: expected string or bytes-like object
  msg: |-
    MODULE FAILURE
    See stdout/stderr for the exact error
  rc: 1

[Andre15711]

If I see this correct, the entire syntax of the storage declaration is totally diffrent. Is this correct?

[lae]

If I see this correct, the entire syntax of the storage declaration is totally diffrent. Is this correct?

No.

      File "/tmp/ansible_proxmox_storage_payload_3vsafbr6/ansible_proxmox_storage_payload.zip/ansible/modules/proxmox_storage.py", line 219, in __init__
   TypeError: expected string or bytes-like object

This appears to be the following function if I look at the branch as-is. Is line 219 the same in your merged library/proxmox_storage.py file?

https://github.com/lae/ansible-role-proxmox/blob/9014790beb11a0ef4be76a174287c04a6bf287ef/library/proxmox_storage.py#L217-L221

If so, can you try changing it to:

        if self.fingerprint is not None and not fingerprint_re.match(self.fingerprint):

and try again?

[Andre15711]

Sure!

next failure.... ;-)

TASK [lae.proxmox : Configure Proxmox Storage] **************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: AttributeError: 'ProxmoxStorage' object has no attribute 'master_pubkey'
failed: [vm-host-test] (item={'name': 'PBS', 'type': 'pbs', 'content': ['backup'], 'server': 'xxx.xxx.xxx.xxx', 'username': 'xyz@pbs', 'password': 'cl5.a4kt', 'datastore': 'Backup-Pool-3TB', 'fingerprint': '6b:70:ca:3......', 'encryption_key': 'autogen'}) => changed=false
  ansible_loop_var: item
  item:
    content:
    - backup
    datastore: Backup
    encryption_key: autogen
    fingerprint: 6b:70:ca:3.......
    name: PBS
    password: abc
    server: xxx.xxx.xxx.xxx
    type: pbs
    username: xyz@pbs
  module_stderr: |-
    Shared connection to xxx.xxx.xxx.xx closed.
  module_stdout: |-
    Traceback (most recent call last):
      File "/home/ansible/.ansible/tmp/ansible-tmp-1685174352.5319731-29475-273802405905516/AnsiballZ_proxmox_storage.py", line 107, in <module>
        _ansiballz_main()
      File "/home/ansible/.ansible/tmp/ansible-tmp-1685174352.5319731-29475-273802405905516/AnsiballZ_proxmox_storage.py", line 99, in _ansiballz_main
        invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
      File "/home/ansible/.ansible/tmp/ansible-tmp-1685174352.5319731-29475-273802405905516/AnsiballZ_proxmox_storage.py", line 47, in invoke_module
        runpy.run_module(mod_name='ansible.modules.proxmox_storage', init_globals=dict(_module_fqn='ansible.modules.proxmox_storage', _modlib_path=modlib_path),
      File "/usr/lib/python3.9/runpy.py", line 210, in run_module
        return _run_module_code(code, init_globals, run_name, mod_spec)
      File "/usr/lib/python3.9/runpy.py", line 97, in _run_module_code
        _run_code(code, mod_globals, init_globals,
      File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/tmp/ansible_proxmox_storage_payload_vb1t2jo0/ansible_proxmox_storage_payload.zip/ansible/modules/proxmox_storage.py", line 451, in <module>
      File "/tmp/ansible_proxmox_storage_payload_vb1t2jo0/ansible_proxmox_storage_payload.zip/ansible/modules/proxmox_storage.py", line 436, in main
      File "/tmp/ansible_proxmox_storage_payload_vb1t2jo0/ansible_proxmox_storage_payload.zip/ansible/modules/proxmox_storage.py", line 311, in create_storage
      File "/tmp/ansible_proxmox_storage_payload_vb1t2jo0/ansible_proxmox_storage_payload.zip/ansible/modules/proxmox_storage.py", line 274, in prepare_storage_args
    AttributeError: 'ProxmoxStorage' object has no attribute 'master_pubkey'
  msg: |-
    MODULE FAILURE
    See stdout/stderr for the exact error
  rc: 1

BTW: I needed to commend out the other Storage definitions in my environment variables. With the NFS and ZFS storage it was failing again with the other error. So for now I just have the PBS storage declaration in my variables.

regards André

[lae]

Okay, now it's failing for the PBS definition itself so that's progress.

It looks like some people actually brought up the master_pubkey issue in that PR, and I just haven't managed to investigate thoroughly. My first guess (which may be wrong since it's been years) is that it's missing an assignment in the __init__ method. Do you think you could add the self.master_pubkey assignment like below at line 201 (with the other assignments) and test?

self.encryption_key = module.params['encryption_key']
self.master_pubkey = module.params['master_pubkey']

Please also test with all of your storage definitions uncommented. We wouldn't've realised there was an issue affecting all storage definitions if I didn't see the first error you provided.

---

According to PVE API docs, master_pubkey is:

Base64-encoded, PEM-formatted public RSA key. Used to encrypt a copy of the encryption-key which will be added to each encrypted backup.

which I'm not exactly sure what it means. I might need to revisit (test from the UI and stuff).

[Andre15711]

The encryption is optional.... I made a screenshot of the encryption page in the PBS setup dialog.

Anyhow. If I added the recommanded line in the proxmox_storage.py library. It failed with the error 'msg: Unable to import proxmox_storage due to invalid syntax'.

my current storage definition: `

• name: PBS type: pbs content: [ "backup" ] server: xxx.xxx.xxx.xxx username: backup@pbs password: secret datastore: Backup fingerprint: 6b:70:ca:3.......... encryption_key: autogen`

[lae]

Yes, the encryption is optional. That doesn't mean I should leave it out of the module for users who want to use it.

Invalid syntax sounds like you may have incorrectly edited the Python file. It needs to be properly indented, using spaces.

Anyway, thanks for your help. I'm guessing I need to look at this more in-depth myself when I'm free, but I'm not sure when I'll be able to do that currently.

[Andre15711]

Here is the code....

[Andre15711]

The error message looks like an eeror from the API....

[lae]

error from the API

No, it's an error with Python importing the proxmox_storage module. So something had to have broken during editing.

I can't provide much help right now, and I really shouldn't be proxying development onto you anyway. Per my edit to my last comment, I'll try to look at the feature in more depth when I'm free.

[Andre15711]

I found the typo in the library and fixed it. It is working now. :-)

[Andre15711]

As this is working, can you merge this into the main branch?