TypeError: 'NoneType' object is not iterable

lloydbayley commented 1 month ago

G'day,

Got the following error when I tried run a backup with Grapple. Unfortunately, it's quite a generic error message and I've no idea where to start. It seems that the HOOK-ENV ought to have something it in and not just None Could someone point me in the right direction please? I'm using the following line in /etc/vzdump.conf;

script: /root/.local/bin/proxmox-grapple

All other entries were commented out...I didn't touch them because it wasn't clear if I should.

Error is:

Header
Proxmox
Virtual Environment 8.2.2
Datacenter
Logs
INFO: HOOK: /root/.local/bin/proxmox-grapple job-init
INFO: HOOK-ENV: vmtype=None; dumpdir=None; storeid=local; hostname=None; tarfile=None; logfile=None
INFO: Traceback (most recent call last):
INFO:   File "/root/.local/bin/proxmox-grapple", line 8, in <module>
INFO:     sys.exit(main())
INFO:              ^^^^^^
INFO:   File "/root/.local/pipx/venvs/proxmox-grapple/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
INFO:     return self.main(*args, **kwargs)
INFO:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO:   File "/root/.local/pipx/venvs/proxmox-grapple/lib/python3.11/site-packages/click/core.py", line 1078, in main
INFO:     rv = self.invoke(ctx)
INFO:          ^^^^^^^^^^^^^^^^
INFO:   File "/root/.local/pipx/venvs/proxmox-grapple/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
INFO:     return ctx.invoke(self.callback, **ctx.params)
INFO:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO:   File "/root/.local/pipx/venvs/proxmox-grapple/lib/python3.11/site-packages/click/core.py", line 783, in invoke
INFO:     return __callback(*args, **kwargs)
INFO:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO:   File "/root/.local/pipx/venvs/proxmox-grapple/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
INFO:     return f(get_current_context(), *args, **kwargs)
INFO:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO:   File "/root/.local/pipx/venvs/proxmox-grapple/lib/python3.11/site-packages/proxmox_grapple/vzdump_hook_script.py", line 112, in main
INFO:     for exec_line in settings[phase][mode]:
INFO: TypeError: 'NoneType' object is not iterable
INFO: notified via target `mail-to-root`
TASK ERROR: command '/root/.local/bin/proxmox-grapple job-init' failed: exit code 1

I'd be grateful for any help.

Cheers!

lingfish commented 1 month ago

Heya, thanks for the report.

When you say:

All other entries were commented out...I didn't touch them because it wasn't clear if I should.

Do you mean all entries in /etc/vzdump.conf, or the grapple config?

Please attach your proxmox_grapple.yml from when you got this error too.

lloydbayley commented 1 month ago

Hello! Thanks for the reply! Yes, I meant all other entries in the /etc/vzdump.conf

Here's the proxmox_grapple.yml

production:
  backup-start:
    script:
      - "curl -fsS -m 10 --retry 5 -o /dev/null 'http://10.61.2.113/win&PL=06'"
#  backup-abort:
#    script:
#      - "curl -fsS -m 10 --retry 5 -o /dev/null 'http://10.61.2.113/win&PL=03'"
  backup-end:
    script:
      - "curl -fsS -m 10 --retry 5 -o /dev/null 'http://10.61.2.113/win&PL=03'"

I even tried one of your examples just to see if it would get past it in case there was a hard-to-find error in my yaml code.

production:
  job-end:
    script:
      - echo 'hi'
      - sleep 1
      - echo 'there'
      - echo 'This is a test.'

  backup-end:
    extract:
      enabled: false
      source_directory: /tmp
      destination_directory: /tmp
  #    exclude_storeids:

  job-abort:
    shell:
      - echo 'your strange command' | tee some_logfile.txt

I've looked through the /root/.local/pipx/venvs/proxmox-grapple/lib/python3.11/site-packages/proxmox_grapple/vzdump_hook_script.py file and it appears it should be receiving some environment variables but it seems only to be showing the storage: local.

lingfish commented 1 month ago

Hello! Thanks for the reply! Yes, I meant all other entries in the /etc/vzdump.conf

Right, understood. You should only need that one script: line.

Here's the proxmox_grapple.yml

Looks clean to me.

I even tried one of your examples just to see if it would get past it in case there was a hard-to-find error in my yaml code.

Good thinking. That helps!

I've looked through the /root/.local/pipx/venvs/proxmox-grapple/lib/python3.11/site-packages/proxmox_grapple/vzdump_hook_script.py file and it appears it should be receiving some environment variables but it seems only to be showing the storage: local.

The env vars vary based on what phase the backup is in (and hence what the script: receives). So in your output, it's at the initial stage, job-init (see the first INFO: HOOK: log line), so it's quite normal to only have the storeid.

As you've specified no config for the job-init phase, the final generated config should basically render to None, and should be skipped... very strange.

Can you paste the output of running grapple only with --dump-config ? Only need all the YAML stuff (you'll see a JSON blob at the end, ignore that/don't paste). Feel free to redact anything if needed.

The current: key is basically the final config it'll run with, and inside that, it should look like:

  JOB-INIT:
    script: None

Can you also paste the pipx list output, and a full pip list (via runpip)?

I'm wondering if this isn't a Python version issue, or a breaking change from something that grapple depends on.

Thanks heaps for helping out!

lloydbayley commented 1 month ago

It's a pleasure to help. I'm just hoping it's not something stupid I've overlooked. I'm pretty bloody good at Linux, Bash and python et al but even we tech people miss the obvious sometimes because we're looking too deeply! :)

Here's the grapple dump;

root@development:~/.local/bin# ./proxmox-grapple --dump-config
header:
  env_filter: None
  key_filter: None
  new_first: 'True'
  history_limit: None
  include_internal: 'False'
current:
  JOB-INIT:
    script: None
  JOB-START:
    script: None
  JOB-END:
    script: None
  JOB-ABORT:
    script: None
  BACKUP-START:
    script: None
  BACKUP-END:
    extract:
      exclude_storeids: None
      enabled: false
    script: None
  BACKUP-ABORT:
    script: None
  LOG-END:
    script: None
  PRE-STOP:
    script: None
  PRE-RESTART:
    script: None
  POST-RESTART:
    script: None
history:
- loader: set_method
  identifier: undefined
  env: global
  merged: false
  value:
    JOB-INIT:
      script: None
    JOB-START:
      script: None
    JOB-END:
      script: None
    JOB-ABORT:
      script: None
    BACKUP-START:
      script: None
    BACKUP-END:
      extract:
        exclude_storeids: None
        enabled: false
      script: None
    BACKUP-ABORT:
      script: None
    LOG-END:
      script: None
    PRE-STOP:
      script: None
    PRE-RESTART:
      script: None
    POST-RESTART:
      script: None
- loader: setdefault
  identifier: unique
  env: production
  merged: false
  value:
    JOB-INIT:
      script: None
    JOB-START:
      script: None
    JOB-END:
      script: None
    JOB-ABORT:
      script: None
    BACKUP-START:
      script: None
    BACKUP-END:
      extract:
        exclude_storeids: None
        enabled: false
      script: None
    BACKUP-ABORT:
      script: None
    LOG-END:
      script: None
    PRE-STOP:
      script: None
    PRE-RESTART:
      script: None
    POST-RESTART:
      script: None
- loader: set_method
  identifier: settings_module_method
  env: global
  merged: false
  value:
    SETTINGS_MODULE:
    - proxmox_grapple.yml
    - /etc/proxmox_grapple.yml
- loader: set_method
  identifier: init_kwargs
  env: global
  merged: false
  value:
    SETTINGS_FILE_FOR_DYNACONF:
    - proxmox_grapple.yml
    - /etc/proxmox_grapple.yml
    APPLY_DEFAULT_ON_NONE_FOR_DYNACONF: true
    ENV_FOR_DYNACONF: production
    ENVIRONMENTS_FOR_DYNACONF: true
    CORE_LOADERS_FOR_DYNACONF:
    - YAML

pipx list;

root@development:~# pipx list
venvs are in /root/.local/pipx/venvs
apps are exposed on your $PATH at /root/.local/bin
   package proxmox-grapple 1.5.5, installed using Python 3.11.2
    - proxmox-grapple

Here's something.....unexpected....Do I have the commandline wrong or is my penv screwed. Did I manage to bugger it up and install things first system-wide (as told not to) and then try to recover?

root@development:~# pipx runpip pip list
venv for 'pip' was not found. Was 'pip' installed with pipx?

lingfish commented 1 month ago

It's a pleasure to help. I'm just hoping it's not something stupid I've overlooked. I'm pretty bloody good at Linux, Bash and python et al but even we tech people miss the obvious sometimes because we're looking too deeply! :)

Agreed. And you're from Sydney :wink:

Here's the grapple dump;

root@development:~/.local/bin# ./proxmox-grapple --dump-config
header:
  env_filter: None
  key_filter: None
  new_first: 'True'
  history_limit: None
  include_internal: 'False'
current:
  JOB-INIT:
    script: None
  JOB-START:
    script: None
  JOB-END:
    script: None
  JOB-ABORT:
    script: None
  BACKUP-START:
    script: None
  BACKUP-END:
    extract:
      exclude_storeids: None
      enabled: false
    script: None
  BACKUP-ABORT:
    script: None
  LOG-END:
    script: None
  PRE-STOP:
    script: None
  PRE-RESTART:
    script: None
  POST-RESTART:
    script: None

Yeah, this is odd. There should be some config there, which means your config file wasn't loaded/found. Is the config file you have located as per here? Dynaconf is a little odd in that it assumes missing config files are perfectly ok, and I have an issue logged for it that isn't resolved yet.

   package proxmox-grapple 1.5.5, installed using Python 3.11.2

All good here.

Here's something.....unexpected....Do I have the commandline wrong or is my penv screwed. Did I manage to bugger it up and install things first system-wide (as told not to) and then try to recover?
root@development:~# pipx runpip pip list
venv for 'pip' was not found. Was 'pip' installed with pipx?

Yeah, your command is a bit wrong...

runpip implies pip, so not needed
You forgot the name of your pipx package name

So, something like pipx runpip proxmox-grapple list

lloydbayley commented 1 month ago

I am indeed from Syd! :) Ok...I need to stop doing things in the wee-small hours and pay attention!!

Here's the pip listing....Tsk! :)

root@development:~# pipx runpip proxmox-grapple list
Package            Version
------------------ -------
click              8.1.7
daiquiri           3.2.5.1
dynaconf           3.2.6
humanize           4.10.0
pip                24.2
proxmox-grapple    1.5.5
psutil             6.0.0
python-json-logger 2.0.7
PyYAML             6.0.1
setuptools         72.1.0
wheel              0.43.0

Is that of any earth-shattering use?

lingfish commented 1 month ago

Here's the pip listing....Tsk! :)

Looks all good too... sorry, can you answer the other stuff I tucked into the previous reply around config files?

lloydbayley commented 1 month ago

Sorry, sorry, sorry.... Missed that. Yes it's at /etc/proxmox_grapple.yml

Interestingly, in my initial testing, I renamed the file to test if it was finding it and the continued merrily along with the same error message - so when you said that you had an open-issue, that makes perfect sense.

lloydbayley commented 1 month ago

How about this; Better?

root@development:~/.local/bin# ./proxmox-grapple --dump-config
header:
  env_filter: None
  key_filter: None
  new_first: 'True'
  history_limit: None
  include_internal: 'False'
current:
  BACKUP-START:
    script:
    - curl -fsS -m 10 --retry 5 -o /dev/null 'http://10.61.2.113/win&PL=06'
  BACKUP-END:
    extract:
      exclude_storeids: None
      enabled: false
    script:
    - curl -fsS -m 10 --retry 5 -o /dev/null 'http://10.61.2.113/win&PL=03'
  JOB-INIT:
    script: None
  JOB-START:
    script: None
  JOB-END:
    script: None
  JOB-ABORT:
    script: None
  BACKUP-ABORT:
    script: None
  LOG-END:
    script: None
  PRE-STOP:
    script: None
  PRE-RESTART:
    script: None
  POST-RESTART:
    script: None
history:
- loader: set_method
  identifier: undefined
  env: global
  merged: false
  value:
    JOB-INIT:
      script: None
    JOB-START:
      script: None
    JOB-END:
      script: None
    JOB-ABORT:
      script: None
    BACKUP-START:
      script:
      - curl -fsS -m 10 --retry 5 -o /dev/null 'http://10.61.2.113/win&PL=06'
    BACKUP-END:
      extract:
        exclude_storeids: None
        enabled: false
      script:
      - curl -fsS -m 10 --retry 5 -o /dev/null 'http://10.61.2.113/win&PL=03'
    BACKUP-ABORT:
      script: None
    LOG-END:
      script: None
    PRE-STOP:
      script: None
    PRE-RESTART:
      script: None
    POST-RESTART:
      script: None
- loader: setdefault
  identifier: unique
  env: production
  merged: false
  value:
    JOB-INIT:
      script: None
    JOB-START:
      script: None
    JOB-END:
      script: None
    JOB-ABORT:
      script: None
    BACKUP-ABORT:
      script: None
    LOG-END:
      script: None
    PRE-STOP:
      script: None
    PRE-RESTART:
      script: None
    POST-RESTART:
      script: None
    BACKUP-END:
      extract:
        exclude_storeids: None
        enabled: false
      script:
      - curl -fsS -m 10 --retry 5 -o /dev/null 'http://10.61.2.113/win&PL=03'
- loader: yaml
  identifier: /etc/proxmox_grapple.yml
  env: production
  merged: false
  value:
    BACKUP-START:
      script:
      - curl -fsS -m 10 --retry 5 -o /dev/null 'http://10.61.2.113/win&PL=06'
    BACKUP-END:
      script:
      - curl -fsS -m 10 --retry 5 -o /dev/null 'http://10.61.2.113/win&PL=03'
- loader: set_method
  identifier: settings_module_method
  env: global
  merged: false
  value:
    SETTINGS_MODULE:
    - proxmox_grapple.yml
    - /etc/proxmox_grapple.yml
- loader: set_method
  identifier: init_kwargs
  env: global
  merged: false
  value:
    ENVIRONMENTS_FOR_DYNACONF: true
    CORE_LOADERS_FOR_DYNACONF:
    - YAML
    SETTINGS_FILE_FOR_DYNACONF:
    - proxmox_grapple.yml
    - /etc/proxmox_grapple.yml
    ENV_FOR_DYNACONF: production
    APPLY_DEFAULT_ON_NONE_FOR_DYNACONF: true

lingfish commented 1 month ago

Much! The current: key is now filled?!? What did you do/change?

Oh, and with that change, does it now run?

lloydbayley commented 1 month ago

Do I have to tell you? :( I may have left the file renamed by accident during testing when I ran that dump before. Wouldn't THAT be embarrassing?! Also, alas, no, same result.

lingfish commented 1 month ago

Do I have to tell you? :( I may have left the file renamed by accident during testing when I ran that dump before. Wouldn't THAT be embarrassing?! Also, alas, no, same result.

lol, right, that unhurts my brain a bit because I just replicated what you're seeing, have a few ideas, and that would have borked everything :rofl:

So, a few issues here -- the dump-config isn't doing what I want it to do; that is, take into account any --config args before dumping, so kinda useless and I shall fix.

On using your first initial config both on a 1.5.4 and 1.5.5 install, I see it work fine on 1.5.4, but not 1.5.5. Doing some testing and comparisons to code and 3rd party differences now.

lloydbayley commented 1 month ago

Ahhh....well at least we're getting somewhere! sorry about that earlier. As I said, this is what happens when I do things in the wee-small hours and forget to put them back... :)

lingfish commented 1 month ago

Ahhh....well at least we're getting somewhere! sorry about that earlier. As I said, this is what happens when I do things in the wee-small hours and forget to put them back... :)

Nah nah, all good!

My turn to be embarrassed -- I broke it in 1.5.5... and it has revealed my unit tests didn't catch the breakage too.

Will fix tomorrow. If urgent, or just for fun, downgrade to 1.5.4 and it should work fine:

host ~ $ pipx uninstall proxmox-grapple
uninstalled proxmox-grapple! ✨ 🌟 ✨
host ~ $ pipx install proxmox-grapple==1.5.4
  installed package proxmox-grapple 1.5.4, installed using Python 3.11.2
  These apps are now globally available
    - proxmox-grapple
    - vzdump-hook-script

lloydbayley commented 1 month ago

Not urgent at all. Just happy to help out. It actually changes the RGB lights in my rack when a backup happens so it's purely decorative but I like to automate tnings. It's a bit of a challenge and helps hone my skills in everything and keeps me up to date.

If it helps for tomorrow, (and I have no idea WHY) but if I send a: root@development:~/.local/bin# ./proxmox-grapple backup-start It actually works and changes the leds as expected.

root@development:~/.local/bin# ./proxmox-grapple backup-start
HOOK: ./proxmox-grapple backup-start
HOOK-ENV: vmtype=None; dumpdir=None; storeid=None; hostname=None; tarfile=None; logfile=None
    Running (mode script): curl -fsS -m 10 --retry 5 -o /dev/null 'http://10.61.2.113/win&PL=06'

However... root@development:~/.local/bin# ./proxmox-grapple backup-end fails miserably with;

root@development:~/.local/bin# ./proxmox-grapple backup-end
HOOK: ./proxmox-grapple backup-end
HOOK-ENV: vmtype=None; dumpdir=None; storeid=None; hostname=None; tarfile=None; logfile=None
    Running (mode extract): exclude_storeids
BACKUP-END: ERROR! Something went wrong
    [Errno 2] No such file or directory: 'exclude_storeids'
    Running (mode extract): enabled
BACKUP-END: ERROR! Something went wrong
    [Errno 2] No such file or directory: 'enabled'

and the job-init still yields our old favourite, the original error.

Just a FYI! Have a good evening! :)

lloydbayley commented 1 month ago

Just for reference, downgrading broke the backup-start:

root@development:~/.local/bin# ./proxmox-grapple backup-start
HOOK: ./proxmox-grapple backup-start
HOOK-ENV: vmtype=None; dumpdir=None; storeid=None; hostname=None; tarfile=None; logfile=None
    Running: curl -fsS -m 10 --retry 5 -o /dev/null 'http://10.61.2.113/win&PL=06'
    curl: (3) URL using bad/illegal format or missing URL
BACKUP-START: ERROR! Something went wrong
    Command '['curl', '-fsS', '-m', '10', '--retry', '5', '-o', '/dev/null', "'http://10.61.2.113/win&PL=06'"]' returned non-zero exit status 3.

Anyway...Absolutely no rush. In your own time. As I said, It's in no way urgent.

lingfish commented 1 month ago

Right, this has prompted me to do a bit of a rewrite; I didn't foresee the person in #2 wanting two different run modes, and then I implemented them in the way I did (lazily and badly).

So this means a breaking change update, will be pushing that out soon. It should fix all things in this issue, and will now dump config after it is parsed.

lingfish commented 1 month ago

Done and released as 2.0.0, give it a whirl and let me know, thanks :smile:

It actually changes the RGB lights in my rack when a backup happens so it's purely decorative but I like to automate tnings. It's a bit of a challenge and helps hone my skills in everything and keeps me up to date.

Also, love this! Big fan of home automation, and Home Assistant myself :grin:

lloydbayley commented 1 month ago

I see you got a lot of feedback and thanks in #2 as well! :) Will give the new version a spin in a minute. I've just come in from the shed with non-keyboard hands from turning things on the lathe and I not only need Solvol but a vodka and tonic. More soon!

lloydbayley commented 1 month ago

Ooh err.... I think I broke it...

INFO: HOOK: /root/.local/bin/proxmox-grapple job-init
INFO: HOOK-ENV: vmtype=None; dumpdir=None; storeid=local; hostname=None; tarfile=None; logfile=None
INFO: backup-start.mode is required in env production
INFO: notified via target `mail-to-root`
TASK ERROR: command '/root/.local/bin/proxmox-grapple job-init' failed: exit code 1

Let me investigate...

lloydbayley commented 1 month ago

Ok. Did some more testing under previous commits , It works in 1.5.4 if I remove the quotes from my curl statement and just leave:

backup-start:
    script:
      - curl -fsS -m 10 --retry 5 -o /dev/null http://10.61.2.113/win&PL=06

However, it gives the .mode error in v2.00 (Bugger!)

lingfish commented 1 month ago

Hrm, ok... Attach new config, and a dump please?

lloydbayley commented 1 month ago

Config:

root@development:/etc# cat proxmox_grapple.yml
production:
  backup-start:
    script:
      - curl -fsS -m 10 --retry 5 -o /dev/null http://10.61.2.113/win&PL=06
#  backup-abort:
#    script:
#      - "curl -fsS -m 10 --retry 5 -o /dev/null 'http://10.61.2.113/win&PL=03'"
  backup-end:
    script:
      - curl -fsS -m 10 --retry 5 -o /dev/null http://10.61.2.113/win&PL=03

Dump didn't go well... (unlike the one I had earlier...Ahem)

root@development:~/.local/bin# ./proxmox-grapple --dump-config
Usage: proxmox-grapple [OPTIONS] PHASE [MODE] [VMID]
Try 'proxmox-grapple --help' for help.

Error: Missing argument 'PHASE'.

lingfish commented 1 month ago

Ah, right... And you're on 2.0.0?

If so, see the README 😉

You need to update your config, version 2 brings a breaking API/config format change.

lloydbayley commented 1 month ago

Oh crap, sorry. Even think about RTFM! :)

lloydbayley commented 1 month ago

Amazing what happens when you think like an end-user instead of a dev. Works like a dream! Thank a lot!

lingfish commented 1 month ago

Lovely! Good to hear.

Thanks heaps for helping out here, appreciate it I might just leave this open a few days.

lloydbayley commented 1 month ago

Not a problem. Glad to help out. If you want any more testing in the future, I will leave it on my development server so I can play around/test things without fear or recrimination! :) Just let me know.

lingfish / proxmox-grapple

TypeError: 'NoneType' object is not iterable #6