search

pyinfra-dev / pyinfra

pyinfra turns Python code into shell commands and runs them on your servers. Execute ad-hoc commands and write declarative operations. Target SSH servers, local machine and Docker containers. Fast and scales from one server to thousands.
https://pyinfra.com
MIT License
3.91k stars 382 forks source link

Not clear how to make operations be followed by fact gathering - make them be executed linearly #889

Closed romanchyla closed 9 months ago

romanchyla commented 2 years ago

Describe the bug

Pyinfra gives impression that the flow is synchronous; but behind the scenes operations will be executed after fact_gathering - even if the underlying facts are created by the operation. What is the recommended way to deal with the following scenario?

  1. install (modify) a server
  2. based on the installation (change), make further updates

To Reproduce

To illustrate, this is the deployment

@deploy("Pycharm Ultimate")
def pycharm(_sudo=True):
    homedir = host.get_fact(Home)
    tar_file = "/tmp/pycharm-professional-2022.2.2.tar.gz"
    if not host.get_fact(File, tar_file):
        files.download(
            "https://download.jetbrains.com/python/pycharm-professional-2022.2.2.tar.gz",
            tar_file,
        )

        server.shell(
            commands=[
                "tar -xvf {} -C {}".format(tar_file, homedir),
                "ln -s {home}/pycharm `find {home} -name `pycharm-*`".format(home=homedir),
            ]
        )

    dirs = host.get_fact(ListDirectories, homedir, 1)
    target = sorted(filter(lambda x: "pycharm-" in x, dirs))[-1]
    files.link("{}/pycharm".format(homedir), target=target, symbolic=True, force=True)

Expected behavior

Initially, when writing it I was expecting the files.download and server.shell to happen before host.get_fact -- I'm reading it top down, linearly

Pyinfra will however register server shell operations first, but facts (host.get_fact(ListDirectories)) will be called before the operations - which will fail because the folder doesn't exist yet.

What is the right way/pattern to structure such operations? I would like the fact gathering to happen (sometimes) after the operations

Meta

--> Support information:

    If you are having issues with pyinfra or wish to make feature requests, please
    check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues .
    When adding an issue, be sure to include the following:

    System: Linux
      Platform: Linux-5.15.0-48-generic-x86_64-with-glibc2.35
      Release: 5.15.0-48-generic
      Machine: x86_64
    pyinfra: v2.4
    Executable: /home/ANT.AMAZON.COM/rchyla/BoringStuffAutomated/.venv/bin/pyinfra
    Python: 3.10.6 (CPython, GCC 11.3.0)
pyinfra @local bsa.setup.ubuntu.jetbrains.pycharm -vv --debug
--> Loading config...
--> Loading inventory...
    [pyinfra_cli.inventory] Creating fake inventory...
    [pyinfra_cli.inventory] Checking possible group_data directory: /home/ANT.AMAZON.COM/rchyla/BoringStuffAutomated
    [pyinfra_cli.inventory] Looking for group data in: /home/ANT.AMAZON.COM/rchyla/BoringStuffAutomated/group_data/all.py

--> Connecting to hosts...
    [@local] Connected
INFO:pyinfra:[@local] Connected
    [pyinfra.api.state] Activating host: @local
DEBUG:pyinfra:Activating host: @local

--> Preparing operation...
    [pyinfra.api.host] Starting deploy Pycharm Ultimate (args={'sudo': False, 'sudo_user': None, 'use_sudo_login': False, 'use_sudo_password': False, 'preserve_sudo_env': False, 'su_user': None, 'use_su_login': False, 'preserve_su_env': False, 'su_shell': None, 'doas': None, 'doas_user': None, 'shell_executable': 'sh', 'chdir': None, 'env': {}, 'success_exit_codes': [0], 'timeout': None, 'get_pty': None, 'stdin': None, 'name': None, 'ignore_errors': False, 'continue_on_error': False, 'precondition': None, 'postcondition': None, 'on_success': None, 'on_error': None, 'parallel': 1, 'run_once': False, 'serial': False}, data=None)
DEBUG:pyinfra:Starting deploy Pycharm Ultimate (args={'sudo': False, 'sudo_user': None, 'use_sudo_login': False, 'use_sudo_password': False, 'preserve_sudo_env': False, 'su_user': None, 'use_su_login': False, 'preserve_su_env': False, 'su_shell': None, 'doas': None, 'doas_user': None, 'shell_executable': 'sh', 'chdir': None, 'env': {}, 'success_exit_codes': [0], 'timeout': None, 'get_pty': None, 'stdin': None, 'name': None, 'ignore_errors': False, 'continue_on_error': False, 'precondition': None, 'postcondition': None, 'on_success': None, 'on_error': None, 'parallel': 1, 'run_once': False, 'serial': False}, data=None)
    [pyinfra.api.facts] Getting fact: server.Home () (ensure_hosts: None)
DEBUG:pyinfra:Getting fact: server.Home () (ensure_hosts: None)
[pyinfra.connectors.local] --> Running command on localhost: sh -c 'echo $HOME'
DEBUG:pyinfra:--> Running command on localhost: sh -c 'echo $HOME'
[@local] >>> sh -c 'echo $HOME'
[pyinfra.connectors.util] --> Waiting for exit status...
DEBUG:pyinfra:--> Waiting for exit status...
[pyinfra.connectors.util] --> Command exit status: 0
DEBUG:pyinfra:--> Command exit status: 0
    [@local] Loaded fact server.Home
INFO:pyinfra:[@local] Loaded fact server.Home
    [pyinfra.api.facts] Getting fact: files.File (path=/tmp/pycharm-professional-2022.2.2.tar.gz) (ensure_hosts: None)
DEBUG:pyinfra:Getting fact: files.File (path=/tmp/pycharm-professional-2022.2.2.tar.gz) (ensure_hosts: None)
[pyinfra.connectors.local] --> Running command on localhost: sh -c '! (test -e /tmp/pycharm-professional-2022.2.2.tar.gz || test -L /tmp/pycharm-professional-2022.2.2.tar.gz ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /tmp/pycharm-professional-2022.2.2.tar.gz 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' /tmp/pycharm-professional-2022.2.2.tar.gz )'
DEBUG:pyinfra:--> Running command on localhost: sh -c '! (test -e /tmp/pycharm-professional-2022.2.2.tar.gz || test -L /tmp/pycharm-professional-2022.2.2.tar.gz ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /tmp/pycharm-professional-2022.2.2.tar.gz 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' /tmp/pycharm-professional-2022.2.2.tar.gz )'
[@local] >>> sh -c '! (test -e /tmp/pycharm-professional-2022.2.2.tar.gz || test -L /tmp/pycharm-professional-2022.2.2.tar.gz ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /tmp/pycharm-professional-2022.2.2.tar.gz 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' /tmp/pycharm-professional-2022.2.2.tar.gz )'
[pyinfra.connectors.util] --> Waiting for exit status...
DEBUG:pyinfra:--> Waiting for exit status...
[pyinfra.connectors.util] --> Command exit status: 0
DEBUG:pyinfra:--> Command exit status: 0
    [@local] Loaded fact files.File (path=/tmp/pycharm-professional-2022.2.2.tar.gz)
INFO:pyinfra:[@local] Loaded fact files.File (path=/tmp/pycharm-professional-2022.2.2.tar.gz)
    [pyinfra.api.operation] Adding operation, {'Pycharm Ultimate | Files/Download'}, opOrder=(0, 5, 1027, 1006, 688, 883, 241, 42, 1130, 1055, 1404, 760, 223, 572, 67), opHash=f26ddf87fd11d3f7436a08a72fbe886a7b4823ae
DEBUG:pyinfra:Adding operation, {'Pycharm Ultimate | Files/Download'}, opOrder=(0, 5, 1027, 1006, 688, 883, 241, 42, 1130, 1055, 1404, 760, 223, 572, 67), opHash=f26ddf87fd11d3f7436a08a72fbe886a7b4823ae
    [pyinfra.api.facts] Getting fact: files.File (path=/tmp/pycharm-professional-2022.2.2.tar.gz) (ensure_hosts: None)
DEBUG:pyinfra:Getting fact: files.File (path=/tmp/pycharm-professional-2022.2.2.tar.gz) (ensure_hosts: None)
[pyinfra.connectors.local] --> Running command on localhost: sh -c '! (test -e /tmp/pycharm-professional-2022.2.2.tar.gz || test -L /tmp/pycharm-professional-2022.2.2.tar.gz ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /tmp/pycharm-professional-2022.2.2.tar.gz 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' /tmp/pycharm-professional-2022.2.2.tar.gz )'
DEBUG:pyinfra:--> Running command on localhost: sh -c '! (test -e /tmp/pycharm-professional-2022.2.2.tar.gz || test -L /tmp/pycharm-professional-2022.2.2.tar.gz ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /tmp/pycharm-professional-2022.2.2.tar.gz 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' /tmp/pycharm-professional-2022.2.2.tar.gz )'
[@local] >>> sh -c '! (test -e /tmp/pycharm-professional-2022.2.2.tar.gz || test -L /tmp/pycharm-professional-2022.2.2.tar.gz ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /tmp/pycharm-professional-2022.2.2.tar.gz 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' /tmp/pycharm-professional-2022.2.2.tar.gz )'
[pyinfra.connectors.util] --> Waiting for exit status...
DEBUG:pyinfra:--> Waiting for exit status...
[pyinfra.connectors.util] --> Command exit status: 0
DEBUG:pyinfra:--> Command exit status: 0
    [@local] Loaded fact files.File (path=/tmp/pycharm-professional-2022.2.2.tar.gz)
INFO:pyinfra:[@local] Loaded fact files.File (path=/tmp/pycharm-professional-2022.2.2.tar.gz)
    [pyinfra.api.facts] Getting fact: server.Date () (ensure_hosts: None)
DEBUG:pyinfra:Getting fact: server.Date () (ensure_hosts: None)
[pyinfra.connectors.local] --> Running command on localhost: sh -c 'LANG=C LC_TIME=en_US.UTF-8 date'
DEBUG:pyinfra:--> Running command on localhost: sh -c 'LANG=C LC_TIME=en_US.UTF-8 date'
[@local] >>> sh -c 'LANG=C LC_TIME=en_US.UTF-8 date'
[pyinfra.connectors.util] --> Waiting for exit status...
DEBUG:pyinfra:--> Waiting for exit status...
[pyinfra.connectors.util] --> Command exit status: 0
DEBUG:pyinfra:--> Command exit status: 0
    [@local] Loaded fact server.Date
INFO:pyinfra:[@local] Loaded fact server.Date
    [pyinfra.api.facts] Getting fact: server.Which (command=curl) (ensure_hosts: None)
DEBUG:pyinfra:Getting fact: server.Which (command=curl) (ensure_hosts: None)
[pyinfra.connectors.local] --> Running command on localhost: sh -c 'which curl || true'
DEBUG:pyinfra:--> Running command on localhost: sh -c 'which curl || true'
[@local] >>> sh -c 'which curl || true'
[pyinfra.connectors.util] --> Waiting for exit status...
DEBUG:pyinfra:--> Waiting for exit status...
[pyinfra.connectors.util] --> Command exit status: 0
DEBUG:pyinfra:--> Command exit status: 0
    [@local] Loaded fact server.Which (command=curl)
INFO:pyinfra:[@local] Loaded fact server.Which (command=curl)
    [pyinfra.api.operation] Adding operation, {'Pycharm Ultimate | Server/Shell'}, opOrder=(0, 5, 1027, 1006, 688, 883, 241, 42, 1130, 1055, 1404, 760, 223, 572, 72), opHash=d4f917056a0df8fc9ea5dcf48de1e75c07ec507a
DEBUG:pyinfra:Adding operation, {'Pycharm Ultimate | Server/Shell'}, opOrder=(0, 5, 1027, 1006, 688, 883, 241, 42, 1130, 1055, 1404, 760, 223, 572, 72), opHash=d4f917056a0df8fc9ea5dcf48de1e75c07ec507a
    [pyinfra.api.facts] Getting fact: bsa.pyinfra.ubuntu.facts.system_info.ListDirectories (max_depth=1, path=/home/ANT.AMAZON.COM/rchyla) (ensure_hosts: None)
DEBUG:pyinfra:Getting fact: bsa.pyinfra.ubuntu.facts.system_info.ListDirectories (max_depth=1, path=/home/ANT.AMAZON.COM/rchyla) (ensure_hosts: None)
[pyinfra.connectors.local] --> Running command on localhost: sh -c 'find  "/home/ANT.AMAZON.COM/rchyla" -type d -maxdepth 1'
DEBUG:pyinfra:--> Running command on localhost: sh -c 'find  "/home/ANT.AMAZON.COM/rchyla" -type d -maxdepth 1'
[@local] >>> sh -c 'find  "/home/ANT.AMAZON.COM/rchyla" -type d -maxdepth 1'
[pyinfra.connectors.util] --> Waiting for exit status...
DEBUG:pyinfra:--> Waiting for exit status...
[pyinfra.connectors.util] --> Command exit status: 0
DEBUG:pyinfra:--> Command exit status: 0
    [@local] Loaded fact bsa.pyinfra.ubuntu.facts.system_info.ListDirectories (max_depth=1, path=/home/ANT.AMAZON.COM/rchyla)
INFO:pyinfra:[@local] Loaded fact bsa.pyinfra.ubuntu.facts.system_info.ListDirectories (max_depth=1, path=/home/ANT.AMAZON.COM/rchyla)
--> An unexpected internal exception occurred:

  File "/home/ANT.AMAZON.COM/rchyla/BoringStuffAutomated/bsa/setup/ubuntu/jetbrains.py", line 80, in pycharm
    target = sorted(filter(lambda x: "pycharm-" in x, dirs))[-1]
IndexError: list index out of range

    [pyinfra_cli.exceptions]   File "/home/ANT.AMAZON.COM/rchyla/BoringStuffAutomated/.venv/lib/python3.10/site-packages/pyinfra_cli/main.py", line 223, in cli
    _main(*args, **kwargs)
  File "/home/ANT.AMAZON.COM/rchyla/BoringStuffAutomated/.venv/lib/python3.10/site-packages/pyinfra_cli/main.py", line 572, in _main
    add_op(state, op, *args, **kwargs)
  File "/home/ANT.AMAZON.COM/rchyla/BoringStuffAutomated/.venv/lib/python3.10/site-packages/pyinfra/api/operation.py", line 109, in add_op
    results[op_host] = op_func(*args, **kwargs)
  File "/home/ANT.AMAZON.COM/rchyla/BoringStuffAutomated/.venv/lib/python3.10/site-packages/pyinfra/api/deploy.py", line 112, in decorated_func
    return func(*args, **kwargs)
  File "/home/ANT.AMAZON.COM/rchyla/BoringStuffAutomated/bsa/setup/ubuntu/jetbrains.py", line 80, in pycharm
    target = sorted(filter(lambda x: "pycharm-" in x, dirs))[-1]

DEBUG:pyinfra:  File "/home/ANT.AMAZON.COM/rchyla/BoringStuffAutomated/.venv/lib/python3.10/site-packages/pyinfra_cli/main.py", line 223, in cli
    _main(*args, **kwargs)
  File "/home/ANT.AMAZON.COM/rchyla/BoringStuffAutomated/.venv/lib/python3.10/site-packages/pyinfra_cli/main.py", line 572, in _main
    add_op(state, op, *args, **kwargs)
  File "/home/ANT.AMAZON.COM/rchyla/BoringStuffAutomated/.venv/lib/python3.10/site-packages/pyinfra/api/operation.py", line 109, in add_op
    results[op_host] = op_func(*args, **kwargs)
  File "/home/ANT.AMAZON.COM/rchyla/BoringStuffAutomated/.venv/lib/python3.10/site-packages/pyinfra/api/deploy.py", line 112, in decorated_func
    return func(*args, **kwargs)
  File "/home/ANT.AMAZON.COM/rchyla/BoringStuffAutomated/bsa/setup/ubuntu/jetbrains.py", line 80, in pycharm
    target = sorted(filter(lambda x: "pycharm-" in x, dirs))[-1]

    [pyinfra_cli.exceptions] IndexError: list index out of range

DEBUG:pyinfra:IndexError: list index out of range

--> The full traceback has been written to pyinfra-debug.log
--> If this is unexpected please consider submitting a bug report on GitHub, for more information run `pyinfra --support`.
Fizzadar commented 2 years ago

Hi @romanchyla! So currently the best way to achieve this kind of thing is using nested operations: https://docs.pyinfra.com/en/2.x/using-operations.html#nested-operations

These get executed immediately and so allow for the flow of operations you need. If a fact has already been cached you can explicitly force it to be re-collected using host.reload_fact(...).