search

openSUSE / salt

openSUSE and SUSE patches and backports for SaltStack
Apache License 2.0
23 stars 52 forks source link

[BUG] Salt minion bundle broken for openEuler #577

Closed raulillo82 closed 1 year ago

raulillo82 commented 1 year ago

Description Salt minion bundle stopped working. All comes from the implementation of openEuler client support for Uyuni, see https://github.com/uyuni-project/uyuni/pull/6623

Setup

Steps to Reproduce the behavior I was using a PoC for the salt bundle from this repo in July, which used to work: https://download.opensuse.org/repositories/home:/vzhestkov:/saltstack:/bundle:/next/AlmaLinux_8/ I can see the package got rebuilt in December, and it is not working any more.

Error in the web-ui: 

stderr: "Traceback (most recent call last):
  File "/var/tmp/venv-salt-minion/bin/salt-call", line 10, in <module>
    salt_call()
  File "/var/tmp/venv-salt-minion/lib64/python3.10/site-packages/salt/scripts.py", line 426, in salt_call
    import salt.cli.call
  File "/var/tmp/venv-salt-minion/lib64/python3.10/site-packages/salt/cli/call.py", line 3, in <module>
    import salt.cli.caller
  File "/var/tmp/venv-salt-minion/lib64/python3.10/site-packages/salt/cli/caller.py", line 14, in <module>
    import salt.loader
  File "/var/tmp/venv-salt-minion/lib64/python3.10/site-packages/salt/loader/__init__.py", line 15, in <module>
    import salt.config
  File "/var/tmp/venv-salt-minion/lib64/python3.10/site-packages/salt/config/__init__.py", line 26, in <module>
    import salt.utils.user
  File "/var/tmp/venv-salt-minion/lib64/python3.10/site-packages/salt/utils/user.py", line 7, in <module>
    import ctypes
  File "/var/tmp/venv-salt-minion/lib64/python3.10/ctypes/__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ImportError: libffi.so.6: cannot open shared object file: No such file or directory", stdout: ""

I tried to fool the system and link libffi:

[root@vm-panaditas ~]# ln -s /usr/lib64/libffi.so.8.1.0 /usr/lib64/libffi.so.6
[root@vm-panaditas ~]# ls -l /usr/lib64/libffi.so*
lrwxrwxrwx. 1 root root    15 Mar 15  2022 /usr/lib64/libffi.so -> libffi.so.8.1.0
lrwxrwxrwx. 1 root root    26 Feb  2 16:27 /usr/lib64/libffi.so.6 -> /usr/lib64/libffi.so.8.1.0
lrwxrwxrwx. 1 root root    15 Mar 15  2022 /usr/lib64/libffi.so.8 -> libffi.so.8.1.0
-rwxr-xr-x. 1 root root 43264 Mar 15  2022 /usr/lib64/libffi.so.8.1.0

But obviously it wasn't that easy. The error in the web-ui changed to a non-descriptive "An error has occurred during salt execution: unable to parse json.". The bundle rpm doesn't even get installed (see later for the manual installation). More info can be checked in the salt master api log:

2023-02-02 16:28:30,004 [salt.utils.templates:274 ][ERROR   ][25840] Rendering exception occurred
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 501, in render_jinja_tmpl
    template = jinja_env.from_string(tmplstr)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 880, in from_string
    return cls.from_code(self, self.compile(source), globals, None)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 591, in compile
    self.handle_exception(exc_info, source_hint=source_hint)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<unknown>", line 302, in template
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 497, in _parse
    return Parser(self, source, name, encode_filename(filename)).parse()
  File "/usr/lib/python3.6/site-packages/jinja2/parser.py", line 901, in parse
    result = nodes.Template(self.subparse(), lineno=1)
  File "/usr/lib/python3.6/site-packages/jinja2/parser.py", line 883, in subparse
    rv = self.parse_statement()
  File "/usr/lib/python3.6/site-packages/jinja2/parser.py", line 130, in parse_statement
    return getattr(self, 'parse_' + self.stream.current.value)()
  File "/usr/lib/python3.6/site-packages/jinja2/parser.py", line 213, in parse_if
    'name:endif'))
  File "/usr/lib/python3.6/site-packages/jinja2/parser.py", line 170, in parse_statements
    self.fail_eof(end_tokens)
  File "/usr/lib/python3.6/site-packages/jinja2/parser.py", line 104, in fail_eof
    return self._fail_ut_eof(None, stack, lineno)
  File "/usr/lib/python3.6/site-packages/jinja2/parser.py", line 90, in _fail_ut_eof
    self.fail(' '.join(message), lineno)
  File "/usr/lib/python3.6/site-packages/jinja2/parser.py", line 59, in fail
    raise exc(msg, lineno, self.name, self.filename)
jinja2.exceptions.TemplateSyntaxError: Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 261, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 520, in render_jinja_tmpl
    "Jinja syntax error: {}{}".format(exc, out), line, tmplstr
salt.exceptions.SaltRenderError: Jinja syntax error: Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.; line 302

---
[...]
reboot_transactional_server:
  mgrcompat.module_run:
    - name: transactional_update.reboot
    - require:
      - {{ salt_minion_name }}
{%- endif %}    <======================
---
2023-02-02 16:28:30,006 [salt.state       :4099][CRITICAL][25840] Rendering SLS 'base:bootstrap' failed: Jinja syntax error: Unexpected end of template. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.; line 302

---
[...]
reboot_transactional_server:
  mgrcompat.module_run:
    - name: transactional_update.reboot
    - require:
      - {{ salt_minion_name }}
{%- endif %}    <======================
---

Expected behavior Salt bundle should work just fine. Something wrong with an old version libffi?

Screenshots NA

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) Master (uyuni): ``` vm-cachitomio:~ # salt --versions-report Salt Version: Salt: 3004 Dependency Versions: cffi: 1.13.2 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 2.10.1 libgit2: 1.3.0 M2Crypto: 0.38.0 Mako: Not Installed msgpack: 0.5.6 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: 2.17 pycrypto: 3.9.0 pycryptodome: Not Installed pygit2: 1.7.0 Python: 3.6.15 (default, Sep 23 2021, 15:41:43) [GCC] python-gnupg: Not Installed PyYAML: 5.4.1 PyZMQ: 17.1.2 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.2.3 System Versions: dist: opensuse-leap 15.4 locale: UTF-8 machine: x86_64 release: 5.14.21-150400.24.41-default system: Linux version: openSUSE Leap 15.4 ``` Minion (after manually installing the package venv-salt-minion-3004-54.7.x86_64): ``` [root@vm-panaditas ~]# venv-salt-minion --versions-report Salt Version: Salt: 3004 Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: Not Installed docker-py: 4.2.0 gitdb: Not Installed gitpython: Not Installed Jinja2: 3.0.3 libgit2: Not Installed M2Crypto: 0.38.0 Mako: Not Installed msgpack: 0.5.6 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: Not Installed pycrypto: Not Installed pycryptodome: Not Installed pygit2: Not Installed Python: 3.10.5 (main, Jul 20 2022, 09:44:52) [GCC] python-gnupg: Not Installed PyYAML: 5.4.1 PyZMQ: 23.2.0 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.2.3 System Versions: dist: openeuler 22.03 locale: utf-8 machine: x86_64 release: 5.10.0-60.18.0.50.oe2203.x86_64 system: Linux version: openEuler 22.03 ``` ``` PASTE HERE ```

Additional context Add any other context about the problem here.

raulillo82 commented 1 year ago

It wasn't a full valid test, but the preliminary ones on 2023.09 were already showing the same error with the venv-salt-minion package from EL8 repo. Same with the ones that was succeeding at the beginning (very long ago), from https://download.opensuse.org/repositories/home:/vzhestkov:/saltstack:/bundle:/next/AlmaLinux_8/ (that's "old" salt 3004 anyway). @vzhestkov maybe it's already ready for you to have a look at it whenever you have time.

vzhestkov commented 1 year ago

The repo is outdated. And it's assigned to my previous account. Could you please try to use the package from this project instead: https://build.opensuse.org/project/show/systemsmanagement:saltstack:bundle:next:AlmaLinux8

The build for some of the packages is failing now (will fix them soon), but one of the previous builds of the salt bundle package is published in the repo.

raulillo82 commented 1 year ago

Same problem, even installing the package manually, outside of the context of Uyuni. It refuses to start, complaining about the same libffi:

[root@vm-panaditas tmp]# rpm -ivh systemd-private-5fa6034adfd9439bbae515dc6a68788c-^C
[root@vm-panaditas tmp]# rpm -ivh venv-salt-minion-3006.0-16.2.uyuni.x86_64.rpm 
warning: venv-salt-minion-3006.0-16.2.uyuni.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 0d20833e: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:venv-salt-minion-3006.0-16.2.uyun################################# [100%]
uavc:  op=load_policy lsm=selinux seqno=2 res=1[root@vm-panaditas tmp]# systemctl restart venv-salt-minion.service 
[root@vm-panaditas tmp]# journalctl --no-pager -u venv-salt-minion.service 
-- Journal begins at Mon 2023-10-02 11:42:48 CEST, ends at Mon 2023-10-02 11:58:10 CEST. --
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de systemd[1]: Started The venvjailed Salt Minion.
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]: Traceback (most recent call last):
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:   File "/usr/lib/venv-salt-minion/bin/salt-minion", line 27, in <module>
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:     salt_minion()
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:   File "/usr/lib/venv-salt-minion/lib64/python3.10/site-packages/salt/scripts.py", line 175, in salt_minion
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:     import salt.cli.daemons
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:   File "/usr/lib/venv-salt-minion/lib64/python3.10/site-packages/salt/cli/daemons.py", line 47, in <module>
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:     import salt.utils.parsers
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:   File "/usr/lib/venv-salt-minion/lib64/python3.10/site-packages/salt/utils/parsers.py", line 26, in <module>
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:     import salt.config as config
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:   File "/usr/lib/venv-salt-minion/lib64/python3.10/site-packages/salt/config/__init__.py", line 26, in <module>
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:     import salt.utils.user
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:   File "/usr/lib/venv-salt-minion/lib64/python3.10/site-packages/salt/utils/user.py", line 7, in <module>
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:     import ctypes
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:   File "/usr/lib/venv-salt-minion/lib64/python3.10/ctypes/__init__.py", line 8, in <module>
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:     from _ctypes import Union, Structure, Array
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]: ImportError: libffi.so.6: cannot open shared object file: No such file or directory
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]: [ERROR   ] An un-handled exception was caught by Salt's global exception handler:
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]: ImportError: libffi.so.6: cannot open shared object file: No such file or directory
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]: Traceback (most recent call last):
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:   File "/usr/lib/venv-salt-minion/bin/salt-minion", line 27, in <module>
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:     salt_minion()
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:   File "/usr/lib/venv-salt-minion/lib64/python3.10/site-packages/salt/scripts.py", line 175, in salt_minion
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:     import salt.cli.daemons
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:   File "/usr/lib/venv-salt-minion/lib64/python3.10/site-packages/salt/cli/daemons.py", line 47, in <module>
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:     import salt.utils.parsers
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:   File "/usr/lib/venv-salt-minion/lib64/python3.10/site-packages/salt/utils/parsers.py", line 26, in <module>
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:     import salt.config as config
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:   File "/usr/lib/venv-salt-minion/lib64/python3.10/site-packages/salt/config/__init__.py", line 26, in <module>
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:     import salt.utils.user
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:   File "/usr/lib/venv-salt-minion/lib64/python3.10/site-packages/salt/utils/user.py", line 7, in <module>
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:     import ctypes
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:   File "/usr/lib/venv-salt-minion/lib64/python3.10/ctypes/__init__.py", line 8, in <module>
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]:     from _ctypes import Union, Structure, Array
Oct 02 11:58:09 vm-panaditas.hwlab.suse.de salt-minion[6129]: ImportError: libffi.so.6: cannot open shared object file: No such file or directory
Oct 02 11:58:10 vm-panaditas.hwlab.suse.de systemd[1]: venv-salt-minion.service: Main process exited, code=exited, status=1/FAILURE
Oct 02 11:58:10 vm-panaditas.hwlab.suse.de systemd[1]: venv-salt-minion.service: Failed with result 'exit-code'.
[root@vm-panaditas tmp]# rpm -qa|grep -i venv
venv-salt-minion-3006.0-16.2.uyuni.x86_64
[root@vm-panaditas tmp]# cat /etc/os-release 
NAME="openEuler"
VERSION="22.03 LTS"
ID="openEuler"
VERSION_ID="22.03"
PRETTY_NAME="openEuler 22.03 LTS"
ANSI_COLOR="0;31"
vzhestkov commented 1 year ago

Could you please check which libffi is present in the standard openEuler repos and which .so does it countain, actually there could be multiple libffi versions like libffi6 or libffi8. For me it's still unclear if we can use the bundle package we build for EL8 or it's better to build openEuler specific package.

vzhestkov commented 1 year ago

As an alternative it could be really easier just to copy into the bundle package from the build worker, but for me this solution looks very hacky and we are loosing the control of what sources was used to build this binary library.

raulillo82 commented 1 year ago 
[root@vm-panaditas tmp]# rpm -qa|grep -i libffi
libffi-3.4.2-2.oe2203.x86_64
libffi-devel-3.4.2-2.oe2203.x86_64
[root@vm-panaditas tmp]# rpm -ql libffi
/etc/ima/digest_lists.tlv/0-metadata_list-compact_tlv-libffi-3.4.2-2.oe2203.x86_64
/etc/ima/digest_lists/0-metadata_list-compact-libffi-3.4.2-2.oe2203.x86_64
/usr/lib64/libffi.so.8
/usr/lib64/libffi.so.8.1.0
/usr/share/licenses/libffi
/usr/share/licenses/libffi/LICENSE
[root@vm-panaditas tmp]# rpm -ql libffi-devel
/etc/ima/digest_lists.tlv/0-metadata_list-compact_tlv-libffi-devel-3.4.2-2.oe2203.x86_64
/etc/ima/digest_lists/0-metadata_list-compact-libffi-devel-3.4.2-2.oe2203.x86_64
/usr/include/ffi.h
/usr/include/ffitarget.h
/usr/lib64/libffi.so
/usr/lib64/pkgconfig/libffi.pc
[root@vm-panaditas tmp]# ls -l /usr/lib64/libffi.so /usr/lib64/libffi.so.8 /usr/lib64/libffi.so.8.1.0
lrwxrwxrwx. 1 root root    15 Mar 15  2022 /usr/lib64/libffi.so -> libffi.so.8.1.0
lrwxrwxrwx. 1 root root    15 Mar 15  2022 /usr/lib64/libffi.so.8 -> libffi.so.8.1.0
-rwxr-xr-x. 1 root root 43264 Mar 15  2022 /usr/lib64/libffi.so.8.1.0
vzhestkov commented 1 year ago

The problem that AlmaLinux 8 we are using to build the bundle is using the following:

[root@vz-alma8 ~]# rpm -qa | grep libffi
libffi-3.1-24.el8.x86_64
[root@vz-alma8 ~]# rpm -ql libffi
/usr/lib/.build-id
/usr/lib/.build-id/59
/usr/lib/.build-id/59/860b98f7e57b8388e74dfc44734013bf9c9b99
/usr/lib64/libffi.so.6
/usr/lib64/libffi.so.6.0.2
/usr/share/doc/libffi
/usr/share/doc/libffi/README
/usr/share/licenses/libffi
/usr/share/licenses/libffi/LICENSE

So it's not really compatible with openEuler on that level. I would suggest to build separate bundle package for openEuler at least we already have openEuler:22.03 in OBS.

raulillo82 commented 1 year ago

A new salt bundle was built, specifically for openEuler, where the error disappeared. It will be needed to adapt the url's of the uyuni tools to the right paths as soon as they exist (as of today, it can be tested with this one: https://download.opensuse.org/repositories/systemsmanagement:/saltstack:/bundle:/next:/openEuler2203/openEuler_22.03/ ). Special thanks to @vzhestkov for helping close this issue.