saltclass error when expanding pillar value that is not of type 'str'

ghost commented 2 years ago

Description of Issue

When using saltclass for pillar data, variable expansion/interpolation fails when referenced pillar value is not a string. This does not appear to be an issue when simply referencing non-string pillar data within states, only when doing interpolation within saltclass itself.

Using the example below, referencing defaults:default_service_port in a state works fine. Interpolating as the value for myservice:port fails with the following exception shown in the master log: [salt.pillar :1218][CRITICAL][22870] Pillar render error: Failed to load ext_pillar saltclass: replace() argument 2 must be str, not int

This appears to be coming from salt.utils.saltclass in find_and_process_re() and/or find_value_to_expand(). The call to str.replace() in find_and_process_re() fails because find_value_to_expand() returns expanded pillar data in its original type, and str.replace() needs its second argument to be a string, as the log entry states.

Setup

Example pillar data

pillar:
  defaults:
    default_service_port: 1234
  myservice:
    port: ${defaults:default_service_port}

Steps to Reproduce Issue

Call salt <target> pillar.get defaults:default_service_port. This should return 1234 correctly.
Call salt <target> pillar.get myservice:port, salt <target> pillar.items, or anything else that would make the interpolation attempt occur. The only log entry Salt provides that I have seen is [salt.pillar :1218][CRITICAL][22870] Pillar render error: Failed to load ext_pillar saltclass: replace() argument 2 must be str, not int, but more details can be seen with pdb.

Versions Report

Salt Version:
          Salt: 3003.4

Dependency Versions:
          cffi: 1.14.0
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.1
       libgit2: 0.28.4
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: 2.6.1
  pycryptodome: 3.13.0
        pygit2: 1.0.3
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 5.3
         PyZMQ: 17.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4

Salt Extensions:
        sseape: 8.6.0.3

System Versions:
          dist: centos 7 Core
        locale: UTF-8
       machine: x86_64
       release: 3.10.0-1160.59.1.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core

welcome[bot] commented 2 years ago

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

ketzacoatl commented 5 months ago

I see this behavior in saltclass setup with salt-minion v3006.8.

ketzacoatl commented 5 months ago

Actually, to be clear, I see a similar but slightly different problem. The net result is the same, but the failure seems to be happening differently now. Instead of getting an error about the str, the values simply (silently) fail to render for the interpolation. Eg, interpolation references don't get resolved.

for a class with:

pillar:
  foo: bar
  baz: ${foo}/baq

the baz pillar comes out at ${foo}/baq instead of bar/baq.

ketzacoatl commented 4 months ago

AFAICT, this bug breaks interpolation in pillar for saltclass.

Given that sensible and powerful interpolation is the primary feature and purpose of reclass/saltclass, for someone who actually relies on saltclass/reclass in their setup, this bug effectively breaks all use of saltstack and makes saltstack completely useless.

My guess is that there are not many people in this position, so there are only a few that are impacted.

ATM I have to weigh out my options, do I wait for this to be resolved? do I refactor all of my use of pillar and saltclass? do I refactor my whole AMI build to not use saltstack? I really don't like any of these options.

@whytewolf, suggestions?

whytewolf commented 4 months ago

so, a couple of things.

I believe saltclass is being removed as part of the module migration in 3008. so unless a user picks it up as a salt-ext it is going away.
I no longer work for the salt project so most likely will not be working on much code for it now or in the future unless i take on a new role somewhere that needs salt.

given those two things it is up to you about how you feel going forward.

ketzacoatl commented 4 months ago

@whytewolf ok, super thanks for that info! Especially the saltclass going away part, I don't follow saltstack closely enough to know those things. Thanks for all that you've done with the project, and goodluck to your future endeavors.

saltstack / salt