[BUG] win_pkg fails to find any packages if cachedir is relocated to "hidden" folder

[yangskyboxlabs]

Description In a masterless minion, if cachedir is configured to be part of a "hidden" folder (begins with .), win_pkg will fail to find any packages.

Setup (Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

• [x] on-prem machine
• [ ] VM (Virtualbox, KVM, etc. please specify)
• [ ] VM running on a cloud service, please be explicit and add details
• [ ] container (Kubernetes, Docker, containerd, etc. please specify)
• [ ] or a combination, please be explicit
• [ ] jails if it is FreeBSD
• [ ] classic packaging
• [ ] onedir packaging
• [ ] used bootstrap to install

Steps to Reproduce the behavior (My winrepo-ng is a fork that contains far fewer packages, but is confirmed to be working in other contexts)

# minion conf

file_client: local

root_dir: .
cachedir: D:/infra/salt-config-skyboxlabs-pitch/.cache

winrepo_remotes: []
winrepo_remotes_ng: []

PS D:\infra\salt-config-skyboxlabs-pitch> salt-call -c .local -l debug pkg.refresh_db
[DEBUG   ] Reading configuration from D:\infra\salt-config-skyboxlabs-pitch\.local\minion
[DEBUG   ] Including configuration from 'D:\infra\salt-config-skyboxlabs-pitch\.local\minion.d\grain_test.conf'
[DEBUG   ] Reading configuration from D:\infra\salt-config-skyboxlabs-pitch\.local\minion.d\grain_test.conf
[DEBUG   ] Including configuration from 'D:\infra\salt-config-skyboxlabs-pitch\.local\minion.d\roots.conf'
[DEBUG   ] Reading configuration from D:\infra\salt-config-skyboxlabs-pitch\.local\minion.d\roots.conf
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Configuration file path: D:\infra\salt-config-skyboxlabs-pitch\.local\minion
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from D:\infra\salt-config-skyboxlabs-pitch\.local\minion
[DEBUG   ] Including configuration from 'D:\infra\salt-config-skyboxlabs-pitch\.local\minion.d\grain_test.conf'
[DEBUG   ] Reading configuration from D:\infra\salt-config-skyboxlabs-pitch\.local\minion.d\grain_test.conf
[DEBUG   ] Including configuration from 'D:\infra\salt-config-skyboxlabs-pitch\.local\minion.d\roots.conf'
[DEBUG   ] Reading configuration from D:\infra\salt-config-skyboxlabs-pitch\.local\minion.d\roots.conf
[DEBUG   ] The functions from module 'core' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'disks' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'extra' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'lvm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'mdadm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'minion_process' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'opts' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'package' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'pending_reboot' are being loaded by dir() on the loaded module
[DEBUG   ] Override  __utils__: <module 'salt.loaded.int.grains.zfs' from 'C:\\Program Files\\Salt Project\\Salt\\Lib\\site-packages\\salt\\grains\\zfs.py'>
[DEBUG   ] Determining pillar cache
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Popen(['git', 'version'], cwd=D:\infra\salt-config-skyboxlabs-pitch, stdin=None, shell=False, universal_newlines=False)
[DEBUG   ] Popen(['git', 'version'], cwd=D:\infra\salt-config-skyboxlabs-pitch, stdin=None, shell=False, universal_newlines=False)
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] Updating roots fileserver cache
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] compile template: .local/pillar\top.sls
[DEBUG   ] Jinja search path: ['.local/pillar', 'pillar']
[PROFILE ] Time (in seconds) to render '.local/pillar\top.sls' using 'jinja' renderer: 0.0010001659393310547
[PROFILE ] Time (in seconds) to render '.local/pillar\top.sls' using 'yaml' renderer: 0.0
[DEBUG   ] The functions from module 'confirm_top' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded confirm_top.confirm_top
[DEBUG   ] The functions from module 'compound_match' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded compound_match.match
[DEBUG   ] compound_match: __local__ ? *
[DEBUG   ] The functions from module 'glob_match' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded glob_match.match
[DEBUG   ] compound_match __local__ ? "*" => "True"
[DEBUG   ] compile template: pillar\fastbuild/worker.sls
[DEBUG   ] Jinja search path: ['.local/pillar', 'pillar']
[PROFILE ] Time (in seconds) to render 'pillar\fastbuild/worker.sls' using 'jinja' renderer: 0.0010001659393310547
[PROFILE ] Time (in seconds) to render 'pillar\fastbuild/worker.sls' using 'yaml' renderer: 0.0
[DEBUG   ] compile template: pillar\teamcity/agent.sls
[DEBUG   ] Jinja search path: ['.local/pillar', 'pillar']
[PROFILE ] Time (in seconds) to render 'pillar\teamcity/agent.sls' using 'jinja' renderer: 0.0019996166229248047
[PROFILE ] Time (in seconds) to render 'pillar\teamcity/agent.sls' using 'yaml' renderer: 0.0010001659393310547
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] Failed to import module kernelpkg_linux_yum:
Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\loader\lazy.py", line 806, in _load_module
    spec.loader.exec_module(mod)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\modules\kernelpkg_linux_yum.py", line 8, in <module>
    import salt.modules.yumpkg
ModuleNotFoundError: No module named 'salt.modules.yumpkg'
[DEBUG   ] The functions from module 'pkg_resource' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'pkg' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded pkg.refresh_db
[DEBUG   ] The functions from module 'direct_call' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded direct_call.execute
[DEBUG   ] Refreshing pkg metadata db for saltenv 'base' (age of existing metadata is -1 day, 23:59:59)
[INFO    ] Removing all *.sls files under 'D:/infra/salt-config-skyboxlabs-pitch/.cache\files\base\win\repo-ng'
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Removed base file list cache for saltenv 'roots'
[INFO    ] Fetching *.sls files from salt://win/repo-ng/
[DEBUG   ] The functions from module 'cp' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded cp.cache_dir
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] Updating roots fileserver cache
[INFO    ] Caching directory 'win/repo-ng/' for environment 'base'
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] In saltenv 'base', looking at rel_path 'win/repo-ng/salt-winrepo-skyboxlabs/java/amazon-corretto.sls' to resolve 'salt://win/repo-ng/salt-winrepo-skyboxlabs/java/amazon-corretto.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path 'D:\infra\salt-config-skyboxlabs-pitch\.cache\files\base\win\repo-ng\salt-winrepo-skyboxlabs\java\amazon-corretto.sls' to resolve 'salt://win/repo-ng/salt-winrepo-skyboxlabs/java/amazon-corretto.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://win/repo-ng/salt-winrepo-skyboxlabs/java/amazon-corretto.sls'
[DEBUG   ] No dest file found
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/salt-winrepo-skyboxlabs/java/amazon-corretto.sls'
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] In saltenv 'base', looking at rel_path 'win/repo-ng/salt-winrepo-skyboxlabs/prometheus/windows_exporter.sls' to resolve 'salt://win/repo-ng/salt-winrepo-skyboxlabs/prometheus/windows_exporter.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path 'D:\infra\salt-config-skyboxlabs-pitch\.cache\files\base\win\repo-ng\salt-winrepo-skyboxlabs\prometheus\windows_exporter.sls' to resolve 'salt://win/repo-ng/salt-winrepo-skyboxlabs/prometheus/windows_exporter.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://win/repo-ng/salt-winrepo-skyboxlabs/prometheus/windows_exporter.sls'
[DEBUG   ] No dest file found
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/salt-winrepo-skyboxlabs/prometheus/windows_exporter.sls'
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] In saltenv 'base', looking at rel_path 'win/repo-ng/salt-winrepo-skyboxlabs/salt-minion.sls' to resolve 'salt://win/repo-ng/salt-winrepo-skyboxlabs/salt-minion.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path 'D:\infra\salt-config-skyboxlabs-pitch\.cache\files\base\win\repo-ng\salt-winrepo-skyboxlabs\salt-minion.sls' to resolve 'salt://win/repo-ng/salt-winrepo-skyboxlabs/salt-minion.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://win/repo-ng/salt-winrepo-skyboxlabs/salt-minion.sls'
[DEBUG   ] No dest file found
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/salt-winrepo-skyboxlabs/salt-minion.sls'
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] In saltenv 'base', looking at rel_path 'win/repo-ng/salt-winrepo-skyboxlabs/visualstudio/visualstudio-installer.sls' to resolve 'salt://win/repo-ng/salt-winrepo-skyboxlabs/visualstudio/visualstudio-installer.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path 'D:\infra\salt-config-skyboxlabs-pitch\.cache\files\base\win\repo-ng\salt-winrepo-skyboxlabs\visualstudio\visualstudio-installer.sls' to resolve 'salt://win/repo-ng/salt-winrepo-skyboxlabs/visualstudio/visualstudio-installer.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://win/repo-ng/salt-winrepo-skyboxlabs/visualstudio/visualstudio-installer.sls'
[DEBUG   ] No dest file found
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/salt-winrepo-skyboxlabs/visualstudio/visualstudio-installer.sls'
[DEBUG   ] Skipping files in directory: D:/infra/salt-config-skyboxlabs-pitch/.cache\files\base\win\repo-ng
[DEBUG   ] Skipping files in directory: D:/infra/salt-config-skyboxlabs-pitch/.cache\files\base\win\repo-ng\salt-winrepo-skyboxlabs
[DEBUG   ] Skipping files in directory: D:/infra/salt-config-skyboxlabs-pitch/.cache\files\base\win\repo-ng\salt-winrepo-skyboxlabs\java
[DEBUG   ] Skipping files in directory: D:/infra/salt-config-skyboxlabs-pitch/.cache\files\base\win\repo-ng\salt-winrepo-skyboxlabs\prometheus
[DEBUG   ] Skipping files in directory: D:/infra/salt-config-skyboxlabs-pitch/.cache\files\base\win\repo-ng\salt-winrepo-skyboxlabs\visualstudio
[DEBUG   ] The functions from module 'nested' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded nested.output
local:
    ----------
    failed:
        0
    success:
        0
    total:
        0

Expected behavior win_pkg should succeed in finding packages.

Versions Report

salt --versions-report

```yaml Salt Version: Salt: 3007.1 Python Version: Python: 3.10.14 (heads/main:c1ec015, Apr 3 2024, 21:36:37) [MSC v.1938 64 bit (AMD64)] Dependency Versions: cffi: 1.16.0 cherrypy: 18.8.0 dateutil: 2.8.2 docker-py: Not Installed gitdb: 4.0.10 gitpython: 3.1.43 Jinja2: 3.1.4 libgit2: Not Installed looseversion: 1.3.0 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.7 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 23.1 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.19.1 pygit2: Not Installed python-gnupg: 0.5.2 PyYAML: 6.0.1 PyZMQ: 25.1.2 relenv: 0.16.0 smmap: 5.0.1 timelib: 0.3.0 Tornado: 6.3.3 ZMQ: 4.3.4 Salt Package Information: Package Type: onedir System Versions: dist: locale: utf-8 machine: AMD64 release: 10 system: Windows version: 10 10.0.19045 SP0 Multiprocessor Free ```

Additional context The same setup with a different cachedir works as expected:

PS D:\infra\salt-config-skyboxlabs-pitch> salt-call -c .local -l info pkg.refresh_db
[INFO    ] Removing all *.sls files under 'D:/infra/salt-config-skyboxlabs-pitch/_cache\files\base\win\repo-ng'
[INFO    ] Fetching *.sls files from salt://win/repo-ng/
[INFO    ] Caching directory 'win/repo-ng/' for environment 'base'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/salt-winrepo-skyboxlabs/java/amazon-corretto.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/salt-winrepo-skyboxlabs/prometheus/windows_exporter.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/salt-winrepo-skyboxlabs/salt-minion.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/salt-winrepo-skyboxlabs/visualstudio/visualstudio-installer.sls'
local:
    ----------
    failed:
        0
    success:
        4
    total:
        4

This is due to an overly aggressive filter in win_pkg.genrepo().

        # Skip hidden directories (.git)
        if re.search(r"[\\/]\..*", root):
            log.debug("Skipping files in directory: %s", root)
            continue

An alternative implementation is to filter such directories as part of walk():

    for root, dirs, files in salt.utils.path.os_walk(
        repo_details.local_dest, followlinks=False
    ):

        # Skip hidden directories (.git/, et al)
        # (this is concise, but loses logging information)
        dirs[:] = [d for d in dirs if not d.startswith(".")]

[welcome[bot]]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!