OrangeDog
commented
1 year ago Have you changed file_roots from the default?
Open DaAwesomeP opened 1 year ago
OrangeDog
commented
1 year ago Have you changed file_roots from the default?
DaAwesomeP
commented
1 year ago @OrangeDog No, I have not directly touched that exact directive. Though I am using a different fileserver backend. I suppose that could be it! Sorry I did not remember that initially!
I have verified that /srv/salt/win/repo-ng/salt-winrepo-ng exists and has files and folders in it.
On the master the only other config is:
fileserver_backend:
- gitfs
gitfs_provider: pygit2
gitfs_remotes:
- ssh://git@github.com/REDACTED/REDACTED.git:
- base: main
- root: salt
- pubkey: REDACTED.pub
- privkey: REDACTED
ext_pillar:
- git:
- main ssh://git@github.com/REDACTED/REDACTED.git:
- env: base
- root: pillar
- pubkey: REDACTED.pub
- privkey: REDACTED@OrangeDog should I try something like:
fileserver_backend:
- gitfs
- rootsI do want Git to be first priority.
DaAwesomeP
commented
1 year ago OK, that was definitely a big part of the problem! Will leave this issue open until I submit a pull to add a note to docs about that.
But now I'm seeing this:
----------
ID: common_pkg_win_refresh
Function: module.run
Name: pkg.refresh_db
Result: True
Comment: Module function pkg.refresh_db executed
Started: 09:02:11.017479
Duration: 225532.735 ms
Changes:
----------
ret:
----------
failed:
0
success:
309
total:
309
----------
ID: common_pkg
Function: pkg.installed
Result: False
Comment: The following packages failed to install/update: 7zip
Failed to cache https://d.7-zip.org/a/7z2201-x64.msi
Error: [Errno 11001] getaddrinfo failed reading /a/7z2201-x64.msi
The following packages were already installed: firefox-esr_x64
Started: 09:05:58.980199
Duration: 3418.045 ms
Changes:Firefox ESR was installed separately on this machine before running Salt, so that part is correct.
DaAwesomeP
commented
1 year ago Hmm, OK I ran it again and it completed 100% fine. That sounds like maybe #58792.
DaAwesomeP
commented
1 year ago Another thing is just that pkg.refresh_db takes almost 4 minutes to complete. That's a lot.
OrangeDog
commented
1 year ago Yes, it syncs the git repo to a file root and then serves it from there.
It would probably also work over gitfs, and then you won't need any of the master-side commands. It seems like that should be the default option, but the docs never even mention the possibility.
DaAwesomeP
commented
1 year ago Yes, it syncs the git repo to a file root and then serves it from there.
Yes, I see.
It would probably also work over gitfs, and then you won't need any of the master-side commands.
How would I configure that?
It seems like that should be the default option, but the docs never even mention the possibility.
I will definitely add that to the docs if it works!
OrangeDog
commented
1 year ago Like this maybe?
gitfs_remotes:
- https://github.com/saltstack/salt-winrepo-ng.git:
- all_saltenvs: master
- mountpoint: salt://win/repo-ng
# ...
barbaricyawps
commented
1 year ago Since this is a bug, can we open a separate documentation issue for the docs request portion of this bug? That will make it easier for me to triage and track.
I'm removing the documentation label for now.
DaAwesomeP
commented
1 year ago OK, so I resolved the issue with either of the following solutions:
Quick and easy, works for all installs (but not ideal for updating):
fileserver_backend:
- gitfs
- roots # for winrepo
gitfs_provider: pygit2
gitfs_remotes:
- ssh://git@github.com/REDACTED/REDACTED.git:
- base: main
- root: salt
- pubkey: REDACTED.pub
- privkey: REDACTED
ext_pillar:
- git:
- main ssh://git@github.com/REDACTED/REDACTED.git:
- env: base
- root: pillar
- pubkey: REDACTED.pub
- privkey: REDACTEDWhat I ended up doing instead to avoid needing adding roots (@OrangeDog's mountpoint path was nearly correct):
fileserver_backend:
- gitfs
gitfs_provider: pygit2
gitfs_remotes:
- ssh://git@github.com/REDACTED/REDACTED.git:
- base: main
- root: salt
- pubkey: REDACTED.pub
- privkey: REDACTED
- https://github.com/saltstack/salt-winrepo-ng.git:
- all_saltenvs: master
- mountpoint: salt://win/repo-ng/salt-winrepo-ng
ext_pillar:
- git:
- main ssh://git@github.com/REDACTED/REDACTED.git:
- env: base
- root: pillar
- pubkey: REDACTED.pub
- privkey: REDACTEDThe latter method seems smoothest because the repo is updated whenever fileserver updates.
I was also able to very easily place my own package definitions in win/repo-ng/custom_defs, and that was very convenient and painless.
@barbaricyawps The above solution is a documentation issue, but I think this could be smoother out-of-the-box. I thought about maybe making the gitfs method default, but that doesn't work for every case since pygit2 may not be installed. I think maybe having some sort of warning print out if pkg.refresh_db returns 0 results would be good. It could link to a specific section of updated documentation.
OrangeDog
commented
1 year ago @barbaricyawps it's not a bug. The documentation just assumes you only have file roots.
barbaricyawps
commented
1 year ago Then we should label it as a docs issue rather than a bug in the title. Can someone summarize the action needed here?
DaAwesomeP
commented
1 year ago I think there are a few things that tripped me up in the docs when working on this. I should preface that I have been using Saltstack on Debian and RHEL machines extensively and this is the first time I have ventured into Windows with Saltstack.
Everything here is intended to be constructive and helpful. I will admit I became a bit frustrated while reading these docs. It was really @OrangeDog's comment that gave me the aha moment with how everything works. I didn't get that initially from the docs.
gitfs). More on this below. It was not clear how this all works. It took me a while to realize that all pkg.refresh_db cared about was finding the repos in salt://win/repo-ng.salt-run winrepo.update_git_repos on the Master pulls the winrepo_remotes_ng repos into /srv/salt/win/repo-ng (Linux default) or the location defined by winrepo_dir_ng. 2) Alternatively, repos can be defined in any fileserver method (such as gitfs) to provide directories in salt://win/repo-ng and will be updated with that fileserver's method (and link to an example). 3) Running pkg.refresh_db on a Minion pulls from salt://win/repo-ng and updates the package database on a Minion.I'm sure there was ample discussion and decisions made to use a separate Git system for the Windows package system, but it is very confusing. It is a redundant system considering that gitfs already provides Git mounts for salt://. I would guess the main reason was to have the default repo always work out-of-the-box, but I think this could also be solved by displaying a warning simply if salt://win/repo-ng does not exist or is empty when running pkg.refresh_db. This would have directly prevented me from opening this issue. It does report that it found 0 packages, but it doesn't say that the whole directory was missing. This warning should link to a section in the docs.
There should also maybe be a warning if you run winrepo.update_git_repos but roots is not configured. It should be very detectable if winrepo.update_git_repos pulls but nothing is configured to make use of the location it pulled to. This warning should also link to a section in the docs.
The last two are not blocking bugs per say but are usability improvements to the application. Everything above that are docs suggestions.
I am happy to turn one or more of these suggestions into pulls!
Description Running
pkg.refresh_dbon Windows does not show any repositories. It errors because it did not update any repos.Setup
Master: OVH VPS running Debian 11 Windows Minion: On-premises Windows 10 Enterprise IoT LTSC 2019
Steps to Reproduce the behavior
salt-run -l debug winrepo.update_git_reposApply states:
salt 'REDACTED*' state.apply -l debugExpected behavior Database should refresh on the minion and the packages should install.
Screenshots All console output is above in text form.
Versions Report
Master:
```yaml Salt Version: Salt: 3005.1 Dependency Versions: cffi: 1.14.6 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.0 libgit2: 1.1.0 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.9.8 pygit2: 1.6.1 Python: 3.9.16 (main, Jan 6 2023, 22:49:58) python-gnupg: 0.4.8 PyYAML: 5.4.1 PyZMQ: 23.2.0 smmap: Not Installed timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: debian 11 bullseye locale: utf-8 machine: x86_64 release: 5.10.0-20-cloud-amd64 system: Linux version: Debian GNU/Linux 11 bullseye ```salt --versions-reportMinion:
```yaml REDACTED: Salt Version: Salt: 3005.1 Dependency Versions: cffi: 1.14.6 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: 4.0.7 gitpython: 3.1.18 Jinja2: 3.1.0 libgit2: Not Installed M2Crypto: Not Installed Mako: 1.1.4 msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.10.1 pygit2: Not Installed Python: 3.8.15 (tags/v3.8.15:44adf8a, Nov 2 2022, 20:58:02) [MSC v.1916 64 bit (AMD64)] python-gnupg: 0.4.8 PyYAML: 5.4.1 PyZMQ: 22.0.3 smmap: 4.0.0 timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: locale: cp1252 machine: AMD64 release: 2021 system: Windows version: 2021 10.0.19044 SP0 Multiprocessor Free ```salt "REDACTED*" cmd.run 'salt --versions-report'Additional context The documentation is kind of confusing. It is not immediately obvious that the repo is stored on the master and sent to the minion, and it seems that placing
refresh: Trueinpkg.installeddoes not work (or maybe it doesn't work due to this issue). Instead, you must run thepkg.refresh_dbmodule each time, which is not very clean.