search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.16k stars 5.48k forks source link

[2016.3.0] win_pkg: pkg.list_upgrades loops (almost) endlessly - cmds take VERY long #33620

Closed TheBigBear closed 8 years ago

TheBigBear commented 8 years ago

Description of Issue/Question

winrepo-ng running of pkg.list_upgrades takes between 5 - 10 minutes to get a result back

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).) using 2016.3.0 on master and minion and running salt winminion pkg.list_upgrades always times out. But if run on the windows minion itself as salt-call pkg.list_upgrades it does show output but takes in excess of 5 minutes to finish.

The output shown looks like an (almost) endless repeat of a pkg.refresh_db loop, it looks as if it is running a whole pkg.refresh_db run once for each single pkg it is trying to see if a pkg has an upgrade.

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Ch3LL commented 8 years ago

@TheBigBear I'm having a hard time replicating this on 2016.3.0. How many packages on your windows minion need updates? I just have a simple test case with one package that needs an update and i'm not seeing te command taking very long.

TheBigBear commented 8 years ago

@Ch3LL on a VM that needs 14 updates it takes in excess of 8 minutes. and on another that needs 11 updates it takes 6 minutes.

uk-it-20
    local:
    ----------
    adobeair:
        latest
    check-mk-agent-msi:
        1.2.8b4
    chocolatey:
        0.9.9
    chrome:
        latest
    eea:
        6.3.2016.0
    fiddler:
        4.6.2.2
    firefox:
        46.0.1
    ms-mbsa:
        2.3.2211
    nsis:
        3.0b2
    pdf24creator:
        7.6.4
    python2_x64:
        2.7.11150
    sourcetree:
        1.6.21
    ultradefrag:
        7.0.0

uk-it-70

local:
    ----------
    adobeair:
        latest
    check-mk-agent-msi:
        1.2.8b4
    chrome:
        latest
    eea:
        6.3.2016.0
    firefox:
        46.0.1
    grepwin:
        1.6.661
    gvim:
        7.4
    pdf24creator:
        7.6.4
    python2_x86:
        2.7.11150
    quicktime:
        7.79.80.95
    salt-minion:
        2016.3.0
TheBigBear commented 8 years ago

@Ch3LL I am in the process of logging the output to a file, so I can count how many times it iterates through the loops. But when done from the linux master it times out. and I/O redirection under windows does not seem to let me catch stdout and stederr, will keep trying.

TheBigBear commented 8 years ago

@Ch3LL OK, so for the uk-it-20 machine that needs 13 upgrades it runs through the loop 53 times. It loops from line 1 - 230 and then repeats itself for another 52 times after that.

the command I issue on the local windows minion is:

salt-call pkg.list_upgrades

    1 [INFO    ] Caching directory u'win/repo-ng/' for environment 'base'
    2 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/baregrep/init.sls'
    3 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/baretail/init.sls'
    4 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/bginfo/init.sls'
    5 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/never10/init.sls'
    6 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/salt-win-tools/init.sls'
    7 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/7zip.sls'
    8 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/activeperl_x64.sls'
    9 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/activeperl_x86.sls'
   10 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/adobeair.sls'
   11 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/adobeflashplayeractivex.sls'
   12 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/adobeflashplayerplugin.sls'
   13 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/adobereader-dc-classic.sls'
   14 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/adobereader-xi.sls'
   15 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/adobereader.sls'
   16 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/adobeshockwaveplayer.sls'
   17 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/adv-ip-scanner.sls'
   18 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/adv-port-scanner.sls'
   19 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/advancedlogging.sls'
   20 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/applicationrequestrouting.sls'
   21 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/aspnet-mvc1.sls'
   22 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/atom.sls'
   23 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/audacity.sls'
   24 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/autohotkey.sls'
   25 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/autoit.sls'
   26 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/autopsy.sls'
   27 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/awscli.sls'
   28 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/bandizip.sls'
   29 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/bitnami-nginxstack.sls'
   30 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/blender.sls'
   31 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/bulk_extractor.sls'
   32 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ccleaner.sls'
   33 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/cdburnerxp.sls'
   34 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/cdex.sls'
   35 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/cdroller.sls'
   36 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/check-mk-agent-msi.sls'
   37 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/check-mk-agent.sls'
   38 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/chocolatey.sls'
   39 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/chrome.sls'
   40 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/clamav.sls'
   41 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/clamwin.sls'
   42 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/classicshell.sls'
   43 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/clink.sls'
   44 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/conemu-alpha.sls'
   45 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/conemu-preview.sls'
   46 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/cpu-z.sls'
   47 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/curl.sls'
   48 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/cyberduck-cli.sls'
   49 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/cyberduck.sls'
   50 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/defraggler.sls'
   51 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/dotnet.sls'
   52 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/duplicati.sls'
   53 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/dvdstyler.sls'
   54 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/eea.sls'
   55 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/emet.sls'
   56 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/emsisoft-anti-malware.sls'
   57 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/fiddler.sls'
   58 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/filehippo-app-manager.sls'
   59 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/filezilla.sls'
   60 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/firefox-esr.sls'
   61 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/firefox.sls'
   62 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/gedit.sls'
   63 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/gimp.sls'
   64 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/git-extensions.sls'
   65 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/git.sls'
   66 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/glarysoft-absolute-uninstaller.sls'
   67 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/gnucash.sls'
   68 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/golang.sls'
   69 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/goodsync.sls'
   70 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/gow.sls'
   71 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/gpg4win-light.sls'
   72 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/gpg4win-vanilla.sls'
   73 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/gpg4win.sls'
   74 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/gpu-z.sls'
   75 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/grepwin.sls'
   76 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/gvim.sls'
   77 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/handbrake.sls'
   78 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ice.sls'
   79 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/icecast.sls'
   80 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/icloud.sls'
   81 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/iismediaservices.sls'
   82 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/influx-capacitor.sls'
   83 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/infrarecorder.sls'
   84 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/inkscape.sls'
   85 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/intellij.sls'
   86 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/internet-evidence-finder.sls'
   87 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/irfanview-plugins.sls'
   88 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/irfanview.sls'
   89 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/isapirewrite-lite.sls'
   90 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/itunes.sls'
   91 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/jre.sls'
   92 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/jre8.sls'
   93 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/kdiff3.sls'
   94 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/keepass-2x.sls'
   95 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/keepass.sls'
   96 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/libreoffice.sls'
   97 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/logparser.sls'
   98 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/malwarebytes.sls'
   99 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/mercurial.sls'
  100 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/messageanalyzer.sls'
  101 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/microsoft-build-tools.sls'
  102 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/mikogo.sls'
  103 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/miktex-basic.sls'
  104 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/mongodb.sls'
  105 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-mbsa.sls'
  106 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2005-atl-redist_x64.sls'
  107 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2005-atl-redist_x86.sls'
  108 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2005-redist_x64.sls'
  109 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2005-redist_x86.sls'
  110 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2005-sp1-mfc-redist_x64.sls'
  111 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2005-sp1-mfc-redist_x86.sls'
  112 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2005-sp1-redist_x64.sls'
  113 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2005-sp1-redist_x86.sls'
  114 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2008-redist_x64.sls'
  115 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2008-redist_x86.sls'
  116 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2008-sp1-atl-redist_x64.sls'
  117 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2008-sp1-atl-redist_x86.sls'
  118 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2008-sp1-mfc-redist_x64.sls'
  119 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2008-sp1-mfc-redist_x86.sls'
  120 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2010-sp1-mfc-redist_64.sls'
  121 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2010-sp1-mfc-redist_x86.sls'
  122 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2012-redist_x64.sls'
  123 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2012-redist_x86.sls'
  124 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2013-redist_x64.sls'
  125 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2013-redist_x86.sls'
  126 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2015-redist_x64.sls'
  127 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ms-vcpp-2015-redist_x86.sls'
  128 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/mysql-essential.sls'
  129 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/mysql-installer-community.sls'
  130 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/nmap.sls'
  131 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/npp.sls'
  132 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/nsclient.sls'
  133 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/nsis.sls'
  134 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/nxlog.sls'
  135 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/openlp.sls'
  136 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/openoffice.sls'
  137 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/openvpn.sls'
  138 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ossec-agent.sls'
  139 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/owncloud.sls'
  140 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/pandoc.sls'
  141 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/passware-kit-agent.sls'
  142 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/passware-kit-forensic.sls'
  143 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/patchmypc-free.sls'
  144 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/pdf24creator.sls'
  145 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/pdfcreator.sls'
  146 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/peazip.sls'
  147 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/pgina.sls'
  148 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/pidgin.sls'
  149 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/postgresql.sls'
  150 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/putty.sls'
  151 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/python2_x64.sls'
  152 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/python2_x86.sls'
  153 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/python3_x64.sls'
  154 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/python3_x86.sls'
  155 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/qemu.sls'
  156 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/quicktime.sls'
  157 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/rakudo-star-no-jit_x86.sls'
  158 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/rdcman.sls'
  159 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/rubyinstaller_x64.sls'
  160 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/rubyinstaller_x86.sls'
  161 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/salt-minion.sls'
  162 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/sandboxie.sls'
  163 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/secunia.psi.sls'
  164 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/sensu.sls'
  165 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/sharpdevelop.sls'
  166 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/silverlight.sls'
  167 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/skitch.sls'
  168 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/skype-msi.sls'
  169 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/slack.sls'
  170 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/smartmontools.sls'
  171 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/snmptools.sls'
  172 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/software-informer.sls'
  173 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/sourcetree.sls'
  174 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/spybot.sls'
  175 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/sscserv-free.sls'
  176 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/stellarium.sls'
  177 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/strawberryperl_x64.sls'
  178 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/strawberryperl_x86.sls'
  179 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/stunnel.sls'
  180 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/subinacl.sls'
  181 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/sums.sls'
  182 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/svn.sls'
  183 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/teamviewer.sls'
  184 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/teracopy.sls'
  185 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/texmaker.sls'
  186 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/texniccenter.sls'
  187 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/texstudio.sls'
  188 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/texworks.sls'
  189 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/thunderbird.sls'
  190 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/tightvnc.sls'
  191 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/tortoise-bzr.sls'
  192 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/tortoise-git.sls'
  193 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/tortoise-hg.sls'
  194 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/tortoise-svn.sls'
  195 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/truecrypt.sls'
  196 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ultradefrag.sls'
  197 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/ultravnc.sls'
  198 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/urlrewrite.sls'
  199 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/vagrant.sls'
  200 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/vcforpython27.sls'
  201 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/vcredist2010_x64.sls'
  202 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/vcredist2010_x86.sls'
  203 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/virtualbox.sls'
  204 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/vlc.sls'
  205 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/vsee.sls'
  206 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/wamp-server-3.sls'
  207 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/wamp-server.sls'
  208 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/wamp-stack.sls'
  209 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/webdeploy.sls'
  210 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/webplatforminstaller.sls'
  211 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/win-app-manager.sls'
  212 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/windirstat.sls'
  213 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/winhttpcertcfg.sls'
  214 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/winmerge.sls'
  215 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/winpcap.sls'
  216 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/winscp.sls'
  217 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/wireshark-devel.sls'
  218 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/wireshark.sls'
  219 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/wmi_provider/init.sls'
  220 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/wscc.sls'
  221 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/xampp.sls'
  222 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/xming.sls'
  223 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/zipinstaller/init.sls'
  224 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/sums/init.sls'
  225 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/sysinternalssuite/init.sls'
  226 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/tarma-uninstaller/init.sls'
  227 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/test-pwd/init.sls'
  228 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/zipinstaller/init.sls'
  229 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/zipinstaller/new_init.sls'
  230 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/zipinstaller/old_init.sls'

and then repeats for another 52 times. 

  231 [INFO    ] Caching directory u'win/repo-ng/' for environment 'base'
  232 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/baregrep/init.sls'
  233 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/baretail/init.sls'
  234 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/bginfo/init.sls'
  235 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/never10/init.sls'
  236 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/salt-win-tools/init.sls'
  237 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/7zip.sls'
  238 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/activeperl_x64.sls'
  239 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/activeperl_x86.sls'
  240 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/adobeair.sls'

then cut some and it ends in:

12180 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/wscc.sls'
12181 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/xampp.sls'
12182 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/xming.sls'
12183 [INFO    ] Fetching file from saltenv 'base', ** done ** u'win/repo-ng/salt-winrepo-ng/zipinstaller/init.sls'
12184 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/sums/init.sls'
12185 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/sysinternalssuite/init.sls'
12186 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/tarma-uninstaller/init.sls'
12187 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/test-pwd/init.sls'
12188 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/zipinstaller/init.sls'
12189 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/zipinstaller/new_init.sls'
12190 [INFO    ] Fetching file from saltenv 'base', ** skipped ** latest already in cache u'salt://win/repo-ng/zipinstaller/old_init.sls'

So it runs through this 53 times in total. Why? This takes a very long time.

damon-atkins commented 8 years ago

I have seen this, as well. I think the backend code the win pkg calls does a DB refresh, I suspect expect it does it for every package it evaluates. i.e. 3 packages evaluated 3 fetches of all the definitions from the master. The only work around is salt-call --local from the client.

TheBigBear commented 8 years ago

@Ch3LL did my instructions not help? can you really still not replicate this?

TheBigBear commented 8 years ago

@damon-atkins

I have seen this, as well. I think the backend code the win pkg calls does a DB refresh, I suspect expect it does it for every package it evaluates. i.e. 3 packages evaluated 3 fetches of all the definitions from the master. The only work around is salt-call --local from the client.

Can you explain how you work around this? If I do a cmd.run salt-call --local pkg.list_upgrades using your arguments from the master, or literally as-above from a windows minion salt-call --local pkg.list_upgrades I simply get an empty result in both cases?

damon-atkins commented 8 years ago

On windows itself --local stops it from fetching 12189 [INFO ] Fetching file from saltenv 'base', * skipped * latest already in cache u'salt://win/repo-ng/...... The DB & files needs to exist first locally pkg.genrepo you can test with pkg.get_repo_data

ghost commented 8 years ago

Hi people, I've hopefully dealt with a similar issue with #34768 (just finalized a few minutes ago and hopefully will be merged soonish). Note that one of the problems was because of the following fragment in pkg.latest_version

if salt.utils.is_true(kwargs.get('refresh', True)):
        refresh_db(saltenv)

a call to pkg.latest version (without refresh being passed into kwargs) will always result in a db refresh. I haven't personally looked into this issue yet, but I wonder if something like this is the culprit. If this is the case, then when pkg.latest_version is called, you (or whoever fixes this) will have to make sure that 'refresh' is given a value of False.

ghost commented 8 years ago

Hi, I just looked at the code, and I think the problem is as I described. Will issue a PR sooner or later to fix this (if no else does it first).

TheBigBear commented 8 years ago

@hrumph thank you so much. I agree it definitely looks as if refresh_db is getting called unnecessarily every single time. I had also looked deeper and found that even during a '--local' run from the windows minion as suggested by @damon-atkins it still ran a refresh_db 60 + times, on a machine with just a few pkgs installed.

This will be really good to get sorted, and if we can get the windows 'upgrade' implemented or switched on, to do a full system upgrade of installed packages (just as the win_pkg doc page says it should do - https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.win_pkg.html), then hopefully that is going to not have or duplicate the same needless loop in its implementation. also see https://github.com/saltstack/salt/issues/31889

damon-atkins commented 8 years ago

other package manager would ignore the request if it happen in less than a few minutes. Their are more smarts required.

ghost commented 8 years ago

@damon-atkins, Dude, I'm just trying to fix a bug in a system I didn't design.

damon-atkins commented 8 years ago

I agree someone from saltstack needs to spend more time on it.