search

Kintyre / ksconf

Kintyre's Splunk Configuration tool
Apache License 2.0
50 stars 13 forks source link

Ksconf 0.10 package corrupted after upgrade #94

Closed lowell80 closed 1 year ago

lowell80 commented 1 year ago

The problem

Upgrading the kintyre-splunk-conf package to version 0.10 breaks the ksconf command with a "command not found" message. Upgrade was done using pip.

Environment

Details

Copying from the release notes:

The official Python package was renamed ksconf. The kintyre-splunk-conf package continue to be released in parallel at least until version 1.0. You can continue updating the kintyre-splunk-conf package and it will install the ksconf package as a dependency.

That's how I want it to work, The older kintyre-splunk-conf becomes a "stub" that simply causes the newer ksconf package to be installed. However what happens is that the pip package inventory shows that it kintyre-splunk-conf was updated and ksconf was installed (both which show the same version #), but the act of upgrades cause things to break. I think it's because the same file are involved with both, so pip seems to get confused about which app owns the files. The end result is that all of the shared *.py files (which is 90%+ of the code base) is missing in the installation folder. Forcing a re-install fixes it. (So the default behavior of simply running pip install -U kintyre-splunk-conf results in the executable breaking.). I couldn't find any other python packages out there that have done this kind of thing and how they've gotten around this problem.

My other approach is to release both packages with identical content (other than package metadata, of course), and force users to manually switch between to the new package name eventually. I don't really like this approach as much, but I guess it's better than breaking existing installs.

Requesting help of python packaging experts for guidance!

Steps To Reproduce Issue

  1. Install kintyre-splunk-conf via pip. (Assuming 0.9.x, but any earlier version would likely result in a similarly broken install)
  2. Upgrade kintyre-splunk-conf (past version 0.10)
  3. ksconf is now dead :-(

Commands

# Create & activate virtual environment for isolated test
python3 -m venv ksconf_upgrade_bug_venv
. ksconf_upgrade_bug_venv/bin/activate

# Make sure pip is latest version (wheel to speedup lxml install)
pip install -U pip wheel

# Install ksconf 0.9.3 (as of Dec 2022)
pip install kintyre-splunk-conf

# Upgrade latest version (since 0.10 not officially, released; pin to latest beta)
pip install -U kintyre-splunk-conf==0.10b4

# Test ksconf (spoiler:  it's now dead)
ksconf --help

# Output: command not found: ksconf

Example terminal output

lalleman@MAC-PRO-LALLEMAN Downloads % python3 -m venv ksconf_upgrade_bug_venv
lalleman@MAC-PRO-LALLEMAN Downloads % . ./ksconf_upgrade_bug_venv/bin/activate
Requirement already satisfied: pip in ./ksconf_upgrade_bug_venv/lib/python3.9/site-packages (21.2.4)
Collecting pip
  Using cached pip-22.3.1-py3-none-any.whl (2.1 MB)
Collecting wheel
  Using cached wheel-0.38.4-py3-none-any.whl (36 kB)
Installing collected packages: wheel, pip
  Attempting uninstall: pip
    Found existing installation: pip 21.2.4
    Uninstalling pip-21.2.4:
      Successfully uninstalled pip-21.2.4
Successfully installed pip-22.3.1 wheel-0.38.4
(ksconf_upgrade_bug_venv) lalleman@MAC-PRO-LALLEMAN Downloads % pip install kintyre-splunk-conf
Collecting kintyre-splunk-conf
  Using cached kintyre_splunk_conf-0.9.3-py2.py3-none-any.whl (130 kB)
Collecting lxml
  Using cached lxml-4.9.2-cp39-cp39-macosx_10_9_universal2.whl
Collecting entrypoints
  Using cached entrypoints-0.4-py3-none-any.whl (5.3 kB)
Installing collected packages: lxml, entrypoints, kintyre-splunk-conf
Successfully installed entrypoints-0.4 kintyre-splunk-conf-0.9.3 lxml-4.9.2
(ksconf_upgrade_bug_venv) lalleman@MAC-PRO-LALLEMAN Downloads % pip install kintyre-splunk-conf
Collecting kintyre-splunk-conf
  Using cached kintyre_splunk_conf-0.9.3-py2.py3-none-any.whl (130 kB)
Collecting lxml
  Using cached lxml-4.9.2-cp39-cp39-macosx_10_9_universal2.whl
Collecting entrypoints
  Using cached entrypoints-0.4-py3-none-any.whl (5.3 kB)
Installing collected packages: lxml, entrypoints, kintyre-splunk-conf
Successfully installed entrypoints-0.4 kintyre-splunk-conf-0.9.3 lxml-4.9.2
(ksconf_upgrade_bug_venv) lalleman@MAC-PRO-LALLEMAN Downloads %find ksconf_upgrade_bug_venv/lib/python3.9/site-packages/ksconf -name '*.py' | wc -l
      44
(ksconf_upgrade_bug_venv) lalleman@MAC-PRO-LALLEMAN Downloads % pip install -U kintyre-splunk-conf==0.10b4
Collecting kintyre-splunk-conf==0.10b4
  Downloading kintyre_splunk_conf-0.10.0b4-py2.py3-none-any.whl (5.5 kB)
Requirement already satisfied: ksconf==0.10.0-beta4 in ./ksconf_upgrade_bug_venv/lib/python3.9/site-packages (from kintyre-splunk-conf==0.10b4) (0.10.0b4)
Requirement already satisfied: lxml in ./ksconf_upgrade_bug_venv/lib/python3.9/site-packages (from ksconf==0.10.0-beta4->kintyre-splunk-conf==0.10b4) (4.9.2)
Requirement already satisfied: entrypoints in ./ksconf_upgrade_bug_venv/lib/python3.9/site-packages (from ksconf==0.10.0-beta4->kintyre-splunk-conf==0.10b4) (0.4)
Installing collected packages: kintyre-splunk-conf
  Attempting uninstall: kintyre-splunk-conf
    Found existing installation: kintyre-splunk-conf 0.9.3
    Uninstalling kintyre-splunk-conf-0.9.3:
      Successfully uninstalled kintyre-splunk-conf-0.9.3
Successfully installed kintyre-splunk-conf-0.10.0b4
(ksconf_upgrade_bug_venv) lalleman@MAC-PRO-LALLEMAN Downloads % ksconf --version
zsh: command not found: ksconf
(ksconf_upgrade_bug_venv) lalleman@MAC-PRO-LALLEMAN Downloads % ksconf --help
zsh: command not found: ksconf
(ksconf_upgrade_bug_venv) lalleman@MAC-PRO-LALLEMAN Downloads % pip list
Package             Version
------------------- --------
entrypoints         0.4
kintyre-splunk-conf 0.10.0b4
ksconf              0.10.0b4
lxml                4.9.2
pip                 22.3.1
setuptools          58.0.4
wheel               0.38.4
(ksconf_upgrade_bug_venv) lalleman@MAC-PRO-LALLEMAN Downloads % find ksconf_upgrade_bug_venv/lib/python3.9/site-packages/ksconf -name '*.py' -ls
7728278       24 -rw-r--r--    1 lalleman         staff               10405 Dec 20 12:19 ksconf_upgrade_bug_venv/lib/python3.9/site-packages/ksconf/combine.py
7728277       24 -rw-r--r--    1 lalleman         staff               11430 Dec 20 12:19 ksconf_upgrade_bug_venv/lib/python3.9/site-packages/ksconf/cli.py

Fix command:

% pip install --force-reinstall ksconf
Collecting ksconf
  Using cached ksconf-0.10.0b4-py2.py3-none-any.whl (123 kB)
Collecting lxml
  Using cached lxml-4.9.2-cp39-cp39-macosx_10_9_universal2.whl
Collecting entrypoints
  Using cached entrypoints-0.4-py3-none-any.whl (5.3 kB)
Installing collected packages: lxml, entrypoints, ksconf
  Attempting uninstall: lxml
    Found existing installation: lxml 4.9.2
    Uninstalling lxml-4.9.2:
      Successfully uninstalled lxml-4.9.2
  Attempting uninstall: entrypoints
    Found existing installation: entrypoints 0.4
    Uninstalling entrypoints-0.4:
      Successfully uninstalled entrypoints-0.4
  Attempting uninstall: ksconf
    Found existing installation: ksconf 0.10.0b4
    Uninstalling ksconf-0.10.0b4:
      Successfully uninstalled ksconf-0.10.0b4
Successfully installed entrypoints-0.4 ksconf-0.10.0b4 lxml-4.9.2
 ksconf --version
 ___  ____           ______                     ___
|_  ||_  _|        .' ___  |                  .' ..]
  | |_/ /    .--. / .'   \_|  .--.   _ .--.  _| |_
  |  __'.   ( (`\]| |       / .'`\ \[ `.-. |'-| |-'
 _| |  \ \_  `'.'.\ `.___.'\| \__. | | | | |  | |
|____||____|[\__) )`.____ .' '.__.' [___||__][___]

ksconf 0.10.0-beta4  (Build 1080)
Python: 3.9.6  (/Users/lalleman/Downloads/ksconf_upgrade_bug_venv/bin/python3)
Git SHA1 8d4176e5 committed on 2022-04-20
...

Alternate fix once in a broken state:

pip install -IU kintyre-splunk-conf

Workaround

pip uninstall kintyre-splunk-conf
pip install ksconf==0.10b4                # Specific version won't be required once (non-beta) 0.10.0 is released

Pip issue???

I think the core issue is the order in which pip is performing the operations. Because both kintyre-spunk-conf and ksconf share most of the same files, the fact that pip uninstalls kintyre-splunk-conf-0.9.3 after it installs ksconf.... ctually, it thinks that ksconf is already installed ksconf==0.10.0-beta4 (Requirement already satisfied). So maybe it just never get's installed. (Explicitly just telling pip install ksconf doesn't work either, it really thinks it installed it already, so this all feels very weird.)

(ksconf_upgrade_bug_venv) lalleman@MAC-PRO-LALLEMAN Downloads % pip install -U kintyre-splunk-conf==0.10b4
Collecting kintyre-splunk-conf==0.10b4
  Downloading kintyre_splunk_conf-0.10.0b4-py2.py3-none-any.whl (5.5 kB)
Requirement already satisfied: ksconf==0.10.0-beta4 in ./ksconf_upgrade_bug_venv/lib/python3.9/site-packages (from kintyre-splunk-conf==0.10b4) (0.10.0b4)
Requirement already satisfied: lxml in ./ksconf_upgrade_bug_venv/lib/python3.9/site-packages (from ksconf==0.10.0-beta4->kintyre-splunk-conf==0.10b4) (4.9.2)
Requirement already satisfied: entrypoints in ./ksconf_upgrade_bug_venv/lib/python3.9/site-packages (from ksconf==0.10.0-beta4->kintyre-splunk-conf==0.10b4) (0.4)
Installing collected packages: kintyre-splunk-conf
  Attempting uninstall: kintyre-splunk-conf
    Found existing installation: kintyre-splunk-conf 0.9.3
    Uninstalling kintyre-splunk-conf-0.9.3:
      Successfully uninstalled kintyre-splunk-conf-0.9.3
Successfully installed kintyre-splunk-conf-0.10.0b4

I'm not sure if there's a packaging change that can fix this?

pmeyerson commented 1 year ago

BTW, I can confirm the behavior looks the same on other package manager poetry. binary executables and other files are removed from the venv/bin directory.

Just trying to rule out any pip-only issues. HTH

Reproduce

pipx install poetry
poetry new ksconf-test
cd ksconf-test
poetry shell
poetry add ksconf
poetry env info
cd to virtualenv.path displayed, confirm binary ksconf present
cd ~/ksconf-test
poetry add ksconf-splunk-conf==0.10b4
cd to virtualenv.path displayed, confirm binary ksconf missing
lowell80 commented 1 year ago

Thanks for confirming that for me. That's helpful to know. I generally stick with pip, but have heard of poetry, just never taken the dive myself.

I may just have to push forward and explain/document the one-time workaround (re-install). It's annoying, but I don't want to continue holding up the next release when there may not be a great solution, and the workaround is pretty simple.

lowell80 commented 1 year ago

Closing this issue as v0.10 has been release.

The build automation simply builds and releases two package with the exact same content (ksconf + kintyre-splunk-conf). The only difference between the two is the README. Eventually we'll add deprecation warnings in the ksconf-splunk-conf package telling everyone to upgrade to ksconf. The final ksconf-splunk-conf release may just "require" the new ksconf package. At that point, hopefully the only references to the old name are old automation jobs (and for fresh-installs, the package dependency approach works fine anyways).