Dependencies reinstalled on every update

[janLo]

Home Assistant release (hass --version):

0.30.1

Python release (python3 --version):

Python 3.4.2

Component/platform:

homeassistant.util.package

Description of problem:

I've an installation of Homassistant in a python3 virtualenv. I update with pip3 install --upgrade homeassistant. After every upgrade i restart HA using hass --open-ui and on every start after an update it reinstalls the dependencies to ~/.homeassistant/deps. This also causes a reinstallation of e.g. pycrypto which takes a rally long time to compile on the target-banana-pi (15+ minutes for all modules). It will also be installed every time if I install it directly to the virtualenv HA is running from.

Example:

16-10-10 19:29:41 INFO (Thread-1) [homeassistant.util.package] Attempting install of sqlalchemy==1.0.15

Expected:

Only upgrade dependencies that have actually changed versions.

[fabaff]

If a dependency is present pip will skip it. Can be check by looking for Requirement already up-to-date: .. during the upgrade of Home Assistant. We are updating dependencies on a regular base thus this could lead to a bunch of dependencies which needed to be updated.

[MartinHjelmare]

This line removes the deps folder in the beginning of hass start after upgrade.

https://github.com/home-assistant/home-assistant/blob/dev/homeassistant/config.py#L251

[balloob]

Closing this issue as it is old. You can re-open it if you are able to reproduce it running the latest version.

[janLo]

The bug still remains. All dependencies got reinstalled on latest update. @MartinHjelmare pointed to the cause. (And I cannot reopen the issue as contributor @balloob )

[janLo]

From the Upgrade from 0.33.2 to 0.33.3:

16-11-24 15:59:06 INFO (Thread-2) [homeassistant.util.package] Attempting install of sqlalchemy==1.1.3
16-11-24 16:00:38 INFO (MainThread) [homeassistant.core] Bus:Handling <Event component_loaded[L]: component=recorder>
16-11-24 16:00:38 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=logger>
16-11-24 16:00:38 homeassistant.util.package: Attempting install of aiohttp_cors==0.4.0
16-11-24 16:03:18 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=http>
16-11-24 16:03:18 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=api>
16-11-24 16:03:18 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=frontend>
16-11-24 16:03:18 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=zone>
16-11-24 16:03:18 homeassistant.util.package: Attempting install of paho-mqtt==1.2
16-11-24 16:03:31 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=mqtt>
16-11-24 16:03:31 homeassistant.util.package: Attempting install of libnacl==1.5.0
16-11-24 16:03:46 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=device_tracker>
16-11-24 16:03:46 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=proximity>
16-11-24 16:03:46 homeassistant.util.package: Attempting install of python-mpd2==0.5.5
16-11-24 16:04:00 homeassistant.util.package: Attempting install of pychromecast==0.7.6
16-11-24 16:05:50 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=media_player>
16-11-24 16:05:50 homeassistant.util.package: Attempting install of distro==1.0.1
16-11-24 16:05:57 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=updater>
16-11-24 16:05:57 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=shell_command>
16-11-24 16:05:57 homeassistant.util.package: Attempting install of pyhomematic==0.1.16
16-11-24 16:06:11 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=homematic>
16-11-24 16:06:11 homeassistant.util.package: Attempting install of pilight==0.1.1
16-11-24 16:06:24 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=pilight>
16-11-24 16:06:24 homeassistant.util.package: Attempting install of wakeonlan==0.2.2
16-11-24 16:06:39 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=switch>
16-11-24 16:06:39 homeassistant.util.package: Attempting install of netdisco==0.7.7
16-11-24 16:07:56 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=discovery>
16-11-24 16:07:56 homeassistant.util.package: Attempting install of fastdotcom==0.0.1
16-11-24 16:08:14 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=sensor>
16-11-24 16:08:14 homeassistant.util.package: Attempting install of PyJWT==1.4.2
16-11-24 16:08:20 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=notify>
16-11-24 16:08:20 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=logbook>
16-11-24 16:08:20 homeassistant.util.package: Attempting install of fuzzywuzzy==0.14.0
16-11-24 16:08:27 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=conversation>
16-11-24 16:08:27 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=binary_sensor>
16-11-24 16:08:27 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=history>
16-11-24 16:08:27 homeassistant.util.package: Attempting install of astral==1.3
16-11-24 16:08:38 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=sun>
16-11-24 16:08:38 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=group>
16-11-24 16:08:39 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=automation>
16-11-24 16:08:39 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=script>
16-11-24 16:08:39 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=scene>
16-11-24 16:08:39 homeassistant.core: Bus:Handling <Event component_loaded[L]: component=climate>

The only thing that changed was the homeassistant-upgrade in the virtualenv:

(hass_env)hass@host:~$ pip install --upgrade homeassistant                                                                                                                                       
Collecting homeassistant
  Downloading homeassistant-0.33.3-py2.py3-none-any.whl (6.2MB)
    100% |████████████████████████████████| 6.2MB 25kB/s 
Requirement already up-to-date: jinja2>=2.8 in ./hass_env/lib/python3.4/site-packages (from homeassistant)
Requirement already up-to-date: pip>=7.0.0 in ./hass_env/lib/python3.4/site-packages (from homeassistant)
Requirement already up-to-date: pytz>=2016.7 in ./hass_env/lib/python3.4/site-packages (from homeassistant)
Requirement already up-to-date: async-timeout==1.0.0 in ./hass_env/lib/python3.4/site-packages (from homeassistant)
Requirement already up-to-date: aiohttp==1.0.5 in ./hass_env/lib/python3.4/site-packages (from homeassistant)
Requirement already up-to-date: voluptuous==0.9.2 in ./hass_env/lib/python3.4/site-packages (from homeassistant)
Requirement already up-to-date: pyyaml<4,>=3.11 in ./hass_env/lib/python3.4/site-packages (from homeassistant)
Requirement already up-to-date: typing<4,>=3 in ./hass_env/lib/python3.4/site-packages (from homeassistant)
Requirement already up-to-date: requests<3,>=2 in ./hass_env/lib/python3.4/site-packages (from homeassistant)
Requirement already up-to-date: MarkupSafe in ./hass_env/lib/python3.4/site-packages (from jinja2>=2.8->homeassistant)
Requirement already up-to-date: chardet in ./hass_env/lib/python3.4/site-packages (from aiohttp==1.0.5->homeassistant)
Requirement already up-to-date: multidict>=2.0 in ./hass_env/lib/python3.4/site-packages (from aiohttp==1.0.5->homeassistant)
Requirement already up-to-date: setuptools>=0.6b1 in ./hass_env/lib/python3.4/site-packages (from voluptuous==0.9.2->homeassistant)
Installing collected packages: homeassistant
  Found existing installation: homeassistant 0.33.2
    Uninstalling homeassistant-0.33.2:
      Successfully uninstalled homeassistant-0.33.2
Successfully installed homeassistant-0.33.3
(hass_env)hass@host:~$ hass

So the start after an bugfix-upgrade took more than eight minutes on a Banana Pi.

[balloob]

After every version upgrade all dependencies will have to be re-installed. That is by design.

[janLo]

Can you give this in the hands of pip so that I can install them outside "deps" in the virtualenv to not wait at least eight minutes after every bugfix release (or at least reinstall only after a new feature-release instead of every bugfix)?

Or why not just use install --upgrade ?

[balloob]

Home Assistant will already only install dependencies in the deps folder if it cannot find that package to be installed in the virtual environment.

We do not call pip install, as it means that we would install packages inside the running Python environment - which either could require root write access (if not in virtualenv).

There is no proper CLI to uninstall from a folder, so we decided to just remove the deps folder between version upgrades. You can find our code that deals with packages here: https://github.com/home-assistant/home-assistant/blob/dev/homeassistant/util/package.py

PR welcome that can easily upgrade package versions 👍

[balloobbot]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment :+1:

[lwis]

Do we want to continue tracking this, or is more information required?

[MartinHjelmare]

I have an idea for a solution. Don't know if it works yet though. Will test and report back.

[otherdave]

I believe I am seeing things being re-installed everytime I restart HA. I'm running off a hassbian image on a PI2 and just upgraded HA last night. I know for a fact it re-installed the pylutron packages for the Lutron Caseta support. Beyond that I don't have any specifics at this time.

After the upgrade the service restart took a bit of time and things were re-installed. Then I started to edit some config files (groups) and restarted the HA service again. I noticed that on that second restart, which was just a config file change, some stuff was re-installed.

I don't have my log files handy now but I can post them tonight. Is there anything specific to post? This would be version 0.44.2.

[MartinHjelmare]

Changes in config that change required dependencies will mean new dependencies installed during first start after the change.

[otherdave]

@MartinHjelmare thanks for the tip. I'll check tonight and see if I can confirm that's what is happening. I believe all I did was edit the group.yaml file to put some of my lutron devices into their own groups. Maybe that was enough to trigger a lutron re-install?

[balloob]

If something is re-installed, it means the entry in REQUIREMENTS is wrong. Probably a github url with the wrong package name contained in that zip file after the hash.

[otherdave]

@balloob can you tell me where/how to verify the REQUIREMENTS entry (or if I should create another separate issue to track this?) I'm using hassbian with the latest HA.

I just restarted my service and it is trying to reinstall pylutron-caseta based on the logs and what ps auxww is doing.

[lifeisafractal]

@balloob @otherdave I've seen the exact same issue with my setup. I'm on a fresh Hassbian image and also using the Lutron Caseta module which is reinstalling the pylutron-caseta on every restart of HA. I'd be interested exactly how the REQUIREMENTS entry could be wrong and how to check this as well. Happy to put in a PR if I get it fixed.

[balloob]

Please open a new issue and keep this issue on topic. I will be removing all off-topic replies after my reply.