Request to not pin dependencies

[allenporter]

It would be super helpful if this project did not pin required dependencies to specific versions. The common symptom is that basically if downstream projects are ahead then we'll see errors like this:

The conflict is caused by:
    The user requested pytest-asyncio==0.23.7
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
    pytest-homeassistant-custom-component 0.13.124 depends on pytest-asyncio==0.23.6

I understand why it may be useful to block major versions of pytest or something or known bad issues, so that is ok, but minor version pinning is too strict.

(My impression is this happens because this project shared dependencies between setup and dev. Its usually fine to pin dependencies for dev -- i do this also -- but left the required packages in setup.py drift with a minimum version).

Thank you for maintaining this great library, i use this in nearly all of my home assistant related projects.

[MatthewFlamm]

This project uses the exact same testing versions as homeassistant for that version of homeassistant. Why would you want to be ahead of homeassistant?

[allenporter]

I use automated dependency updates like renovatebot/mend. It will run tests via CI though and fail if something is broken and not merge the update. This is pretty common for lots of projects and its done on the development updates to be prepared for moving to newer releases.

(We also could talk more about moving ahead with pre-release versions of home assistant though. It's nice to have things working before releases happen, though i know it can be churny)

[MatthewFlamm]

The downside of this approach is that most users probably want to test with the versions that are known to work with homeassistant for that version they are testing. I often find that homeassistant tests will only work on specific versions of the test dependencies, and I fear there will be more issues raised here if we don't pin. Is there a way to accomplish both goals?

(We also could talk more about moving ahead with pre-release versions of home assistant though. It's nice to have things working before releases happen, though i know it can be churny)

This project tries to release a version for every pre-release version. See https://github.com/MatthewFlamm/pytest-homeassistant-custom-component/releases/tag/0.13.126 as an example.

[allenporter]

The downside of this approach is that most users probably want to test with the versions that are known to work with homeassistant for that version they are testing. I often find that homeassistant tests will only work on specific versions of the test dependencies, and I fear there will be more issues raised here if we don't pin. Is there a way to accomplish both goals?

I would be surprised if that happens for minor patch releases.

This project tries to release a version for every pre-release version. See https://github.com/MatthewFlamm/pytest-homeassistant-custom-component/releases/tag/0.13.126 as an example.

OK, that sounds right. (I was doing pre-pre-release this time since i'm helping with some of the new AI feature, but realize that is not common)

[allenporter]

I also just noticed this on one of my systems:

ERROR: Could not find a version that satisfies the requirement mypy-dev==1.10.0a3 (from pytest-homeassistant-custom-component) (from versions: 1.9.0b1, 1.10.0b1, 1.11.0a1, 1.11.0a2, 1.11.0a3, 1.11.0a5)

So a mypy-dev version is pinned... that version doesn't exist anymore or something? Not sure that we're getting lots of value from having specific pins like this, but now i'm going to have to upgrade anyway to a new pinned version.

[MatthewFlamm]

This might be caused by this line, which we definitely do not need: https://github.com/MatthewFlamm/pytest-homeassistant-custom-component/blob/741bbeaed38f1c94841e31d6dcd419e0192d551f/requirements_test.txt#L10

[MatthewFlamm]

I would be surprised if that happens for minor patch releases.

I think this is besides the point, but this package history shows that not infrequently a patch release has significant testing changes that require reconfiguring the extraction code, let alone significant changes in testing library versions.

[pnbruckner]

I also just noticed this on one of my systems:

ERROR: Could not find a version that satisfies the requirement mypy-dev==1.10.0a3 (from pytest-homeassistant-custom-component) (from versions: 1.9.0b1, 1.10.0b1, 1.11.0a1, 1.11.0a2, 1.11.0a3, 1.11.0a5)

So a mypy-dev version is pinned... that version doesn't exist anymore or something?

I just ran into this, too. I use this repo for automated testing of my HA custom integrations, and I use several versions to test my integrations with several versions of HA. I try to maintain backwards compatibility, at least for a while.

Anyway, this was working until just very recently. I think whoever maintains mypy-dev on pypi.org just recently dropped the 1.10.0a3 release, so now all versions of this repo, until maybe the last couple of days, no longer work. 😞

[MatthewFlamm]

Actually this is a problem in the requirements file from homeassistant itself. There really is nothing we can do here until homeassistant updates itself to fix the problem unfortunatley... https://github.com/MatthewFlamm/pytest-homeassistant-custom-component/blob/741bbeaed38f1c94841e31d6dcd419e0192d551f/requirements_test.txt#L14

[pnbruckner]

That's not really the problem. homeassistant already did update to 1.11.0a3, which does exist on pypi.org.

The problem, at least for me, is being able to test a custom integration with older versions of HA using older versions of this repo. Actually, the problem is the maintainer of mypy-dev on pypi.org might not understand that. The version needed (in my case, 1.10.0a3), although still available via the mypy-dev repo, was pulled down from pypi.org.

Possibly the real problem is homeassistant using mypy-dev in the first place. It warns "Use at your own risk!"

[MatthewFlamm]

If home assistant updated, then this package will also update once a release is made there and thus here. Again, this package is meant to be as close to homeassistant testing as possible, if there is a problem in homeassistant, then there is a problem here too.

[allenporter]

My take is that this library should probably not have an opinion on mypy versioning. Home assistant does for its own code base for running its own tests, and does not maintain a separate way to express dependencies for the core of the home assistant tests. It might make sense to limit the number of dependencies copied from home assistant dependencies. (Either allow list or deny list).

(I don't necessarily think the ha requirements test file is a problem but agree depending on mypy dev is suspect. But this project should care about hole assistants mypy preferences I think)

[MatthewFlamm]

This slipped in because we were looking for mypy and not mypy-dev

https://github.com/MatthewFlamm/pytest-homeassistant-custom-component/blob/741bbeaed38f1c94841e31d6dcd419e0192d551f/generate_phacc/const.py#L27

[MatthewFlamm]

mypy-dev is now also excluded from the package, and just released with the most recent pre-release of homeassistant.

[cdce8p]

Sorry for the issue I caused when I removed the old alpha releases for mypy-dev. As I explain in https://github.com/cdce8p/mypy-dev/issues/20#issuecomment-2143122562 I mainly tried to stay clear of the PyPI project limit. Maybe a bit to aggressively in the past.

[allenporter]

Thank you for fixing the mypy exclusion. Glad to see that was already the case and just slipped in because of the package name change.

I would be surprised if that happens for minor patch releases.

I think this is besides the point, but this package history shows that not infrequently a patch release has significant testing changes that require reconfiguring the extraction code, let alone significant changes in testing library versions.

The point I was trying to raise with a semvar patching is was about expectations about drift (major=breaking changes, minor=new functionality, patch=bug fixes) e.g. some pinning semvar ranges to avoid a new major versions is a common pattern I see.

If you're saying minor patches for dependencies (e.g. pytest, pytest-asyncio, etc) break things regularly that is surprising.

It sounds like your expectation would be that all projects that depend on pytest-homeassistant-custom-component do not try to do any of their own pinning of any test dependencies and rely on them coming through this library, or pin the exact same versions? This isn't really a pattern i've observed before, but i can see it might be unique to testing packages closely tied to another project. It'll essentially mean automated dependency management is more difficult. (Maybe solvable with renovate package groupings, but cumbersome)

I do understand home assistant test fixtures are (1) quite involved and (2) change quite a bit and (3) try to stay at the bleeding edge and have a close relationship with features in upstream projects. Hence -- why I really appreciate you taking the effort to maintain this project, and have some idea about the effort it takes to do so.

I think you've got all the information to make a decision (including status quo) so happy to close. The result of status quo

[MatthewFlamm]

Let's leave this open as my stance is not set in stone. I agree with you for 99.x% of projects, it is preferable to not pin dependencies like this. I can see why pinning can tie the hands of developers here. My main stake in the ground is that this project is akin to a mirror of homeassistant; we are just extracting the testing and packaging it in a different way. This package tries to do as little as possible in changing what comes from homeassistant.

One smaller part of this puzzle is that the homeassistant core devs often fork popular repositories to add in their own bleeding edge functionality, see mypy-dev above as an example.

I had one thought, maybe crazy, not even sure it works. One thought is we could have unpinned dependencies, but have pinned dependencies as optional, e.g. pytest-homeassistant-custom-component[pinned].

[allenporter]

I'd propose tqdm should also be excluded. I recommend moving to an allow list rather than a deny list.