search

jaraco / jaraco.abode

MIT License
11 stars 9 forks source link

Tests failing due to missing user data dir #16

Closed jaraco closed 1 year ago

jaraco commented 1 year ago 
* Tests were all failing until I created the `/root/.local/share/Abode` folder for the shelved cookie jar. I haven't looked at the code for `jaraco.net` yet, but maybe it needs to check whether the path exists before trying to save the cookies?

The tests do attempt to ensure a path is present. I suspect in the refactor to use platformdirs that the monkeypatch is failing and the tests are using the user's data dir (which is undesirable). But if that's the case, why are tests passing in CI? Probably this concern should be a separate issue.

Originally posted by @jaraco in https://github.com/jaraco/jaraco.abode/issues/15#issuecomment-1372502995

jaraco commented 1 year ago

I'm unable to replicate this behavior. I deleted my local user data dir, then ran the tests successfully, then checked and the user data dir is still missing:

 jaraco.abode main $ rm -r '~/Library/Application Support/Abode/'
 jaraco.abode main $ tox
.pkg: _optional_hooks> python /Users/jaraco/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_editable> python /Users/jaraco/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: build_editable> python /Users/jaraco/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
python: install_package> python -I -m pip install --force-reinstall --no-deps .tox/.tmp/package/81/jaraco.abode-3.1.2-0.editable-py3-none-any.whl
python: commands[0]> pytest -s
================================================================ test session starts =================================================================
platform darwin -- Python 3.11.1, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/python/.pytest_cache
rootdir: /Users/jaraco/code/jaraco/jaraco.abode, configfile: pytest.ini
plugins: black-0.3.12, mypy-0.10.3, checkdocs-2.9.0, flake8-1.1.1, enabler-2.0.0, cov-4.0.0, requests-mock-1.10.1.dev1
collected 327 items                                                                                                                                  

conftest.py ....
. .
docs/conf.py s.s
jaraco/abode/__init__.py s.s
jaraco/abode/__main__.py s.s
jaraco/abode/_itertools.py ...
jaraco/abode/automation.py ...
jaraco/abode/cli.py s.s.
jaraco/abode/client.py ...
jaraco/abode/config.py s.s
jaraco/abode/event_controller.py ....
jaraco/abode/exceptions.py s.s
jaraco/abode/settings.py s.s
jaraco/abode/socketio.py ....
jaraco/abode/devices/__init__.py s.s
jaraco/abode/devices/alarm.py s.s
jaraco/abode/devices/base.py ...
jaraco/abode/devices/binary_sensor.py s.s
jaraco/abode/devices/camera.py ...
jaraco/abode/devices/control.py s.s
jaraco/abode/devices/cover.py s.s
jaraco/abode/devices/light.py s.s
jaraco/abode/devices/lock.py s.s
jaraco/abode/devices/pkg.py s.s
jaraco/abode/devices/sensor.py s.s
jaraco/abode/devices/switch.py s.s
jaraco/abode/devices/valve.py s.s
jaraco/abode/helpers/__init__.py s.s
jaraco/abode/helpers/constants.py s.s
jaraco/abode/helpers/errors.py s.s
jaraco/abode/helpers/timeline.py s.s
jaraco/abode/helpers/urls.py s.s
tests/__init__.py s.s
tests/test_abode.py .............................
tests/test_alarm.py s.s...
tests/test_automation.py s.s......
tests/test_binary_sensor.py s.s.
tests/test_camera.py s.s..........
tests/test_device.py ..................
tests/test_dimmer.py s.s..
tests/test_door_lock.py s.s..
tests/test_event_controller.py s.s................
tests/test_hue.py s.s....
tests/test_lm.py s.s...
tests/test_power_switch_meter.py s.s..
tests/test_power_switch_sensor.py s.s..
tests/test_secure_barrier.py s.s..
tests/test_valve.py s.s..
tests/mock/__init__.py s.s
tests/mock/automation.py s.s
tests/mock/login.py s.s
tests/mock/logout.py s.s
tests/mock/oauth_claims.py s.s
tests/mock/panel.py s.s
tests/mock/user.py s.s
tests/mock/devices/__init__.py s.s
tests/mock/devices/alarm.py s.s
tests/mock/devices/dimmer.py s.s
tests/mock/devices/door_contact.py s.s
tests/mock/devices/door_lock.py s.s
tests/mock/devices/glass.py s.s
tests/mock/devices/hue.py s.s
tests/mock/devices/ipcam.py s.s
tests/mock/devices/ir_camera.py s.s
tests/mock/devices/keypad.py s.s
tests/mock/devices/lm.py s.s
tests/mock/devices/pir.py s.s
tests/mock/devices/power_switch_meter.py s.s
tests/mock/devices/power_switch_sensor.py s.s
tests/mock/devices/remote_controller.py s.s
tests/mock/devices/secure_barrier.py s.s
tests/mock/devices/siren.py s.s
tests/mock/devices/status_display.py s.s
tests/mock/devices/unknown.py s.s
tests/mock/devices/valve.py s.s
tests/mock/devices/water_sensor.py s.s

---------- coverage: platform darwin, python 3.11.1-final-0 ----------
Name                                        Stmts   Miss  Cover   Missing
-------------------------------------------------------------------------
conftest.py                                    53      0   100%
docs/conf.py                                    9      0   100%
jaraco/abode/__init__.py                        3      0   100%
jaraco/abode/_itertools.py                      6      0   100%
jaraco/abode/automation.py                     45      0   100%
jaraco/abode/cli.py                           242    169    30%   25-32, 50-58, 213, 222-227, 231, 235, 239-245, 250-251, 254-271, 274-278, 281-283, 286-291, 294-297, 300-307, 310-317, 320-327, 330-337, 340-353, 356-359, 362-369, 372-379, 382-389, 392-403, 406-419, 422-425, 428-439, 442-462, 466-468, 472-476, 481-487
jaraco/abode/client.py                        197      3    98%   155-156, 329
jaraco/abode/config.py                          7      2    71%   7-8
jaraco/abode/devices/__init__.py                0      0   100%
jaraco/abode/devices/alarm.py                  86      0   100%
jaraco/abode/devices/base.py                  127      2    98%   32-33
jaraco/abode/devices/binary_sensor.py          10      1    90%   22
jaraco/abode/devices/camera.py                114      6    95%   150-152, 186, 195, 205
jaraco/abode/devices/control.py                 9      0   100%
jaraco/abode/devices/cover.py                  24      0   100%
jaraco/abode/devices/light.py                  65      0   100%
jaraco/abode/devices/lock.py                   17      0   100%
jaraco/abode/devices/pkg.py                     9      0   100%
jaraco/abode/devices/sensor.py                 43      0   100%
jaraco/abode/devices/switch.py                 27      3    89%   44, 51, 58
jaraco/abode/devices/valve.py                  20      0   100%
jaraco/abode/event_controller.py              169     28    83%   61, 65, 69-76, 80-87, 127, 175, 180, 184, 188-200, 204-208
jaraco/abode/exceptions.py                     21      1    95%   30
jaraco/abode/helpers/__init__.py                0      0   100%
jaraco/abode/helpers/constants.py              82      0   100%
jaraco/abode/helpers/errors.py                 33      0   100%
jaraco/abode/helpers/timeline.py               34      0   100%
jaraco/abode/helpers/urls.py                   20      0   100%
jaraco/abode/settings.py                      130      0   100%
jaraco/abode/socketio.py                      230    135    41%   89, 92, 115-118, 122-125, 130, 140-147, 151-159, 162-185, 188-211, 214-218, 221-227, 230-247, 250-264, 267, 270-282, 285-289, 292-293, 296-308, 311-313, 316-320, 323-325, 328-339, 342-349
tests/__init__.py                               0      0   100%
tests/mock/__init__.py                          6      0   100%
tests/mock/automation.py                        2      0   100%
tests/mock/devices/__init__.py                  6      0   100%
tests/mock/devices/alarm.py                     5      0   100%
tests/mock/devices/dimmer.py                    5      0   100%
tests/mock/devices/door_contact.py              4      0   100%
tests/mock/devices/door_lock.py                 5      0   100%
tests/mock/devices/glass.py                     4      0   100%
tests/mock/devices/hue.py                      12      0   100%
tests/mock/devices/ipcam.py                    13      0   100%
tests/mock/devices/ir_camera.py                12      0   100%
tests/mock/devices/keypad.py                    4      0   100%
tests/mock/devices/lm.py                        6      0   100%
tests/mock/devices/pir.py                       4      0   100%
tests/mock/devices/power_switch_meter.py        5      0   100%
tests/mock/devices/power_switch_sensor.py       5      0   100%
tests/mock/devices/remote_controller.py         4      0   100%
tests/mock/devices/secure_barrier.py            5      0   100%
tests/mock/devices/siren.py                     4      0   100%
tests/mock/devices/status_display.py            4      0   100%
tests/mock/devices/unknown.py                   4      0   100%
tests/mock/devices/valve.py                     5      0   100%
tests/mock/devices/water_sensor.py              4      0   100%
tests/mock/login.py                            13      0   100%
tests/mock/logout.py                            4      0   100%
tests/mock/oauth_claims.py                      3      0   100%
tests/mock/panel.py                             5      0   100%
tests/mock/user.py                              2      0   100%
tests/test_abode.py                           297      0   100%
tests/test_alarm.py                           126      0   100%
tests/test_automation.py                      140      0   100%
tests/test_binary_sensor.py                    37      0   100%
tests/test_camera.py                          167      0   100%
tests/test_device.py                          189      0   100%
tests/test_dimmer.py                           60      0   100%
tests/test_door_lock.py                        54      0   100%
tests/test_event_controller.py                224      0   100%
tests/test_hue.py                             109      0   100%
tests/test_lm.py                               84      0   100%
tests/test_power_switch_meter.py               54      0   100%
tests/test_power_switch_sensor.py              54      0   100%
tests/test_secure_barrier.py                   51      0   100%
tests/test_valve.py                            55      0   100%
-------------------------------------------------------------------------
TOTAL                                        3688    350    91%

======================================================================== mypy ========================================================================

Success: no issues found in 75 source files
========================================================= 196 passed, 130 skipped in 10.35s ==========================================================
.pkg: _exit> python /Users/jaraco/.local/pipx/venvs/tox/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
  python: OK (12.19=setup[1.01]+cmd[11.18] seconds)
  congratulations :) (12.26 seconds)
 jaraco.abode main $ ls '~/Library/Application Support/Abode'
ls: /Users/jaraco/Library/Application Support/Abode: No such file or directory

@tradel I'm unable to replicate your finding in either CI or locally. Do you have any advice on how I can replicate the issue?

jaraco commented 1 year ago

Feel free to follow up with more details about the conditions lead to reliance on a missing user data dir. In the meantime, I'll close the issue.

tradel commented 1 year ago

Looks like this is only causing the HomeAssistant unit tests to fail under Python 3.10. Not sure why that is. Take a look at this CI run:

https://github.com/home-assistant/core/actions/runs/3869320705/jobs/6596738827

The same tests all passed with py3.9.

jaraco commented 1 year ago

That helps. I think I know what's going on.

First the traceback:

Traceback (most recent call last):
  File "/home/runner/work/core/core/venv/lib/python3.10/site-packages/jaraco/functools.py", line 343, in retry_call
    return func()
  File "/home/runner/work/core/core/venv/lib/python3.10/site-packages/jaraco/abode/client.py", line 38, in _cookies
    return cookies.ShelvedCookieJar.create(config.paths.user_data)
  File "/home/runner/work/core/core/venv/lib/python3.10/site-packages/jaraco/abode/config.py", line 7, in user_data
    self.user_data_path.mkdir(exist_ok=True)
  File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/.local/share/Abode'
  1. The homeassistant tests run without invoking any of the fixtures in jaraco.abode, so the user data isn't overridden for tests (that may not be a problem).
  2. When the application runs, it automatically creates the user_data dir if it doesn't exist. Prior to 3.1, this project used app_paths to ensure the directory existed. With the introduction of platformdirs, I had to roll my own.
  3. The reason it's probably only failing in some environments is because only some have the .local/share parent directory created for some other reason.
  4. That implementation has a bug where if the parent directory doesn't exist, it will fail with the reported error.
  5. Because that code is mocked out during tests, it's untested.

The quick and easy fix would be to add parents=True to that call. I'd like to also produce a test to capture the expectation.

tradel commented 1 year ago

Great job figuring this out!

On Mon, Jan 9, 2023 at 10:51 AM Jason R. Coombs @.***> wrote:

Closed #16 https://github.com/jaraco/jaraco.abode/issues/16 as completed via cd9ecf8 https://github.com/jaraco/jaraco.abode/commit/cd9ecf8c53f289344d7385a45f76ca2e3893c23c .

— Reply to this email directly, view it on GitHub https://github.com/jaraco/jaraco.abode/issues/16#event-8183212706, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANRHP7BCWBLY75X7YZU3B3WRQXZFANCNFSM6AAAAAATSGALVY . You are receiving this because you were mentioned.Message ID: @.***>

jaraco commented 1 year ago

Assuming tests pass, fix should release with v3.2.1.

tradel commented 1 year ago

One question … why does it need to create a folder if you’re no longer saving cache to a file?

On Mon, Jan 9, 2023 at 10:52 AM Todd Radel @.***> wrote:

Great job figuring this out!

On Mon, Jan 9, 2023 at 10:51 AM Jason R. Coombs @.***> wrote:

Closed #16 https://github.com/jaraco/jaraco.abode/issues/16 as completed via cd9ecf8 https://github.com/jaraco/jaraco.abode/commit/cd9ecf8c53f289344d7385a45f76ca2e3893c23c .

— Reply to this email directly, view it on GitHub https://github.com/jaraco/jaraco.abode/issues/16#event-8183212706, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANRHP7BCWBLY75X7YZU3B3WRQXZFANCNFSM6AAAAAATSGALVY . You are receiving this because you were mentioned.Message ID: @.***>

jaraco commented 1 year ago

One question … why does it need to create a folder if you’re no longer saving cache to a file?

The library still saves the state of the cookies. It just no longer calls it a "cache", but instead uses generically "user data" and specifically "cookies". It serves the same purpose as the previous cache, but avoids writing to the current directory and outsources the state management (when to persist/load) to other libraries.

It still uses a JSON format, so the cookies file is human readable.

For example, here's what's on my machine:

 ~ $ cat '~/Library/Application Support/Abode/cookies.json' | jq
{
  "my.goabode.com": {
    "/": {
      "SESSION": {
        "py/object": "http.cookiejar.Cookie",
        "version": 0,
        "name": "SESSION",
        "value": "****",
        "port": null,
        "port_specified": false,
        "domain": "my.goabode.com",
        "domain_specified": false,
        "domain_initial_dot": false,
        "path": "/",
        "path_specified": true,
        "secure": false,
        "expires": null,
        "discard": true,
        "comment": null,
        "comment_url": null,
        "rfc2109": false,
        "_rest": {}
      }
    }
  },
  ".goabode.com": {
    "/": {
      "uuid": {
        "py/object": "http.cookiejar.Cookie",
        "version": 0,
        "name": "uuid",
        "value": "de933f38-8e2a-11ed-976d-2e60085334b8",
        "port": null,
        "port_specified": false,
        "domain": ".goabode.com",
        "domain_specified": true,
        "domain_initial_dot": true,
        "path": "/",
        "path_specified": true,
        "secure": false,
        "expires": 3820683723,
        "discard": false,
        "comment": null,
        "comment_url": null,
        "rfc2109": false,
        "_rest": {}
      }
    }
  }
}