[Bug] (only on linuxserver/homeassistant) PermissionError: [Errno 13] Permission denied: 'solcast-sites.json'

[BJReplay]

This error was first reported by @florie1706 in https://github.com/BJReplay/ha-solcast-solar/issues/32 at https://github.com/BJReplay/ha-solcast-solar/issues/32#issuecomment-2173672386

It can be repro'd with a fresh pull of linuxserver/homeassistant and a fresh manual install of BJReplay/ha-solcast-solar (and also a HACS manual install of ha-solcast-solar).

chmod 777 config within the linuxserver/homeassistant container doesn't resolve the issue.

To repro:

Create a new container - this created a new container for me, and mapped the default port onto 8124, and the config onto a host folder that I had created (hals).

I logged into the container, checked the directories, pulled the repro, and moved the ha-solcast-solar folder into custom components, and restarted HA (from the GUI - not shown).

docker run -d   --name=homeassistantls  -e PUID=1000   -e PGID=1000   -e TZ=Australia/Melbourne -p 8124:8123  -v ./hals:/config lscr.io/linuxserver/homeassistant:latest
docker exec -it homeassistantls /bin/bash
ls
root@2fd2f1d6ad2a:/# ls
app  bin  command  config  defaults  dev  docker-mods  etc  home  init  lib  lsiopy  media  mnt  opt  package  proc  root  run  sbin  srv  sys  tmp  usr  var
root@2fd2f1d6ad2a:/# cd config
root@2fd2f1d6ad2a:/config# ls
automations.yaml  blueprints  configuration.yaml  deps  home-assistant.log  home-assistant.log.1  home-assistant.log.fault  home-assistant_v2.db  home-assistant_v2.db-shm  home-assistant_v2.db-wal  scenes.yaml  scripts.yaml  secrets.yaml  tts
root@2fd2f1d6ad2a:/config# mkdir custom_components
root@2fd2f1d6ad2a:/config# cd~ 
root@2fd2f1d6ad2a:/# git clone https://github.com/BJReplay/ha-solcast-solar
Cloning into 'ha-solcast-solar'...
remote: Enumerating objects: 2207, done.
remote: Counting objects: 100% (588/588), done.
remote: Compressing objects: 100% (166/166), done.
remote: Total 2207 (delta 469), reused 480 (delta 398), pack-reused 1619
Receiving objects: 100% (2207/2207), 926.06 KiB | 3.09 MiB/s, done.
Resolving deltas: 100% (1378/1378), done.
root@2fd2f1d6ad2a:/# cd ha-solcast-solar/
root@2fd2f1d6ad2a:/ha-solcast-solar# ls
CODEOWNERS  LICENSE.md  README.md  custom_components  hacs.json
root@2fd2f1d6ad2a:/ha-solcast-solar#  cd custom_components/
root@2fd2f1d6ad2a:/ha-solcast-solar/custom_components# ls -al
total 12
drwxr-xr-x 3 root root 4096 Jun 18 16:09 .
drwxr-xr-x 5 root root 4096 Jun 18 16:09 ..
drwxr-xr-x 3 root root 4096 Jun 18 16:09 solcast_solar
root@2fd2f1d6ad2a:/ha-solcast-solar/custom_components# mv solcast_solar/ /config/custom_components/

Another thought, is solcast-usage.json getting created in /config?

No.

root@2fd2f1d6ad2a:/config# ls -al
total 772
drwxr-xr-x 8 abc  users   4096 Jun 18 16:11 .
drwxr-xr-x 1 root root    4096 Jun 18 16:09 ..
-rw-r--r-- 1 abc  users      8 Jun 18 15:51 .HA_VERSION
drwxr-xr-x 2 abc  users   4096 Jun 18 15:51 .cloud
drwxr-xr-x 2 abc  users   4096 Jun 18 16:14 .storage
-rw-r--r-- 1 abc  users      2 Jun 18 15:51 automations.yaml
drwxr-xr-x 4 abc  users   4096 Jun 18 15:51 blueprints
-rw-r--r-- 1 abc  users    265 Jun 18 15:51 configuration.yaml
drwxr-xr-x 3 root root    4096 Jun 18 16:10 custom_components
drwxr-xr-x 2 abc  users   4096 Jun 18 15:51 deps
-rw-r--r-- 1 abc  users   4995 Jun 18 16:12 home-assistant.log
-rw-r--r-- 1 abc  users      0 Jun 18 15:58 home-assistant.log.1
-rw-r--r-- 1 abc  users      0 Jun 18 16:11 home-assistant.log.fault
-rw-r--r-- 1 abc  users 253952 Jun 18 16:11 home-assistant_v2.db
-rw-r--r-- 1 abc  users  32768 Jun 18 16:15 home-assistant_v2.db-shm
-rw-r--r-- 1 abc  users 444992 Jun 18 16:15 home-assistant_v2.db-wal
-rw-r--r-- 1 abc  users      0 Jun 18 15:51 scenes.yaml
-rw-r--r-- 1 abc  users      0 Jun 18 15:51 scripts.yaml
-rw-r--r-- 1 abc  users    161 Jun 18 15:51 secrets.yaml
drwxr-xr-x 2 abc  users   4096 Jun 18 15:51 tts

On restart, after configuring the integration with my API key, the following errors - as reported in the earlier issue - are recorded.

root@2fd2f1d6ad2a:/config# cat home-assistant.log
2024-06-18 16:11:25.670 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration solcast_solar which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-06-18 16:11:59.113 ERROR (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - sites_data Exception error: Traceback (most recent call last):
  File "/config/custom_components/solcast_solar/solcastapi.py", line 148, in sites_data
    async with aiofiles.open(apiCacheFileName, 'w') as f:
  File "/usr/local/lib/python3.12/site-packages/aiofiles/base.py", line 98, in __aenter__
    self._obj = await self._coro
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiofiles/threadpool/__init__.py", line 94, in _open
    f = yield from loop.run_in_executor(executor, cb)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/homeassistant/util/loop.py", line 131, in protected_loop_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: 'solcast-sites.json'

2024-06-18 16:11:59.154 ERROR (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - sites_usage error: Traceback (most recent call last):
  File "/config/custom_components/solcast_solar/solcastapi.py", line 215, in sites_usage
    async with aiofiles.open(apiCacheFileName, 'w') as f:
  File "/usr/local/lib/python3.12/site-packages/aiofiles/base.py", line 98, in __aenter__
    self._obj = await self._coro
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiofiles/threadpool/__init__.py", line 94, in _open
    f = yield from loop.run_in_executor(executor, cb)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/homeassistant/util/loop.py", line 131, in protected_loop_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: 'solcast-usage.json'

2024-06-18 16:12:28.103 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_middlewares.py", line 114, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/security_filter.py", line 92, in security_filter_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/forwarded.py", line 83, in forwarded_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/request_context.py", line 26, in request_context_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/ban.py", line 85, in ban_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/auth.py", line 242, in auth_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/homeassistant/components/http/headers.py", line 32, in headers_middleware
    response = await handler(request)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/homeassistant/helpers/http.py", line 73, in handle
    result = await handler(request, **request.match_info)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/homeassistant/components/diagnostics/__init__.py", line 280, in get
    data = await info.config_entry_diagnostics(hass, config_entry)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/solcast_solar/diagnostics.py", line 32, in async_get_config_entry_diagnostics
    "energy_forecasts_graph": coordinator.solcast._dataenergy["wh_hours"],
                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'wh_hours'

What do we know?

A brand new pull and set up of linuxserver/homeassistant and a brand new manual install of solcast repros.

It looks like there is a permissions problems - possibly with aiofiles.open for write. I note that linuxserver notes that: Rebase to alpine 3.19, upgrade to python 3.12. Restructure python packages so all packages are internal (no more venv in /config). Use upstream project's wheels. Due to an upstream issue, on first start of the newly updated container, some custom integrations may be disabled in HA due to missing dependencies. A subsequent container restart should fix that and the integrations should be re-enabled.

@florie1706 - if you can list any other integrations that you're using that are successfully writing config files to your config folder, that might help so we can see what they're doing.

[autoSteve]

I think I have some good news, @BJReplay and @florie1706.

I changed the solcastapi.py to use /options/[filenames]... and the permission errors went away.

It is likely that the file "solcast.json" will be similarly affected for linuxserver instances.

Why? Maybe the change to use aiofiles. Of that I am not certain, but it seems the most likely explanation. 🤷

I cannot be certain, because I have well busted through my API limit for today, because other testing, so solcast.json will not get created for my linuxserver install. But I bet that fails, too.

I will craft a commit for crash-testing.

[autoSteve]

Pull request will be submitted after a successful test, @florie1706.

Commit https://github.com/BJReplay/ha-solcast-solar/commit/2e8d961ef16ca1f1e3a638ebf0e01bb06a0a2c39 adds the /config/ path.

There are two changed files. __init__.py and solcastapi.py.

[autoSteve]

Note that I appear to have crossed some streams here. Still trying to work out how it happened, but 2e8d961 contains some other fixes that I'm actively testing in other branches. They seem fine so far, so test away. I blame GitHub desktop...

[florie1706]

@autoSteve that did the trick, working as expected now :) Thank you so much.

[autoSteve]

I'll almost certainly release this in 4.0.32-beta1, @florie1706.

Because why? Because I am an idiot and included it with other changes that got wrapped in the same pull request. Like adjustments to handle having multiple accounts (shouldn't be a thing because Solcast T&C, so shhh! don't ask...), and prevention of inadvertent frequent Solcast forecast calls (weird happenstance, don't ask...), and handling of "tally" variable issues (don't ask, I hope I've got that one...).

Ugh. It is what it is.

Watch out for that beta+, but given that you manual install it likely won't impact you unless you update in future. But by then there'll be a release that covers this corner case for Docker.

[autoSteve]

I may regret this, but closing. It seems sorted for now -ish, well in the next release...