Sequential scan - Githubissues

sidlgor commented 1 week ago

about: when testing a bit more i found this one: ERROR:velbus-module:channel 96 does not exist for module @ address <None type:82 address:20 loaded:True

I gues this is a spontaneous temperature update when the module (submodule) is not yet completely loaded. This is possible during the scan procedure depending on timing but should never happen again after scan completion. I just changed the log to info instead of error. The error does not occur in my setup so it is hard to further investigate ... to be watched.

Cereal2nd commented 1 week ago

can you run pre-commit on this one?

then the ci should pass

sidlgor commented 1 week ago

How do I run pre-commit?? Lost with git ☹️

Btw: there was still a missing update in controller: signature of add_submodules should be def add_submodules(self, module: Module, subList: dict[int, int]) -> None: #signature changed to module instead of int address

------ Original Message ------ From "Maikel Punie" @.> To "Cereal2nd/velbus-aio" @.> Cc "sidlgor" @.>; "Author" @.> Date 27/06/2024 12:38:07 Subject Re: [Cereal2nd/velbus-aio] Sequential scan (PR #112)

can you run pre-commit on this one?

then the ci should pass

— Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#issuecomment-2194350980, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBL57UGMQMOO6SUHZHITYTZJPTQ7AVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJUGM2TAOJYGA. You are receiving this because you authored the thread.Message ID: @.***>

Cereal2nd commented 1 week ago

if you give me push access to your fork i'll do it

sidlgor commented 1 week ago

To my knowledge this is no longer required with the async scan If it's still needed it is easy to add at the end of the async scan procedure. Access to fork ok?

------ Original Message ------ From "Maikel Punie" @.> To "Cereal2nd/velbus-aio" @.> Cc "sidlgor" @.>; "Author" @.> Date 27/06/2024 13:46:15 Subject Re: [Cereal2nd/velbus-aio] Sequential scan (PR #112)

@Cereal2nd commented on this pull request.

In velbusaio/controller.py https://github.com/Cereal2nd/velbus-aio/pull/112#discussion_r1656986159:

@@ -70,7 +69,7 @@ def _on_end_of_scan(self) -> None: """Notify the scan failure.""" self._handler.scan_finished() This seems to not work anymore, is this still required?

— Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#pullrequestreview-2145115320, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBL57SYGUC3EZZ52FZTUV3ZJP3QPAVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDCNBVGEYTKMZSGA. You are receiving this because you authored the thread.Message ID: @.***>

Cereal2nd commented 1 week ago

this all starts to look good, i want to investigate one more thing:

the module 75 and 201 is always reloaded after a cached restart, once this is fixed i'll merge it.

INFO:velbus:Found module: type:57 address:254
INFO:velbus-module:Load Module
DEBUG:velbus-packet:Scan complete
WARNING:velbus:Waiting for module 75
WARNING:velbus:Waiting for module 201
INFO:velbus:Not all modules loaded yet, waiting 15 seconds

sidlgor commented 1 week ago

this all starts to look good, i want to investigate one more thing: Thanks a lot for your time ... and sorry for my newbie questions. The environment (Debian, HAS, docker, python, git, ...) was (is) a steep learning curve for me the module 75 and 201 is always reloaded after a cached restart, once this is fixed i'll merge it.

I guess those are module addresses (there is no such type as 201). I don't think absolute address numbers matter (I have a bls1 at address 201 which loads fine). Double address or sub address definition?

------ Original Message ------ From "Maikel Punie" @.> To "Cereal2nd/velbus-aio" @.> Cc "sidlgor" @.>; "Author" @.> Date 27/06/2024 15:40:11 Subject Re: [Cereal2nd/velbus-aio] Sequential scan (PR #112)

this all starts to look good, i want to investigate one more thing:

the module 75 and 201 is always reloaded after a cached restart, once this is fixed i'll merge it.

INFO:velbus:Found module: type:57 address:254 INFO:velbus-module:Load Module DEBUG:velbus-packet:Scan complete WARNING:velbus:Waiting for module 75 WARNING:velbus:Waiting for module 201 INFO:velbus:Not all modules loaded yet, waiting 15 seconds — Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#issuecomment-2194719315, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBL57WDTGYDVTQKQLSZ7QLZJQI3XAVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJUG4YTSMZRGU. You are receiving this because you authored the thread.Message ID: @.***>

Cereal2nd commented 1 week ago

i just debugged this and it has to do with the protocol.json for that module type

sidlgor commented 6 days ago

Great news! Next steps are publication on pyPI and PIP INSTALL or can I already install from master (for use in has) ?

sidlgor commented 6 days ago

Fork sidlgor removed

------ Original Message ------ From "Maikel Punie" @.> To "Cereal2nd/velbus-aio" @.> Cc "sidlgor" @.>; "Author" @.> Date 27/06/2024 19:26:39 Subject Re: [Cereal2nd/velbus-aio] Sequential scan (PR #112)

Merged #112 https://github.com/Cereal2nd/velbus-aio/pull/112 into master.

— Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#event-13320891718, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBL57VSWBBS3MNHHA37ZRDZJRDM7AVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJTGMZDAOBZGE3TCOA. You are receiving this because you authored the thread.Message ID: @.***>

Cereal2nd commented 6 days ago

do we want to enable the usecache.yes file by default?

sidlgor commented 6 days ago

Thought about it too, but maybe it's not a good idea to enable by default. At initial start with a brand new system there will be no cache and no detected modules. UseCache seems more useful once you got a stable configuration but somewhere we have to tell users about the option. I did describe it shortly in my forked readme but just deleted the fork (no problem to do it again). Having no default cache will have the disadvantage of a slow start but at least it will work. There is also an info log message at the start of the "scan (log.info(f"Start module scan, reload cache {reload_cache}")"

Maybe, on the long term, it might be an idea to make it an additional startup parameter like the connection string?

------ Original Message ------ From "Maikel Punie" @.> To "Cereal2nd/velbus-aio" @.> Cc "sidlgor" @.>; "Author" @.> Date 27/06/2024 20:25:35 Subject Re: [Cereal2nd/velbus-aio] Sequential scan (PR #112)

do we want to enable the usecache.yes file by default?

— Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#issuecomment-2195416258, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBL57RHPOD7P4BCXK6MI3DZJRKJ7AVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJVGQYTMMRVHA. You are receiving this because you authored the thread.Message ID: @.***>

niobos commented 6 days ago

As a user of Home Assistant, I would expect the Velbus integration to "just work". So needing to change a parameter when I add a module would not be my preferred strategy.

What may be feasible to implement, is a kind of auto-discovery: When there is no module in cache (or no cache), we de a full scan. But once we have a cache, we don't scan at startup. Instead, we listen for messages from/to addresses that are unknown to us. For each address that we encounter, we initiate a targeted scan to learn about the module at that address.

That way we can have a fast startup with the cached information, and still discover newly added modules by simply seeing them on the bus.

To "forget" old modules, we could do a quick "ping" on startup to see if the modules still respond to an RTR request.

Cereal2nd commented 6 days ago

i just found a bigger problem:

after the intial scan it seems the bus is not read anymore, so the system stall

sidlgor commented 6 days ago

Some thoughts about "just works" auto discovery. At first this might be hard to implement: I refer to the link https://forum.velbus.eu/t/velbus-home-assistant-missing-module-instances/17402/3 where cereal notes "sugestion 3 is hard to implement, as we wait until the loading is complete to advertise the entities to hass" At second thought: do we really want that. Do we really want our configuration to change on the fly when a device get's on or offline. For example I have an installation with different segments with separate power supplies. Do I want those devices auto removed when a segment is temporarely down (anyway I am glad velbuslink does not do that) To my preference, I like to be in control. I know when my installation is stable or has been modified and I like it to choose that moment to scan and verify.

sidlgor commented 6 days ago

I had a quick look at the code. When you look at the code, the messages are passed to module.on_message in handler.py line 198. This is only done when test if commandRegistry.has_command(int(command_value), module_type) is True. I am not sure if this is correct. It would be nice to have a trace there. I cannot test this with load_modules because code stops after scan

------ Original Message ------ From "Maikel Punie" @.> To "Cereal2nd/velbus-aio" @.> Cc "sidlgor" @.>; "Author" @.> Date 28/06/2024 08:46:03 Subject Re: [Cereal2nd/velbus-aio] Sequential scan (PR #112)

i just found a bigger problem:

after the intial scan it seems the bus is not read anymore, so the system stall — Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#issuecomment-2196252591, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBL57Q7H2FDUYBMR7GWGF3ZJUBCXAVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJWGI2TENJZGE. You are receiving this because you authored the thread.Message ID: @.***>

Cereal2nd commented 6 days ago

you can use read_bus

this keeps running after the scan and this one is for 90% the same as what hass does

On Fri, Jun 28, 2024 at 9:19 AM sidlgor @.***> wrote:

I had a quick look at the code. When you look at the code, the messages are passed to module.on_message in handler.py line 198. This is only done when test if commandRegistry.has_command(int(command_value), module_type) is True. I am not sure if this is correct. It would be nice to have a trace there. I cannot test this with load_modules because code stops after scan

------ Original Message ------ From "Maikel Punie" @.> To "Cereal2nd/velbus-aio" @.> Cc "sidlgor" @.>; "Author" @.> Date 28/06/2024 08:46:03 Subject Re: [Cereal2nd/velbus-aio] Sequential scan (PR #112)

i just found a bigger problem:

after the intial scan it seems the bus is not read anymore, so the system stall — Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#issuecomment-2196252591,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/ACBL57Q7H2FDUYBMR7GWGF3ZJUBCXAVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJWGI2TENJZGE>.

You are receiving this because you authored the thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#issuecomment-2196296314, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4LF4MMIV6AYI53LOYXI6LZJUE6LAVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJWGI4TMMZRGQ . You are receiving this because you modified the open/close state.Message ID: @.***>

sidlgor commented 6 days ago

ok, I have to setup my dev env again, just deleted about everything of my test fork :-(

sidlgor commented 6 days ago

Again just struggling to setup my dev environment (even reverted to full restore of PI backup but also that didn't work any more because my git fork is deleted)

I reinstalled from git with commands below

sudo git init sudo git clone https://github.com/Cereal2nd/velbus-aio sudo python -m venv /hass/velbus-aio source /hass/velbus-aio/bin/activate sudo chmod -R a+rw velbus-aio

Then when I try to debug an example (fe load_modules) I get a ModuleNotFoundError from the moment I call Velbus code outside the example (seems like path is not found)

Don't know what I am doing wrong. I got this working with the fork but it seems stupid to create a new fork. Please help me setup dev environment the right way.

Cereal2nd commented 6 days ago

add a pip install -e . after the activation of the venv and before you run the example.

On the use-cache file i think we should do the following:

enable this by default
make a param to the scan function to disable this
if we trigger a scan from hass disable the usecache

this give us the best of 2 worlds:

fast startup
once a user modifies his bus he should run a scan from the hass GUI

sidlgor commented 6 days ago

add a pip install -e . after the activation of the venv and before you run the example. Works fine now ... I have to leave now ... but will debug this first priority tomorrow

Concerning use-cache:

enabling by default seems best in most cases, however it is not clear what you mean by "if we trigger a scan from hass". Does this mean every manual restart of hass or an explicit restart of velbus integration. If you mean "every restart", disabling the cache might be annoying (automation scripts are modified much more often then a velbus config)
I would prefer a parameter to the scan function over the presence of a file. I just didn't know how to pass the parameter from hass :-)

sidlgor commented 5 days ago

Tested message passing to modules and found one concurrency error I replaced line 198 in handler (DICTIONARY OF MODULES SHOULD NOT BE ACCESSED OUTSIDE LOCKED SECTION) OLD CODE: await self._velbus.get_module(msg.address).on_message(msg) NEW CODE: await module.on_message(msg)

I also added 2 debug logs for testing and attached a log of the message passing on my installation 1e at entry of handle procedure self._log.debug(f"Enter handle message address={rawmsg.address}, command={rawmsg.command}") 2e at line 198 self._log.debug(f"Pass message {type(msg)} to module type/name/address {module_type},{module._name},{module._address}") My log: velbus.log

Otherwise I found no issues (tested after code correction). Not sure if this caused the problem (always hard to tell with concurrent code). Unlocked access to a dictionary is always dangerous and might corrupt the dictionary at some point. I just tested to the point of the module.on_message call because nothing changed further

Just one other suggestion: I like having a timestamp in the logging. Could this be added to the examples? logging.basicConfig(stream=sys.stdout, level=logging.DEBUG, style='{', datefmt='%Y-%m-%d %H:%M:%S', format='{asctime} {message}')

Cereal2nd commented 5 days ago

both changes added.

on the cache: in hass we have a service to rescan the bus.

so my suggestion:

always use the fast cache (so no bus scan if the cache is available)
if the user changes something on the bus, he needs to trigger a scan from hass (service: "Velbus: scan")

sidlgor commented 5 days ago

Maybe one more suggestion: it would be nice to have a full scan on initial start (I am thinking about people that install velbus integration for the first time). In the current implementation (with the file usecache.yes) this could be easily achieved by creating the usecache.yes file at the end of a successful scan.

Whatever you decide ... just tell me if there is something I need / can do.

------ Original Message ------ From "Maikel Punie" @.> To "Cereal2nd/velbus-aio" @.> Cc "sidlgor" @.>; "Author" @.> Date 29/06/2024 13:47:58 Subject Re: [Cereal2nd/velbus-aio] Sequential scan (PR #112)

both changes added.

on the cache: in hass we have a service to rescan the bus.

so my suggestion:

always use the fast cache (so no bus scan if the cache is available) if the user changes something on the bus, he needs to trigger a scan from hass (service: "Velbus: scan") — Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#issuecomment-2198123179, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBL57W5YMNUK62DL2LU6QTZJ2NG5AVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJYGEZDGMJXHE. You are receiving this because you authored the thread.Message ID: @.***>

Cereal2nd commented 5 days ago

its implemented like above, but if the cachdir has 0 files, we force the reload_cache to True

sidlgor commented 5 days ago

👍

------ Original Message ------ From "Maikel Punie" @.> To "Cereal2nd/velbus-aio" @.> Cc "sidlgor" @.>; "Author" @.> Date 29/06/2024 16:08:50 Subject Re: [Cereal2nd/velbus-aio] Sequential scan (PR #112)

its implemented like above, but if the cachdir has 0 files, we force the reload_cache to True

— Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#issuecomment-2198207628, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBL57TRSNFM4MTUR2UZ4ITZJ25XFAVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJYGIYDONRSHA. You are receiving this because you authored the thread.Message ID: @.***>

sidlgor commented 3 days ago

Hi, seems like a stupid quested, but how do I install new release (2024.7.0) so that HAS uses it?

Cereal2nd commented 3 days ago

modify the manifest of the velbus integration:

https://github.com/home-assistant/core/blob/dev/homeassistant/components/velbus/manifest.json#L16

update that file, to the new release and then restart hass

Cereal2nd commented 3 days ago

maybe its better to move this to an email conversation :) or discord

sidlgor commented 2 days ago

Sorry, but I don's understand what you mean. Where should I find that velbus/manifest.json file.

Do I have to install the velbus integration that comes standard with HAS first? (I did that, but as expected that's an older version). Tried to locate velbus manifest.json but did not find anything refering velbus. I guess this is because it is an integrated and not a custom integration ? Finally removed standard velbus integration

Deactivated and removed previous debug version of velbus-aio and installed the latest version (2024.7.0) from sources sudo git init sudo git clone https://github.com/Cereal2nd/velbus-aio sudo python -m venv /hass/velbus-aio source /hass/velbus-aio/bin/activate sudo chmod -R a+rw velbus-aio cd velbus-aio pip install -e .

Package install seems ok but I can't debug ! Successfully installed velbus-aio-2024.7.0 (velbus-aio) lgor@has:/hass/velbus-aio/velbus-aio $ pip list Package Version Editable project location

backoff 2.2.1 pip 23.0.1 pyserial 3.5 pyserial-asyncio-fast 0.12 setuptools 66.1.1 velbus-aio 2024.7.0 /hass/velbus-aio/velbus-aio (velbus-aio) lgor@has:/hass/velbus-aio/velbus-aio $ ^C

Even if I could debug, HAS will still not load velbus-aio. Should I install velbus somewhere under custom_components? How to? Where comes pip install velbus-aio from pypi comes in? (By the way) I can only execute pip commands when python virtual env is created.

I am hitting exactly the same problems then when I tried to load my fork into HAS (before moving to master). This is definately because I miss background on setting up this environment. Please help!

sidlgor commented 2 days ago

Tried to clone files from home-assistant/core into custom_components (don't know if I need to do this, and if this is the proper way) but anyway: getting the files failed with not found!

git clone https://github.com/home-assistant/core/blob/dev/homeassistant/components/velbus Cloning into 'velbus'... fatal: repository 'https://github.com/home-assistant/core/blob/dev/homeassistant/components/velbus/' not found

Cereal2nd commented 2 days ago

How is haas Running?

It's it hassio?

On Mon, 1 Jul 2024, 23:17 sidlgor, @.***> wrote:

Tried to clone files from home-assistant/core into custom_components (don't know if I need to do this, and if this is the proper way) but anyway: getting the files failed with not found!

git clone https://github.com/home-assistant/core/blob/dev/homeassistant/components/velbus Cloning into 'velbus'... fatal: repository ' https://github.com/home-assistant/core/blob/dev/homeassistant/components/velbus/' not found

— Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#issuecomment-2201089308, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4LF4JUV4OEBV3F37FP2HLZKHBPLAVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBRGA4DSMZQHA . You are receiving this because you modified the open/close state.Message ID: @.***>

Cereal2nd commented 2 days ago

Sorry, but I don's understand what you mean. Where should I find that velbus/manifest.json file.

Do I have to install the velbus integration that comes standard with HAS first? (I did that, but as expected that's an older version). Tried to locate velbus manifest.json but did not find anything refering velbus. I guess this is because it is an integrated and not a custom integration ? Finally removed standard velbus integration

Deactivated and removed previous debug version of velbus-aio and installed the latest version (2024.7.0) from sources sudo git init sudo git clone https://github.com/Cereal2nd/velbus-aio sudo python -m venv /hass/velbus-aio source /hass/velbus-aio/bin/activate sudo chmod -R a+rw velbus-aio cd velbus-aio pip install -e .

Package install seems ok but I can't debug ! Successfully installed velbus-aio-2024.7.0 (velbus-aio) lgor@has:/hass/velbus-aio/velbus-aio $ pip list Package Version Editable project location

backoff 2.2.1 pip 23.0.1 pyserial 3.5 pyserial-asyncio-fast 0.12 setuptools 66.1.1 velbus-aio 2024.7.0 /hass/velbus-aio/velbus-aio (velbus-aio) lgor@has:/hass/velbus-aio/velbus-aio $ ^C

Even if I could debug, HAS will still not load velbus-aio. Should I install velbus somewhere under custom_components? How to? Where comes pip install velbus-aio from pypi comes in? (By the way) I can only execute pip commands when python virtual env is created.

I am hitting exactly the same problems then when I tried to load my fork into HAS (before moving to master). This is definately because I miss background on setting up this environment. Please help!

aiofile is a new dependancy ,,,,

sidlgor commented 2 days ago

No HAS Docker

------ Original Message ------ From "Maikel Punie" @.> To "Cereal2nd/velbus-aio" @.> Cc "sidlgor" @.>; "Author" @.> Date 02/07/2024 05:13:38 Subject Re: [Cereal2nd/velbus-aio] Sequential scan (PR #112)

How is haas Running?

It's it hassio?

On Mon, 1 Jul 2024, 23:17 sidlgor, @.***> wrote:

Tried to clone files from home-assistant/core into custom_components (don't know if I need to do this, and if this is the proper way) but anyway: getting the files failed with not found!

git clone

https://github.com/home-assistant/core/blob/dev/homeassistant/components/velbus Cloning into 'velbus'... fatal: repository '

https://github.com/home-assistant/core/blob/dev/homeassistant/components/velbus/' not found

— Reply to this email directly, view it on GitHub

https://github.com/Cereal2nd/velbus-aio/pull/112#issuecomment-2201089308, or unsubscribe

https://github.com/notifications/unsubscribe-auth/AA4LF4JUV4OEBV3F37FP2HLZKHBPLAVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBRGA4DSMZQHA . You are receiving this because you modified the open/close state.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#issuecomment-2201784616, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBL57SQWOLHJ77R7TFH2ELZKILGFAVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBRG44DINRRGY. You are receiving this because you authored the thread.Message ID: @.***>

sidlgor commented 2 days ago

Do I have to solve new dependency to aiofile manually. How to? What is it?

Cereal2nd commented 2 days ago

then you have to modify it inside the docker container.

I don't know what the correct path is ....

On Tue, Jul 2, 2024 at 8:29 AM sidlgor @.***> wrote:

Do I have to solve new dependency to aiofile manually. How to? What is it?

— Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#issuecomment-2202050243, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4LF4M4EHCRIVWM6M2UBDLZKJCELAVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBSGA2TAMRUGM . You are receiving this because you modified the open/close state.Message ID: @.***>

sidlgor commented 2 days ago

Confused: I am not running debug code in a container. I just run HAS and some other integrations inside a container. Are this not two seperate problems: 1e solving dependency to aiofile (async file io ?), 2e make HAS (docker) load velbus-aio 1e: do I need to solve the new dependy manual? How to? I didn'have to solve other dependencies manually (serial) 2e: from prior dialog with niobis (running HAS docker too) I understood I have to copy the files from https://github.com/home-assistant/core/tree/dev/homeassistant/components/velbus to custom_components/velbus. This seems to be consistent with changes that have to be made to the manifest.json to reference the correct velbus-aio packet version.

So I am still stuck with two problems:

How to solve the dependency to aoifile (even just for debugging)
How to copy the files from https://github.com/home-assistant/core/tree/dev/homeassistant/components/velbus to custom_components/velbus. (Tried git clone but that didn't work)

One last question just for my info: newest packet version looks installed on pypi but is nowere used (pip install velbus-aio from pypi). Where does pypi fits in ??

Cereal2nd commented 2 days ago

if you work with a custom component you will have to solve the dependency manually, a simple pip install will do it.

i never worked with a custom_component, so no idea here. log in to the docker container:

pip install aiofile
modify the: /usr/src/homeassistant/homeassistant/components/velbus/manifest.json file

On Tue, Jul 2, 2024 at 9:40 AM sidlgor @.***> wrote:

Confused: I am not running debug code in a container. I just run HAS and some other integrations inside a container. Are this not two seperate problems: 1e solving dependency to aiofile (async file io ?), 2e make HAS (docker) load velbus-aio 1e: do I need to solve the new dependy manual? How to? I didn'have to solve other dependencies manually (serial) 2e: from prior dialog with niobis (running HAS docker too) I understood I have to copy the files from https://github.com/home-assistant/core/tree/dev/homeassistant/components/velbus to custom_components/velbus. This seems to be consistent with changes that have to be made to the manifest.json to reference the correct velbus-aio packet version.

So I am still stuck with two problems:

How to solve the dependency to aoifile (even just for debugging)

How to copy the files from https://github.com/home-assistant/core/tree/dev/homeassistant/components/velbus to custom_components/velbus. (Tried git clone but that didn't work)

One last question just for my info: newest packet version looks installed on pypi but is nowere used (pip install velbus-aio from pypi). Where does pypi fits in ??

— Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#issuecomment-2202189138, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4LF4L7H54YGWTLCU7ABH3ZKJKQVAVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBSGE4DSMJTHA . You are receiving this because you modified the open/close state.Message ID: @.***>

sidlgor commented 2 days ago

Finally I got it working just by modifying /usr/src/homeassistant/homeassistant/components/velbus/manifest.json within the container. Still there seems to go something wrong (docker version ?). Velbus always does a full scan. I cannot locate the cache files. They are not created at /home//.velbuscache which could be logical but I also cannot locate them in the container. Also some module names are not read (default module type names). Any thoughts ? Everything runs fine when debugging read_bus which of course does not happen in a container

Cereal2nd commented 2 days ago

the cache should be inside the the config directory of hass

/config/.storage/velbuscache/....

On Tue, Jul 2, 2024 at 1:24 PM sidlgor @.***> wrote:

Finally I got it working just by modifying /usr/src/homeassistant/homeassistant/components/velbus/manifest.json within the container. Still there seems to go something wrong (docker version ?). Velbus always does a full scan. I cannot locate the cache files. They are not created at /home//.velbuscache which could be logical but I also cannot locate them in the container. Also some module names are not read (default module type names). Any thoughts ? Everything runs fine when debugging read_bus which of course does not happen in a container

— Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#issuecomment-2202842687, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4LF4PMCWXB55E6ZG5YMEDZKKEU7AVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBSHA2DENRYG4 . You are receiving this because you modified the open/close state.Message ID: @.***>

sidlgor commented 2 days ago

Velbuscache is there but folder name ornamented with some guid? ... but the usecache.yes file ain't there

has:/config/.storage/velbuscache-fd4bfbbd7f96ebdced5e63ebf8f93494# ls 1.json 106.json 114.json 121.json 13.json 134.json 139.json 163.json 170.json 201.json 206.json 5.json 54.json 71.json 10.json 107.json 116.json 124.json 130.json 135.json 14.json 164.json 18.json 202.json 207.json 50.json 56.json 78.json 100.json 108.json 118.json 125.json 131.json 136.json 15.json 168.json 2.json 203.json 208.json 51.json 6.json 8.json 104.json 11.json 12.json 126.json 132.json 137.json 16.json 169.json 20.json 204.json 3.json 52.json 7.json 9.json 105.json 112.json 120.json 127.json 133.json 138.json 162.json 17.json 200.json 205.json 4.json 53.json 70.json has:/config/.storage/velbuscache-fd4bfbbd7f96ebdced5e63ebf8f93494#

Cereal2nd commented 2 days ago

yeah this is the uuid of the integration.

the usecache file is not used anymore see some previous comments

On Tue, Jul 2, 2024 at 1:51 PM sidlgor @.***> wrote:

Velbuscache is there but folder name ornamented with some guid? ... but the usecache.yes file ain't there

has:/config/.storage/velbuscache-fd4bfbbd7f96ebdced5e63ebf8f93494# ls 1.json 106.json 114.json 121.json 13.json 134.json 139.json 163.json 170.json 201.json 206.json 5.json 54.json 71.json 10.json 107.json 116.json 124.json 130.json 135.json 14.json 164.json 18.json 202.json 207.json 50.json 56.json 78.json 100.json 108.json 118.json 125.json 131.json 136.json 15.json 168.json 2.json 203.json 208.json 51.json 6.json 8.json 104.json 11.json 12.json 126.json 132.json 137.json 16.json 169.json 20.json 204.json 3.json 52.json 7.json 9.json 105.json 112.json 120.json 127.json 133.json 138.json 162.json 17.json 200.json 205.json 4.json 53.json 70.json has:/config/.storage/velbuscache-fd4bfbbd7f96ebdced5e63ebf8f93494#

— Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#issuecomment-2202964365, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4LF4KZLHE3UTWG4W4KNU3ZKKH4LAVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBSHE3DIMZWGU . You are receiving this because you modified the open/close state.Message ID: @.***>

sidlgor commented 2 days ago

Missed that in latest implementation you no longer use usecache.yes to detect a valid cache (I liked the idea to create the file at the end of the scan because that's the point we are really sure a full scan has run to completion with success). Anyway in read_bus cache detection is ok but in HAS docker the detection of the cache seems to fail. You can see from log below (made in parallel with velbuslink) that HAS is always doing a full scan.

sidlgor commented 2 days ago

Just some more thinking. Validating the cache at the end of a successful scan seems important. Because a first full scan takes a long time the risk is considerable the user interrupts the scan. This will leave a partial (and possible invalid) cache which will be detected by next start. As a plus, implementation is a lot simpler and does not need asynchronous file scan :-) The detection of at least 1 module can be an additional condition for marking a succesful scan

sidlgor commented 2 days ago

I would like to propose to get back to the parameterless scan procedure and the use of the file usecache.yes for following reasons: validating the cache in a persistent way only at the end of full successful scan makes the code far more robust against HAS or docker container restarts code is simple and does not need async file scan usecache is only auto created at the end of a successful scan and never deleted, resetting the cache can achieved by simply deleting the file usecache.yes manual or as part of a service for some unknown reason the current detection of the cache seems to fail in HAS docker Code changes are very simple and only at start and end of scan procedure

 asyncdef scan(self) -> None:
     cFileUseCache = pathlib.Path(f"{get_cache_dir()}/usecache.yes")
     reload_cache = not os.path.isfile(cFileUseCache)

... just keep existing code and just finish with validating the cache at the end of the scan

     countModules = len(self._velbus.get_modules())
     if reload_cache and countModules > 0:
         os.mknod(cFileUseCache)
         self._log.info("Full scan completed: validating cache")
     self._log.info(f"Scan completed: #{countModules} modules")
     self._scan_complete = True

Message ID: @.***>

sidlgor commented 1 day ago

Do you have already an idea why the cache files are not detected in HAS docker ? What do you think about my proposol ? It's an easy implementation, it's robust for restarts, and some kind of transactional. It's already tested with read_bus but I can't load it in HAS docker (I think I finally found out the role of pypi, HAS auto loads the package version from there, so I would need to publish a new version. Is this correct ?)

Anyway, my configuration is now loading fine (al be it always with a full scan). I can finally start using velbus in HAS :-)

Cereal2nd commented 1 day ago

we can not simply get to this code, at least we need to be able to force a scan even when this file is there. There is a methode in hass to force a scan, triggerd by the user, and this still needs to work.

i'll try to look at it today, but can't promise

Cereal2nd commented 1 day ago

try this one: https://github.com/Cereal2nd/velbus-aio/releases/tag/2024.7.1

its a stupid mistake, the get_cache_dir should only be used to set a default path, its not correct inside hass

sidlgor commented 1 day ago

Works fine now 👌🥂😉

Thx !!!!!!!!!

------ Original Message ------ From "Maikel Punie" @.> To "Cereal2nd/velbus-aio" @.> Cc "sidlgor" @.>; "Author" @.> Date 03/07/2024 13:05:07 Subject Re: [Cereal2nd/velbus-aio] Sequential scan (PR #112)

try this one: https://github.com/Cereal2nd/velbus-aio/releases/tag/2024.7.1

its a stupid mistake, the get_cache_dir should only be used to set a default path, its not correct inside hass

— Reply to this email directly, view it on GitHub https://github.com/Cereal2nd/velbus-aio/pull/112#issuecomment-2205812748, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBL57QGHXZ3L7RKVOMAYQ3ZKPLGHAVCNFSM6AAAAABJ7S6SLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBVHAYTENZUHA. You are receiving this because you authored the thread.Message ID: @.***>

Cereal2nd / velbus-aio

Sequential scan #112