Introduce Dreame map refresh

[auanasgheps]

Description

The current implementation for Dreame vacuums is not able to refresh the map on its own, but relies on the mobile app to do so.

This is the detailed behaviour.

• The HA Map card will never update unless you open the mobile app.
• Once the user opens the mobile app, the HA Map will update once.
• Even if you keep the app open, the HA Map doesn't refresh even if the app does update.
• To trigger a refresh, you have to exit the vacuum on the app and re-enter again to trigger another single refresh.

Unfortunately it means we are still heavily dependent by the mobile app.

I have grabbed the Mobile app's log to see what commands are send to the vacuum. I see a lot of calls regarding "get_properties" and these are MioT commands.

I can share the logs privately with you if you provide an email address.

Solution

When vacuum operations are started, implement a logic similar to the one in the app, so we can get the map refreshed independently.

Alternatives

No response

Context

Other non-Dreame vacuums are able to refresh the map without relying on the app.

[PiotrMachowski]

Sure, you can send me log files. piotr.machowski.dev [at] gmail.com

[Tasshack]

I have the exact same issue with my Z10 Pro. I think the app sends map-req or some other action for device to update the map on the cloud. I can provide more information if you need to.

[SpiGAndromeda]

I could also provide logs from a Dreame D9. I let it clean one room while watching the app and retrieved the logs. Should I send them via mail? Is there anything to remove from the logs beside the serial number?

The logs contain some undocumented PIIDs. There were no update-map actions and only some map-req actions at the beginning. The mult-map-state was request via get_properties very often. The mult-map-info sometimes (which seems to return an object).

By the way: maybe you could put the link to vevs Mi Home App into a 'Development' section of the Readme. It took me a while to find out how to get logs from the app. (or a pinned issue)

[PiotrMachowski]

@SpiGAndromeda sure, you can send them. The problem is that I don't have a Dreame vacuum, so I'm unable to test anything :/

[SpiGAndromeda]

I can test if for you. :) We can discuss the communication way via mail if you want.

[PiotrMachowski]

@SpiGAndromeda I think that the best way of communication would be HA forum, I can make a group conversation here to add everyone else as well https://community.home-assistant.io/u/3_14

[SpiGAndromeda]

Perfect. My username is the same like here.

[auanasgheps]

Allright! My username in HA forum is the same, auanasgheps.

[Tasshack]

I fixed the map refresh problem by calling map-req action periodically with an automation while robot is cleaning the house. I don't know this is the right solution, but it works good for now.

[auanasgheps]

Thanks for the update @Tasshack, can you provide more details?

[Tasshack]

Gen2 Dreame robots (L10, Z10, W10) works different from the Gen1 like D9. New robots does not send map periodically to the cloud automatically. I have checked the valetudo firmware, found out that you need to send map-req action to trigger the map upload to cloud and that is how currently valetudo gets the map from the device.

https://github.com/Hypfer/Valetudo/blob/84de0ef9d869f5150c504fa13bcd49956c71bf4c/backend/lib/robots/dreame/DreameValetudoRobot.js#L54

I have custom miiot integration installed on my hass and that is how i send the map-req to the device currently.

service: xiaomi_miot.call_action
target:
  entity_id: vacuum.dreame.p2028
data:      
  siid: 6
  aiid: 1
  params:
    - piid: 2
      value: '{"frame_type":"I"}'

[PiotrMachowski]

@auanasgheps hmmm, this was my suggestion when you sent me your logs, but it didn't work for you, right?

[Tasshack]

@auanasgheps can you share the logs with me as well to ayigittopcu [at] gmail.com?

[Tasshack]

@PiotrMachowski my username in comminity is also same with here, can you add me to the conversation from there too.

[auanasgheps]

My query was slightly different:

service: xiaomi_miot.call_action
data:
  entity_id: vacuum.battista
  siid: 6
  aiid: 1
  params:
  piid: 2
    value: '{"req_type":1,"frame_type":"I","force_type":1}'

I can confirm that the new query by @Tasshack works, the map gets updated just after a second the command is sent! That's great!

So shall we implement this call (or the template to do so, not all Dreame need it according to Tasshack) every time the maps needs an update?

[Tasshack]

By the way i am trying to develop a custom integration (forked from D9 intregration) specific to those GEN2 robots but most of their new features that introduced on latest firmware are not documented at all. Even valetudo does not implemeted some of those features since it is not currently supported on latest firmware. I am struggling to find the specific siid/piid values with their extra parameter information to send to robot. For example; app can send cleaning room order to robot to store and when you press start even from the button, robot cleans whole house with that room order, but i couldn't find the aiid/piid values to do it from the integration. The main problem is most of the developers still does not have one of those new GEN2 robots yet and because of that currenlty there is no information about them on the internet and how they work exactly. I think we may have to wait a bit longer to fully integrate GEN2 robots to home assistant because it has been more than six months from release of the latest firmware and people still did not exploited the new features on Z10/L10 like only mopping while water tank is attached which is only loosely documented on W10 api specs.

[PiotrMachowski]

@auanasgheps it shouldn't be hard for me to add it. Until that you can handle it with a simple automation

[auanasgheps]

@Tasshack your effort is incredible, but I am against custom integrations. There have been attempts in the past especially for Dreame robots, but quickly become outdated or unmaintained and users get lost. Why not contribute directly to the MioT integration which is well supported and updated? (BTW I still don't understand why the official Xiaomi integration does not support Dreame robots)

[webjib]

I can also confirm that running an automation that request a map update (ssid 6, aiid 3), is working fine! Now, I can get an almost live map. I've setup a request every 5 seconds, do you know if it's reasonable to request an update to the vacuum with a lesser delay, like 2 or 3 seconds?

[auanasgheps]

@webjib the map by default updates every 5 to 10 seconds, so I won't go under 5 seconds to be honest. What automation have you configured? My trigger does not detect when to start the automation, but I am a newbie with HA automations.

[webjib]

I'm using a custom Dreame D9 integration, for which I've created a new service called "vacuum_update_map" (it's derived from https://github.com/pooyashahidi/xiaomi_vacuum). So sharing the entire automation will not help.

But here is the trigger:

trigger:
  - platform: template
    value_template: '{{ is_state(''vacuum.aspirobot'', ''cleaning'') or is_state(''vacuum.aspirobot'',
      ''mopping'') or is_state(''vacuum.aspirobot'', ''returning'') }}'

[auanasgheps]

Thanks! I used the 'status' trigger within the UI, but it didn't work. I will try yours!

[Tasshack]

@Tasshack your effort is incredible, but I am against custom integrations. There have been attempts in the past especially for Dreame robots, but quickly become outdated or unmaintained and users get lost. Why not contribute directly to the MioT integration which is well supported and updated? (BTW I still don't understand why the official Xiaomi integration does not support Dreame robots)

@auanasgheps Xiaomi says Dreame is part of their eco system but they acquired Dreame late after they started to develop robot vacuums. Because of that, not all api features are compatible with the rest of the eco system. Thats why miiot and python-miio are not fully supporting the dreame devices yet. Especially with the lack of documentation you need to do a lot of reverse engineering to support them and to do that you need to own the acual device. The features that i have implemented to my integration can be achieved with miiot integration also but you have to write a lot of yaml and ninja scripts. I just prefer python, it is cleaner and more optimized.

[Tasshack]

@webjib the map by default updates every 5 to 10 seconds, so I won't go under 5 seconds to be honest. What automation have you configured? My trigger does not detect when to start the automation, but I am a newbie with HA automations.

@webjib @auanasgheps If you have Z10 or W10, you can check SIID: 15 PIID: 3 (dust-enable attribute) to check robot is docked or not. This attribute returns 1 when auto empting can be triggered from app and it is the only way to determine robot is docked or not when the battery is 100% (Not charging while docked).

[webjib]

@Tasshack actually, python-miio is already supporting some Dream Vacuums like the D9 or F9, but not all functions. The best situation would be to contribute first to the python-miio project to support more functions, and more models. Then, as the official HASS Miio integration is using python-miio project, it should be easy to officially support Dreame Vacuums. Lot of work.

[Tasshack]

@webjib yes python-miio recently added the Dreame support but the version that shipped with HASS not yet supports it. Also some utilities like mirobo still does not support Dreame yet. I think it is a good idea to contribute to python-miio but i am not so good at python to do that. I can provide information about undocumented features that i have gathered from valetudo firmware though.

[Tasshack]

@SpiGAndromeda how can i access the app logs, is there a guide that you can share with me?

[el-fredo]

I have a Dreame D9 and the map does not update for me either (only when I open the smartphone app). Could someone please explain to me in more detail how you solved the problem?

Do I have to create a script like this:

service: xiaomi_miot.call_action
target:
  entity_id: vacuum.xxxxx
data:      
  siid: 6
  aiid: 1
  params:
    - piid: 2
      value: '{"frame_type":"I"}'

and afterwards an automation that runs this script?

Thanks for your help.

[rbforelle]

@el-fredo I have set up an automation like this and it works to update the map for my Dreame W10

alias: map_update
description: ""
trigger:
  - platform: time_pattern
    seconds: /5
condition:
  - condition: or
    conditions:
      - condition: state
        entity_id: vacuum.xxx
        state: cleaning
      - condition: state
        entity_id: vacuum.xxx
        state: Washing
      - condition: state
        entity_id: vacuum.xxx
        state: Go Washing
action:
  - service: xiaomi_miot.call_action
    data:
      entity_id: vacuum.xxx
      siid: 6
      aiid: 1
      params:
        - piid: 2
          value: "{\"req_type\":1,\"frame_type\":\"I\",\"force_type\":1}"
mode: single

[auanasgheps]

@rbforelle Thank you for the automation! However, the way it's configured means that every 5 seconds HA will check if this automation should be running. I'm not sure if this is is a best practise, I fear it can overload HA.

[el-fredo]

Meanwhile I use this script:

alias: vacuum_update_map
sequence:
  - service: xiaomi_miot.call_action
    data:
      entity_id: vacuum.dreame_p2009_d3ec_robot_cleaner
      siid: 6
      aiid: 1
      params:
        - piid: 2
          value: "{\"req_type\":1,\"frame_type\":\"I\",\"force_type\":1}"
mode: single

If I manually start the script, the map gets updated. So I made following automation:

alias: Update Map
trigger:
  - platform: state
    entity_id: vacuum.dreame_d9
    to: cleaning
condition: []
action:
  - repeat:
      while:
        - condition: device
          device_id: f19b35c1ea84676cf97b91d70ab4a187
          domain: vacuum
          entity_id: vacuum.dreame_p2009_d3ec_robot_cleaner
          type: is_cleaning
      sequence:
        - service: script.1659276322175
          data: {}
mode: single

First it was a perfect solution, but during the last few weeks it stopped working. Don't know why...

[auanasgheps]

Meanwhile I use this script:

alias: vacuum_update_map
sequence:
  - service: xiaomi_miot.call_action
    data:
      entity_id: vacuum.dreame_p2009_d3ec_robot_cleaner
      siid: 6
      aiid: 1
      params:
        - piid: 2
          value: "{\"req_type\":1,\"frame_type\":\"I\",\"force_type\":1}"
mode: single

If I manually start the script, the map gets updated. So I made following automation:

alias: Update Map
trigger:
  - platform: state
    entity_id: vacuum.dreame_d9
    to: cleaning
condition: []
action:
  - repeat:
      while:
        - condition: device
          device_id: f19b35c1ea84676cf97b91d70ab4a187
          domain: vacuum
          entity_id: vacuum.dreame_p2009_d3ec_robot_cleaner
          type: is_cleaning
      sequence:
        - service: script.1659276322175
          data: {}
mode: single

First it was a perfect solution, but during the last few weeks it stopped working. Don't know why...

I was about to create a new automation when you posted this, which is what I had in mind. I'll do mine and let you know!

[PiotrMachowski]

@el-fredo I think you should add some kind of delay to your script/automation. Your current config sends a new command as soon as previous one has been sent

[el-fredo]

Unfortunately, I don't know much about creating scripts/automations manually. Where exactly should the delay be?

Feel free to post your finished code, then I will test it on my end as well.

[PiotrMachowski]

I should be enough:

alias: Update Map
trigger:
  - platform: state
    entity_id: vacuum.dreame_d9
    to: cleaning
condition: []
action:
  - repeat:
      while:
        - condition: device
          device_id: f19b35c1ea84676cf97b91d70ab4a187
          domain: vacuum
          entity_id: vacuum.dreame_p2009_d3ec_robot_cleaner
          type: is_cleaning
      sequence:
        - service: script.1659276322175
          data: {}
        - delay: 5
mode: single

[el-fredo]

I understand. This will call the script only every 5 seconds. Will test it when I am back home.

I wanted to avoid HA permanently updating the map, but only when the robot is cleaning. Should be correct with my automation though, or what do you think?

[auanasgheps]

I wanted to avoid HA permanently updating the map

Indeed. When I first created the automation, the end condition was wrong and the automation would never end. Result? After 3 days my HA host would crash because it filled the RAM lol

I am working on something like this. The idea is to trigger the update for every state change, but keep it going only when cleaning/mopping.

EDIT: It works great and triggers every time I start the cleaning process (with another automation or manually). Feel free to use it.

alias: Vacuum - Map Update
description: ""
trigger:
  - platform: state
    entity_id:
      - vacuum.xxx
action:
  - repeat:
      while:
        - condition: or
          conditions:
            - condition: state
              entity_id: vacuum.xxx
              state: cleaning
            - condition: state
              entity_id: vacuum.xxx
              state: mopping
            - condition: state
              entity_id: vacuum.xxx
              state: returning
      sequence:
        - service: xiaomi_miot.call_action
          data:
            entity_id: vacuum.xxx
            siid: 6
            aiid: 1
            params:
              - piid: 2
                value: "{\"req_type\":1,\"frame_type\":\"I\",\"force_type\":1}"
        - delay:
            hours: 0
            minutes: 0
            seconds: 5
            milliseconds: 0
mode: single

[el-fredo]

Unfortunately, it still does not work as desired.

So when I start the script manually, the map is updated. And if I manually start the automation once, the map is also updated. But without my manual intervention, the map is not updated.

Therefore, I assume that the problem is with the start of the automation.

I also noticed that the map does not update every 5 seconds, but only every 10 - 15 seconds.

[rbforelle]

@rbforelle Thank you for the automation! However, the way it's configured means that every 5 seconds HA will check if this automation should be running. I'm not sure if this is is a best practise, I fear it can overload HA.

I think it just depends on how HA is implemented. Normally something like "trigger by sensor status change" is also implemented by polling. In this case there will be no difference, whether the status is explicitly checked every 5 seconds in automation or checked by HA every x seconds in the background.

This automation runs for 3 days/6 times on my RPI3B+, everything seems great so far. I will report again if there are any problems in the long-term operation

[auanasgheps]

Unfortunately, it still does not work as desired.

So when I start the script manually, the map is updated. And if I manually start the automation once, the map is also updated. But without my manual intervention, the map is not updated.

Therefore, I assume that the problem is with the start of the automation.

I also noticed that the map does not update every 5 seconds, but only every 10 - 15 seconds.

@el-fredo My first test was successful with the automation posted above. Cleaning was actually triggered by another automation, and the map started throughout the cleaning process. Surely it does not update every 5 precise seconds, but it doesn't bother me.

I will let you know if the map will update again in the next clean, but I have no reason to believe it will stop.

[ShadowJonathan]

I can attest that the "Vacuum - Map Update" has helped my case, unfortunately it still takes more than 5 seconds to update the map each time, but at least it updates every so often now, which is better than i had before.

[auanasgheps]

I can attest that the "Vacuum - Map Update" has helped my case, unfortunately it still takes more than 5 seconds to update the map each time, but at least it updates every so often now, which is better than i had before.

Try reducing the time in the delay section, but I don't think it can get much better than this.

[Tasshack]

I am working on a custom integration about three months and i have almost completely reverse engineered the mi home app to do that. Turns out, newer firmware version vacuums does not upload complete map (I Frame) to cloud while it is moving. Instead, it uploads only changed sections (P Frame) of the map every three seconds. App only requests those frames from the cloud and renders ordered to their frame ids and updates the last map accordingly. App never sends request_map to the device when robot is running. Constantly requesting I frames over local api is strictly resticted from app because it has some awkard effects on robot performance and mapping. In my case requesting I frame every 3 seconds with an automation causes dissapearing walls between rooms and more often false positives with the 3d obstacle avoidance. I know this becase when i implemented parsing only P frames from cloud walls are fixed on my map. If robot doesn't know the current map (spot cleaning or fast mapping), it does not responds to force_type = 1 parameter instead request will time out and map will not be refreshed. If there is no saved or current map does not exists, app will always get the data from cloud instead of requesting it from the device.

I am planning to release my integration at the end of this month and it will solve all map refresh issues for the new version dreame device users. Spoiler; it will have multi-map feature too.

[PiotrMachowski]

@Tasshack will it include live map camera?

[Tasshack]

@Tasshack will it include live map camera?

It will have (almost) all features of the app including live map, saved map, auto generated room entities and many more.

This the map i rendered with the color index and center point finding algorithms directly extracted from the app. This is the pixel perfect copy of the map that i see on my phone.

[PiotrMachowski]

Will it work just with dreame vacuums or with all models that are supported by Xiaomi Home?

[Tasshack]

Will it work just with dreame vacuums or with all models that are supported by Xiaomi Home?

It will only support Dreame second generation lidar vacuums (L10, Z10, W10, S10 and maybe new S9). Map on Dreame robots with vslam are implemented differently on the app but i didn't implemeted it on my integration because i don't have a vslam robot. I can only develop for what i have at the moment but in theory my integration sould work with all vacuums that are using the same xiaomi home plugin with L10/Z10. I just wanted to use your integration and maybe fix it but there are tons of logic involving rendering the map that tied to variety of robots status properties. Because of that i decided to include the map parsing too on my integration. For example; I wanted to disable obstacle avoidance on certain rooms of my house with an automation. Since Dreame does not provide robots current room like on the roborock, i needed to know robots position and room coordinates. For determining robots current room, my integration needs to parse map data anyway even it does not renders the map. So i have added the whole map logic to my integration completely reverse engineered from the app (including P Frame parsing) just for 3 seconds refresh rate of robots position.

[Tasshack]

I can also render the map with different color pattern, icon set and with room names too. Even the text font is extracted from the app, maybe i will face some copyright issues in the future for doing it.

[PiotrMachowski]

Actually map extractor can also retrieve current room, it should work with all supported vacuums.

It looks really nice 👍

Do you plan to make it compatible with the Map Card?

[Tasshack]

Actually map extractor can also retrieve current room, it should work with all supported vacuums.

It looks really nice 👍

Do you plan to make it compatible with map card?

Yes map extractor can do that but refreshing issue with it is preventing to use robots current position right now.

Currently i am already using your map card for developing my integration and it will be compatible. Rendering map on server side uses too much cpu and takes some time with python (i have added a caching mechanism for each layer for improving the render time). Because of that i have also added the valetudo map support (it has own camera entitiy) but valetudo map card cannot render the cleaning sequence or room icons and uses different color index algorithm. So i will eventually develop javascript renderer that works on client side like the valetudo map card but it will render the same looking map with the backend renderer.