Closed daufinsyd closed 2 years ago
It might be related with #10276 but since these are made by another manufacturer I don't know ...
I am having the same issue with two different firmwares and multiple different adapters, granted all are CC2652P.
Routers:
The two firmwares I have tried flashing all these devices with are:
I will see these devices behave normally for anywhere from 1-3 days consistently pinging in however eventually they will no longer ping in. I usually only notice this when the last seen for all gets toward the 12hour + range. It does seem that they all stop reporting in around the same time but I cannot verify for sure. Running a network map request reports back that the devices are unreachable. Only error I see for each device: error 2021-12-20 16:37:27: Failed to execute LQI for 'Basement Router'
I have tried doing a packet capture but in all honesty I am not exactly sure what I should be looking for in this case. Simply power cycling the devices seems to return them to a working state in all cases. I just did this to all of them around 2-3 hours ago, so please let me know if there is anything else I can capture that would help troubleshoot this issue.
It might be related with #10276 but since these are made by another manufacturer I don't know ...
One thing I want to point out, none of my other devices are exhibiting this behavior these include but are not limited to: KMPCIL_RES005 (KMPCIL) Xiaomi (MCCGQ11LM, WXKG11LM, WSDCGQ11LM, VOCKQJK11LM) Philips Hue (9290012607, 9290022166, 9290012573A) Innr (sp224, sp234) Smartthings (STS-IRM-250)
The issue only appears with the custom router devices and seems to have been happening fairly consistently across multiple versions of zigbee2mqtt, currently on 1.22.1.
My coordinator is a https://www.tubeszb.com/product/cc2652_coordinator/1?cp=true&sa=false&sbp=false&q=false&category_id=2 with firmware version 20210708
Great to see that we're not the only one ! :)
I usually only notice this when the last seen for all gets toward the 12hour + range. It does seem that they all stop reporting in around the same time but I cannot verify for sure. I had the issue yesterday and noticed that 2 routers were still connected. So it definitely doesn't happend for all a the same time (but within 12hours as you said ?).
I also get the same LQI error message as you.
I bought a Sonoff ZigBee 3.0 USB Dongle plus and I'll do test once I receive it. I'd like to provide more debug info but on't know where to start.
According to HA, the routers disconnect every two days and 5 hours.
I got a new sonoff dongle plus ; I'll try with it.
I flashed the latest firmware (20211217) for the new sonoff dongle 3.0 plus, 2 days ago and every routers disconnected today. :( I didn't set the experimental power tx so that shoudln't be the cause but strangely enough, increasing the tx power seems to lower the reability (I made a few test and although the lqi was a little bit better, I had way more 'lqi failed for...' errors).
So I ordered new plug/sonoff dongle, create a new network from scratch with
@daufinsyd I can report back that I am still having this issue and it does seem to further point to I believe one of a few things..
Standard devices tested (using both zstack router firmware and ptvo basic configured router firmware)
Custom devices (using ptvo firmware, tested configured with 60 second ping. "enable watchdog timer was enabled in some, and not in others to check for any difference".
I created some custom PCB boards based on the designs of the popular coordinator sticks available (tubes router/zigstar/zzh) only with intent of using them as router devices loaded with various sensors (temperature/humidity/voc/ect), currently only using temperature/humidity. These were configured with ptvo firmware, additionally as I had added an additional LED to the board I was able to create a virtual switch in ptvo's firmware that would allow me to control the LED on the board. The intent here was to see if the device would respond to an on/off request when it went into a state of no longer pinging in.
result: device works perfectly pinging in every 60 seconds for days, seemingly 4-5 days? At a certain point I will notice that temperature data is not updating or changing in home assistant, I check back in zigbee2mqtt and the devices will show that they havented pinging in for X hours. Attempting to flip the virtual switch to trigger the LED on the board produces a timeout in zigbee2mqtt. Unplugging the device from power and instantly plugging it back in seems to fix everything with that device until the next "crash".
I have setup a second testing instance of home assistant with ZHA, will probably also install zigbee2mqtt. I have plenty of zigbee devices I can throw at the test network but I would really need a better understanding from @Koenkk or someone with more knowledge on this to help collect useful debugging data.
I can also confirm this behavior with Sonoff Stick flashed with latest router firmware.
I have a Sonoff usb stick (using the 20220103 firmware) configured as a coordinator, and a bunch of zzh's as routers along with some Sonoff s26r2zb's. Interestingly, I see the same behaviour, but only in the s26r2zb devices. All of the zzhs remain connected to the mesh.
Some additional info about the S26R2ZBs. I've taken one apart and photographed in the insides (mine is the UK version, but I would doubt there's much variation on these things).
Looks like it is powered by a CC2652 and has some handy solder pads for RX, TX, etc.
I contacted the Sonoff Support and linked this case, I got a reply :
Dear buyer, the technical department has tested it, this is the problem of mqtt, they have contacted the mqtt staff to deal with it
(I guess they meant zigbee2mqtt and @Koenkk).
@daufinsyd they haven't contacted me about this. Could you make a sniff when command execution fails?
https://www.zigbee2mqtt.io/advanced/zigbee/04_sniff_zigbee_traffic.html#with-cc2531
@daufinsyd they haven't contacted me about this. Could you make a sniff when command execution fails?
https://www.zigbee2mqtt.io/advanced/zigbee/04_sniff_zigbee_traffic.html#with-cc2531
I buy a CC2531 Stick and I'll post the ouput. Maybe @irakhlin can you try yo sniff when command execution fails in the meantime ?
I can also confirm this behaviour.
ZigBee Smart Plug DE Type S26R2ZBTPF Zigbee2MQTT version: 1.22.1 Adapter hardware: CC2652RB from https://slae.sh/projects/cc2652/
I have also experienced the S26R2ZB going offline. On top of that they dont seem to allow end devices to join (even when not offline), even if I hold the end device right next to the S26R2ZB, the end device still joins directly to the coordinator. If i try to add end devices which are out of range of the coordinator but in range of the S26R2ZB it fails to join. On top of this, the range of the S26R2ZB is very bad, and very low LQI numbers are reported.
ZigBee Smart Plug DE Type S26R2ZBTPF Zigbee2MQTT version: 1.22.1 Adapter hardware: ZBBridge with Tasmota, EZSP (which is running rock solid in my setup btw)
I experience the same issue, FR version of the plugs. I can also confirm pretty low LQI values. They do seem to work as routers though.
@daufinsyd @Koenkk I would be more than happy to do a sniff when the issue happens again. Interestingly enough however since my last update ( 8 days ago) all of my routers are online and showing strong signal without any restarts. I did however rearrange my mesh slightly on that day and removed 2 router/repeaters that had terrible signal strength. Could this issue be related to this? Both devices I removed were cc2652p based routers, could this lead to them attempting to router messages and failing due to signal strength and causing other devices to overflow? I will continue monitoring closely and let you now if there is any change.
@Koenkk It looks like my network is partially back to that same state. It seems half of my routers have not responded in 23 hours (running the standard zstack router firmware). The other half of my routing devices are using ptvo firmware they are all pinging but they do have this feature enabled:
Enable watchdog timer – this option enables a built-in watchdog time in the chip. The timer would reset the device if it froze for more than 1 second. I do not recommend using it in complex and untested configurations with many sensors, and the device may unexpectedly reset.
I am going to get a sniffer out, is there anything specific you need that would help troubleshoot this?
Hey @Koenkk and @daufinsyd
I was able to get a capture of my network. One without attempting to trigger any devices manually and once flipping a couple devices that are alive and a few of the routers mentioned above. I see the message to the router but no response back. I am also not all that familiar with what I should be seeing but it does seem that maybe my network is being bombarded with too many packets? Regardless I put the two captures together with a list of the devices on my network with their status and my key.. What is the best way of providing this to you?
@irakhlin could you put upload the pcapng
file + network key here? Also what is the network address of the s26r2zb, you can find this in the z2m frontend)
@Koenkk Here is the capture, in my case the router nodes going out are not s26r2zb but cc2652p boards. The key, a few different captures and a spreadsheet of my network addresses with the affected DOWN nodes is included. capture.zip
@irakhlin thanks for the logs, checked some sniff and it seems the devices don't send "Link status" messages anymore (which routers should do). I don't expect this to be a z2m bug, rather a bug in the TI SDK. My router fw is using a quite old sdk, I will try to provide an update in the coming days.
@Koenkk I would tent to agree that this is probably an issue in the firmware. But I just wanted to make sure you saw my earlier comments; half of the affected devices are running your router firmware but the other half are running @ptvoinfo (https://ptvo.info) firmware. Additionally none of my "none diy" devices are affect, other than lower AQI when the router nodes crash. I did spend some time trying to test using the new SDK with some changes to the patch file but in that case the devices would join and almost instantly leave the network. This is probably because of something I did however. Let me know if you need me to test the firmware when you get around to it.
@ptvoinfo also uses the TI stack, however I don't know which version.
@Koenkk If I understand this correctly, the proper solution would then be to flash an updated firmware onto the S26 plugs? If yes, is there a possibility that this can be done OTA through zigbee2mqtt, or would I need to disassemble the units and flash them? (sorry if these questions are stupid - I am a newbie in all these home automation/hobby small electronic devices).
@Koenkk I use "simplelink_cc13x2_26x2_sdk_5_10_00_48".
This problem looks like a memory leak somewhere in the SDK. After some time the firmware cannot allocate memory for current tasks and stop responding.
@ptvoinfo thanks, so not the latest
I tried the build a new fw with the latest stack but somehow it is not joining, asked ti for help: https://e2e.ti.com/support/wireless-connectivity/zigbee-thread-group/zigbee-and-thread/f/zigbee-thread-forum/1071794/launchxl-cc26x2r1-simplelink-sdk-5_30_01_01-router-running-zr_genericapp-not-joining-network
the proper solution would then be to flash an updated firmware onto the S26 plugs? If yes, is there a possibility that this can be done OTA through zigbee2mqtt, or would I need to disassemble the units and flash them?
Once we know the latest SDK fixes this I will ask sonoff to provide an OTA update.
@Koenkk Good news then, guess I was not the only one to have the issue of the device leaving instantly. It does look like we have a response from TI. That was quick!
BDB_COMMISSIONING_INITIALIZATION commissioning mode is now required to set up the BDB state machine. Please do not modify zclGenericApp_Init and instead call BDB_COMMISIONING_MODE_NWK_STEERING|BDB_COMMISIONING_MODE_FINDING_BINDING during the BDB_COMMISSIONING_INITIALIZATION case of zclGenericApp_ProcessCommissioningStatus, or start a timer with a small delay during initialization which begins this BDB process after timing out.
Regards, Ryan
Managed to get it working, can you test https://github.com/Koenkk/Z-Stack-firmware/tree/develop/router/Z-Stack_3.x.0/bin ?
@Koenkk I just pulled down CC1352P2_CC2652P_launchpad_router_20220125.zip and am testing it now. I see you bumped up nv and the routing tables so this should be great.
I will keep this deployed somewhere central to my mesh for a bit and try and do a sniff to see how its behaving but so far looking good.
Just as an update, 4 days in and router is still up and running. Just performed a capture of the network briefly, the router is broadcasting that is has 20 active links. This matches to the map tool in zigbee2mqtt. I started to move the other router only devices in my network to this firmware, but I am going to give it a few more days to be sure as the issue sometimes took up to 5/6 days. I was able to get LED to blink when device is not on the network by adding a zclGenericApp_UpdateLedState function to call LED_stopBlinking / LED_startBlinking in zcl_genericapp but its really not essential.
The new router firmware needs to be flashed on the sonoff plug (and other misbehaving routers, if I understand correctly the issue) ? The sniffer/debugger still hasn't arrived.
@daufinsyd this fw is not made for the sonoff plugs, Sonoff should provide an OTA for that, I will mail them once @irakhlin has confirmed this fw fixes the issue
OK, thank you @Koenkk !
@Koenkk @daufinsyd The routers have been running for 5+ days without needing a restart and captures are showing they are routing. I feel pretty comfortable to say this should be working. Unfortunately every device I have uses cc2652p or cc1352p chips. I wont be able to test the other device firmwares. If anything changes I will be sure to update.
Great, I've mailed sonoff
Does this affect the coordinator as well?
I created #11216 in case it's not related.
I recently moved from a ConBee II (which was actually running without problems) to the SONOFF Zigbee USB Dongle Plus, with firmware 20220103. Since doing so, I’ve noticed that every 1-2 days, all of my routers start dropping off. I’m also not able to enable join mode. Restarting zigbee2mqtt makes it work for a couple of minutes, but eventually fails again. I then stop zigbee2mqtt, pull out the usb dongle, wait a few seconds, replug the usb, start zigbee2mqtt… then everything is okay again for a day or two (or sometimes only for 30 mins like a reboot).
Debug Received MQTT message on 'zigbee2mqtt/bridge/request/permit_join' with data '{"device":null,"time":254,"transaction":"wjftl-3","value":true}'
Info Zigbee: allowing new devices to join.
Error Request 'zigbee2mqtt/bridge/request/permit_join' failed with error: 'SRSP - ZDO - mgmtPermitJoinReq after 6000ms'
Debug Error: SRSP - ZDO - mgmtPermitJoinReq after 6000ms at Timeout._onTimeout (/app/node_modules/zigbee-herdsman/src/utils/waitress.ts:64:35) at listOnTimeout (internal/timers.js:557:17) at processTimers (internal/timers.js:500:7)
Info MQTT publish: topic 'zigbee2mqtt/bridge/response/permit_join', payload '{"data":{},"error":"SRSP - ZDO - mgmtPermitJoinReq after 6000ms","status":"error","transaction":"wjftl-3"}'
FYI, there is more general discussions about non-specific performance issues with the ITead Sonoff S26R2ZB plugs here -> https://github.com/Koenkk/zigbee2mqtt/discussions/10909
Sonoff asked me which firmware versions you are running, could you provide me the data/database.db entries of your S26R2ZB?
I have three pieces, all of them seem to have the same firmware:
{"id":2,"type":"Router","ieeeAddr":"0x00124b0024c118e9","nwkAddr":40380,"manufId":4742,"manufName":"SONOFF","powerSource":"Mains` (single phase)","modelId":"S26R2ZB","epList":[1,13,242],"endpoints":{"1":{"profId":260,"epId":1,"devId":256,"inClusterList":[0,3,4,5,6,64599],"outClusterList":[25],"clusters":{"genBasic":{"attributes":{"zclVersion":3}},"genOnOff":{"attributes":{"onOff":1}}},"binds":[],"configuredReportings":[],"meta":{}},"13":{"profId":49246,"epId":13,"devId":57694,"inClusterList":[4096],"outClusterList":[4096],"clusters":{},"binds":[],"configuredReportings":[],"meta":{}},"242":{"profId":41440,"epId":242,"devId":97,"inClusterList":[],"outClusterList":[33],"clusters":{},"binds":[],"configuredReportings":[],"meta":{}}},"hwVersion":1,"dateCode":"20210511","swBuildId":"2.0.0","zclVersion":3,"interviewCompleted":true,"meta":{},"lastSeen":1642936853353,"defaultSendRequestWhen":"immediate"}
Two of mine (I'll check the other but I think they have the same firmware.
{"id":5,"type":"Router","ieeeAddr":"0x00124b0024c250cf","nwkAddr":23064,"manufId":4742,"manufName":"SONOFF","powerSource":"Mains (single phase)","modelId":"S26R2ZB","epList":[1,13,242],"endpoints":{"1":{"profId":260,"epId":1,"devId":256,"inClusterList":[0,3,4,5,6,64599],"outClusterList":[25],"clusters":{"genOnOff":{"attributes":{"onOff":0}},"genBasic":{"attributes":{"modelId":"S26R2ZB","manufacturerName":"SONOFF","powerSource":1,"zclVersion":3,"hwVersion":1,"dateCode":"20210511","swBuildId":"2.0.0"}}},"binds":[],"configuredReportings":[],"meta":{}},"13":{"profId":49246,"epId":13,"devId":57694,"inClusterList":[4096],"outClusterList":[4096],"clusters":{},"binds":[],"configuredReportings":[],"meta":{}},"242":{"profId":41440,"epId":242,"devId":97,"inClusterList":[],"outClusterList":[33],"clusters":{},"binds":[],"configuredReportings":[],"meta":{}}},"hwVersion":1,"dateCode":"20210511","swBuildId":"2.0.0","zclVersion":3,"interviewCompleted":true,"meta":{},"lastSeen":1642429814967,"useImplicitCheckin":true}
{"id":2,"type":"Router","ieeeAddr":"0x00124b0024c11920","nwkAddr":29448,"manufId":4742,"manufName":"SONOFF","powerSource":"Mains (single phase)","modelId":"S26R2ZB","epList":[1,13,242],"endpoints":{"1":{"profId":260,"epId":1,"devId":256,"inClusterList":[0,3,4,5,6,64599],"outClusterList":[25],"clusters":{"genBasic":{"attributes":{"modelId":"S26R2ZB","manufacturerName":"SONOFF","powerSource":1,"zclVersion":3,"hwVersion":1,"dateCode":"20210511","swBuildId":"2.0.0"}},"genOnOff":{"attributes":{"onOff":1}}},"binds":[],"configuredReportings":[],"meta":{}},"13":{"profId":49246,"epId":13,"devId":57694,"inClusterList":[4096],"outClusterList":[4096],"clusters":{},"binds":[],"configuredReportings":[],"meta":{}},"242":{"profId":41440,"epId":242,"devId":97,"inClusterList":[],"outClusterList":[33],"clusters":{},"binds":[],"configuredReportings":[],"meta":{}}},"hwVersion":1,"dateCode":"20210511","swBuildId":"2.0.0","zclVersion":3,"interviewCompleted":true,"meta":{},"lastSeen":1642319845568,"useImplicitCheckin":true}
Hello,
I have 2 S26R2ZB and I have the same problem. Firwmare version seems more recent:
{"id":6,"type":"Router","ieeeAddr":"0x00124b00258acdcd","nwkAddr":51874,"manufId":4742,"manufName":"SONOFF","powerSource":"Mains (single phase)","modelId":"S26R2ZB","epList":[1,13,242],"endpoints":{"1":{"profId":260,"epId":1,"devId":9,"inClusterList":[0,3,4,5,6,64599],"outClusterList":[25],"clusters":{"genOnOff":{"attributes":{"onOff":1}},"genBasic":{"attributes":{"modelId":"S26R2ZB","manufacturerName":"SONOFF","powerSource":1,"zclVersion":3,"hwVersion":1,"dateCode":"20210923","swBuildId":"2.0.1"}}},"binds":[],"configuredReportings":[],"meta":{}},"13":{"profId":49246,"epId":13,"devId":57694,"inClusterList":[4096],"outClusterList":[4096],"clusters":{},"binds":[],"configuredReportings":[],"meta":{}},"242":{"profId":41440,"epId":242,"devId":97,"inClusterList":[],"outClusterList":[33],"clusters":{},"binds":[],"configuredReportings":[],"meta":{}}},"hwVersion":1,"dateCode":"20210923","swBuildId":"2.0.1","zclVersion":3,"interviewCompleted":true,"meta":{},"lastSeen":1644686090488,"defaultSendRequestWhen":"immediate"}
{"id":7,"type":"Router","ieeeAddr":"0x00124b00258ac6ab","nwkAddr":49523,"manufId":4742,"manufName":"SONOFF","powerSource":"Mains (single phase)","modelId":"S26R2ZB","epList":[1,13,242],"endpoints":{"1":{"profId":260,"epId":1,"devId":9,"inClusterList":[0,3,4,5,6,64599],"outClusterList":[25],"clusters":{"genBasic":{"attributes":{"modelId":"S26R2ZB","manufacturerName":"SONOFF","powerSource":1,"zclVersion":3,"hwVersion":1,"dateCode":"20210923","swBuildId":"2.0.1"}},"genOnOff":{"attributes":{"onOff":0}}},"binds":[],"configuredReportings":[],"meta":{}},"13":{"profId":49246,"epId":13,"devId":57694,"inClusterList":[4096],"outClusterList":[4096],"clusters":{},"binds":[],"configuredReportings":[],"meta":{}},"242":{"profId":41440,"epId":242,"devId":97,"inClusterList":[],"outClusterList":[33],"clusters":{},"binds":[],"configuredReportings":[],"meta":{}}},"hwVersion":1,"dateCode":"20210923","swBuildId":"2.0.1","zclVersion":3,"interviewCompleted":true,"meta":{},"lastSeen":1644604122189,"defaultSendRequestWhen":"immediate"}
I've mailed sonoff with the details, waiting for a reply now.
Hi, i have the same issue with my Sonoff v3 USB Plus Sticks .. running firmware CC1352P2_CC2652P_launchpad_router_20210128.zip or CC1352P2_CC2652P_launchpad_router_20220125.zip with the 2021 the router goes offline after some minutes may be 10 or so .. as cordinator i use a zzh electrolama
@d0m1n1qu3 This should have been fixed in CC1352P2_CC2652P_launchpad_router_20220125, see above posts. Does the problem occur on the latest firmware (I do not find this clear in your comment)?
hi .. actually i think i made something wrong .. also with flashing the 2022 in the z2m device list the sonoff appers with version "Firmware Version 20210128" .. so finally i don't get the 2022 flashed :-( but i dont know why ... the cc2538-bsl.py says all is fine incl the crc check
Anyone seen similar issues with later shipped "ITead Sonoff ZBMINI Zigbee Two Way Smart Switch" (SKU/MPN: M0802010009)?
Apparently, some people now started receiving ZBMINI with a new board version stating "ZBMINI V2.0 2021.04.28" and this new revision on the ZBMINI board got the same CC2652P1 / CC2652P1F MCU chip so maybe they ship a with a similar firmware too?
ITead site states that Sonoff ZBMINI is a Zigbee Router device -> https://itead.cc/product/sonoff-zbmini-zigbee-smart-switch/
If I understand this correctly, the proper solution would then be to flash an updated firmware onto the S26 plugs? If yes, is there a possibility that this can be done OTA through zigbee2mqtt, or would I need to disassemble the units and flash them?
Once we know the latest SDK fixes this I will ask sonoff to provide an OTA update.
Do you think you could convince ITead to help provide publuc URLs for Sonoff OTA to zigbee-herdsman-converters ota library?
https://github.com/Koenkk/zigbee-herdsman-converters/tree/master/lib/ota
Wondering if ITead's Sonoff ZBBridge offer OTA firmware updates for any of their other Sonoff branded Zigbee devices today?
@Hedda I do not have the bridge nor have been in any serious communication with ITead/Sonoff. However, I spent quite some time searching online for firmware updates, yet I failed. From what I have found at forums, it seems that Sonoff indeed provides OTA updates in eWelink (and their bridge) but they do not want to make them publicly available. I personally really dislike this situation, because the product performance really depends on the specific FW shipped. Currently, I have for example three useless smart plugs- maybe they would work when bought from another seller if the issue is fixed in a newer firmware. In the worst case, I am gonna buy the Sonoff Zigbee gate, update the firmware, and return the gate in 14 days (which is legit at least here in Europe).
From what I have found at forums, it seems that Sonoff indeed provides OTA updates in eWelink (and their bridge) but they do not want to make them publicly available.
I believe that ITead/Sonoff have listed to community requests/demands before in the past once a dialogue has been established.
If representatives from large open-source projects Zigbee2MQTT and Home Assistant's ZHA contact ITead's product managers and in a nice tone request/demand that they open up their OTA server or publish public OTA URLs so that Sonoff Zigbee OTA firmware images can be downloaded from third-party Zigbee implementations then it probably not impossible to convince if argued well.
Alternatives is to either update devices with ITead's Sonoff ZBBridge and then use a debug probe to read firmware images via cJTAG, or use a Zigbee sniffer adapter to sniff the traffic between with ITead's Sonoff ZBBridge and Sonoff Zigbee devices with Wireshark to collect the OTA firmware images and then share them via -> https://github.com/Koenkk/zigbee-OTA
The best would obviously be for the community if ITead could be convinced to just publicly publish URLs for standard OTA images.
PS: See this project which attempts to capture OTA via wireshark -> https://github.com/pipiche38/Capture-OTA-from-Wireshark
Hello :)
What happened
After a day or two running perfectly fine, routers kind of disconnect from the network; trying to execute any command results in timeout. It happens for all routers I have (6 sonoff s26r2zb). End devices (sonoff snzb02 / 03) (directly connected to the coordinator) don't appear to be affected.
example of failed command:
After the router "disconnected", most of the time, simply plugging it off and back solve the issue for a day or two.
I tried increasing the TX power but it didn't help, all devices (with either good or bad LQI disconnect).
What did you expect to happen
Devices are working.
How to reproduce it (minimal and precise)
Debug info
Zigbee2MQTT version: 1.22.1 Adapter hardware: CC2652RB (from slae.sh, as recommended in the official website) Adapter firmware version: transportrev 2, product 1, majorrel 2, minorrel 7, maintrel 1, revision: 20210708
Edit: typo