Fix scanner force-scanning

[uzlonewolf]

The move to argparse in the last release broke the force-scan IP list.

[jasonacox]

Not sure if we want to address this because it is correct, but I suspect some (ok, even me) will specify a host address with netmask:

$ python3 -m tinytuya scan -f 10.0.1.1/24                                                                        pr/511

TinyTuya (Tuya device scanner) [1.14.1]

[Loaded devices.json - 43 devices]

Scanning on UDP ports 6666 and 6667 and 7000 for devices for 18 seconds...

    Option: Network force scanning requested.

    Running Scan...
ERROR: Unable to get network for '10.0.1.1/24', ignoring.
Traceback (most recent call last):
  File "/Users/jason/Code/tinytuya/tinytuya/scanner.py", line 931, in _generate_ip
    network = ipaddress.ip_network(netblock)
  File "/Users/jason/miniforge3/lib/python3.10/ipaddress.py", line 74, in ip_network
    return IPv4Network(address, strict)
  File "/Users/jason/miniforge3/lib/python3.10/ipaddress.py", line 1507, in __init__
    raise ValueError('%s has host bits set' % self)
ValueError: 10.0.1.1/24 has host bits set

[MarkCupitt]

@jasonacox @uzlonewolf

Hi Guys, I grabbed server.py and scanner.py (I think it was only the two files) from @uzlonewolf's commits and ran them here, they did not seem to fix my issue, ie, did not see any v 3.5 devices, have attached log and devices file files for you, hope it helps

What I did notice last night I that 2 of the three devices that were not being seen by tinytuya eventually showed up on the webui, this morning, which was interesting.

tinytuya scan -d >debug.log 2>&1 tinytuya scan -f 192.168.2.0/24 >force.log 2>&1

debug.log devices.json force.log

[uzlonewolf]

@MarkCupitt that force.log is still using the old scanner.py, not this modified one, as shown by:

Traceback (most recent call last):
  File "/home/mark/.local/lib/python3.11/site-packages/tinytuya/scanner.py", line 902, in _generate_ip
    network = ipaddress.ip_network(netblock)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ipaddress.py", line 83, in ip_network
    raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 network')
ValueError: ['192.168.2.0/24'] does not appear to be an IPv4 or IPv6 network

Replace /home/mark/.local/lib/python3.11/site-packages/tinytuya/scanner.py and try it again.

[jasonacox]

@MarkCupitt if you want to pull the full repo down you could do this to test the PR:

# Clone
git clone https://github.com/jasonacox/tinytuya.git
cd tinytuya

# Grab PR
git pull origin pull/511/head

# Test
python3 -m tinytuya scan -f

[MarkCupitt]

@jasonacox @uzlonewolf sorry, my bad, I was using the wrong lib .. so per @jasonacox

From https://github.com/jasonacox/tinytuya
 * branch            refs/pull/511/head -> FETCH_HEAD
Already up to date.

python -m tinytuya scan -d >debug.log 2>&1
python -m tinytuya scan -f 192.168.2.0/24 >force.log 2>&1

So, Im seeing 3.5 Devices now in the Force scan .. fantastic .. but some still show off line in the server web and I dont see any 3.5 Devices in the Debug log. Im not sure whether they are not sending packets, or tinytuya is missing them

force.log devices.json debug.log

[uzlonewolf]

The Wireshark capture you posted to #510 show that the devices are not sending broadcasts. Since the devices are not sending packets there is nothing for TinyTuya to receive.

As the device in the Wireshark capture immediately tries to find your phone after it is rebooted, I suspect Smart Life is still running in the background on your phone.

[jasonacox]

Try to stop the SmartLife app and see if the devices start broadcasting.

[MarkCupitt]

@jasonacox @uzlonewolf Yep, I actually had disabled wifi for my phone with Smartlife on it, restarted all the device.

The pull request does address the 3.5 issue which was the root cause of my issue, and I accept that the devices are "different" and do something different with Broadcast packets.

Ill adjust my use case to work with this approach as I will know the IP addresses and can access them directly. Im comfortable with that, so I would say the pR is Ok

Thanks for the prompt fixes and responses, they are very much appreciated

[MarkCupitt]

I'm going to call this issue successfully closed .. as any remaining issues appear to be device-related. Thanks again ..

[jasonacox]

I sure hope this isn't the start of a new device series/trend that don't broadcast discovery packets.

Thanks for your help with this, @MarkCupitt !

[uzlonewolf]

@jasonacox Do you want to add a "Start Force-scan" button to the server?

[jasonacox]

@jasonacox Do you want to add a "Start Force-scan" button to the server?

Oh, that's a great idea. Hum. Let me poke at that tonight. Might be worth including in this PR.

[uzlonewolf]

I also fixed the "ValueError: 10.0.1.1/24 has host bits set" thing but it looks like I forgot to push it. 1 sec

[MarkCupitt]

@jasonacox @uzlonewolf

As a humble suggestion, If you had a force scan update snapshot.json, perhaps add a key to the JSON like this:

poll_method: "broadcast"
or
poll_method: "poll"

with broadcast being the default, any other code that uses snapshot.json would know that the device was there and that it needed to poll the device and not wait for a broadcast.

Secondly, adding a timestamp recording the time it was last successfully accessed would allow for a last accessed age, thus allowing logic for retries on individual devices, should it pass a threshold, or then showing them as going offline. Because of high disk io, if running this on a pi, it may be better to keep this in memory and just serve an exact copy of snapshot.json out an API, for use in other systems.

Lastly, if you were to add an optional new file, say "local_ips.json` with something like this

[{
    "deviceid": "eb6cfead8a7ce1de12wvaf",
    "ip": "192.168.2.x",
    "key": "akdfjf",
    "version": "3.5"
  }]

You could build in a poll (on a timer) like monitor.py into the scan only for non-broadcast devices shown in snapshot.json, and be able to handle most edge cases I would think

Updating the UI to indicate if it is a broadcast or poll and the age [ eg: formatted DateTime (12 seconds ago) ] would also be very useful

Lastly, a wishlist request, In thee UI, update the Offline devices from info in Devices.json so they are the same display as the Online Table display.

This covers pretty much all the UI and oddball devices I can see from where I sit and may make Tinytuya more flexible

I'm quite happy to contribute, but Python is not my first language, but it is 15 years old, however feel free to ask, dev, testing etc

If you do agree to the above, I would be happy to write a separate MQTT server that uses the above (basically reads snapshot.json, from server.py) to publish values to MQTT on a regular basis, my feeling is that this is best left out of server.py, as its use case is very different, although I see there has been some thought on MQTT already, happy to contribute to that .. if you deem that a better approach

The end result would be a pretty cool setup where a process server.py was running in the background keeping snapshot.json up to date, then a second process that read snapshot.json, ( looked for file changes and reload would be nice) and polled the devices (threaded or asyncio) for status and emitted MQTT publish events

added #512 as an idea