netmanchris commented 6 years ago

First thanks for this project! Is there any specific tests plans for specific HomeKit accessories? I see that you’ve had some people report that the module works with various accessories but I was wondering if there’s a test plan that users can submit to track which devices have been proven to work?

jlusiardi commented 6 years ago

Hey,

thanks for your encouraging words!

By test plan you mean some predefined steps to 'prove' that a device works with the code? Perhaps everyone that tests a device should perform some standard operations on it:

pair the device
perform get_accessories
read all readable accessories
perhaps write some of the writable accessories (which?)
unpair the device

After this, post the results with by doing a pull request would make it transparent for the world.

netmanchris commented 6 years ago

Hey @jlusiardi

I think this could definitely work. Just need a quick "How to test and what to post" inclusion on the readme file to make sure you get what you're expecting and I'd be happy to do some tests and send some PRs with the data!

jlusiardi commented 5 years ago

Hey @netmanchris,

sorry it took me some time to start, but i added a wiki page HomeKit Accessory Testing Guide. I think a little coded support would also be nice i guess.

Regards Joachim

jlusiardi commented 5 years ago

Hey @netmanchris,

could you test test_accessory.py from branch fix_53_testconcept and report back testpairingdata and testresult for your accessory?

Regards Joachim

netmanchris commented 5 years ago

@jlusiardi For sure - Swamped at the day job but this is something I want to get back to and start posting some testresults. Will commit to doing some testing, but won't be in the next couple of days. :)

netmanchris commented 5 years ago

@jlusiardi For the testing.

Would it be ok to post to the wiki with tested devices using the classification.

Thinking that just posting at least this output per device model would be useful. Let me know if that makes sense.

Name: Netatmo Welcome (2)._hap._tcp.local.
Url: http_impl://10.101.0.154:5001
Configuration number (c#): 3
Feature Flags (ff): Supports HAP Pairing (Flag: 1)
Device ID (id): EB:30:84:4C:A6:DA
Model Name (md): Welcome
Protocol Version (pv): 1.1
State Number (s#): 1
Status Flags (sf): Accessory has been paired. (Flag: 0)
Category Identifier (ci): IP Camera (Id: 17)

netmanchris commented 5 years ago

@jlusiardi

Working on first testplan output for Eve Light Strip. Seems like the wiki docs aren't up to date and there's something missing that's preventing me from running the pairing tests. Help?

Step 1 Discovery

Command - (venv) Christophers-MacBook-Air:HomekitPythonPlaying christopheryoung$ python -m homekit.discover > pairingtest.txt

Output

Name: Eve Light Strip 8606._hap._tcp.local. Url: http_impl://10.101.0.175:80 Configuration number (c#): 1 Feature Flags (ff): Supports HAP Pairing (Flag: 1) Device ID (id): 1A:EA:89:9D:04:AF Model Name (md): Eve Light Strip 20EAS9901 Protocol Version (pv): 1.1 State Number (s#): 1 Status Flags (sf): Accessory has not been paired with any controllers. (Flag: 1) Category Identifier (ci): Lightbulb (Id: 5)

Step 2. unpaired identify

(venv) Christophers-MacBook-Air:HomekitPythonPlaying christopheryoung$ python3 -m homekit.identify -d 1A:EA:89:9D:04:AF

Result:

Eve Light strip flashes

Step 3. Pairing

ASSUME That the ${Device Setup Code} is the Homekit pairing code?

(venv) Christophers-MacBook-Air:HomekitPythonPlaying christopheryoung$ python3 -m homekit.pair -d 1A:EA:89:9D:04:AF -p ${Device Setup Code} -f test_report.json usage: pair.py [-h] -d DEVICE [-p PIN] -f FILE -a ALIAS [--log LOGLEVEL] pair.py: error: the following arguments are required: -a (venv) Christophers-MacBook-Air:HomekitPythonPlaying christopheryoung$

Attempted to fill in the missing argument with an Alias of "Eve"

(venv) Christophers-MacBook-Air:HomekitPythonPlaying christopheryoung$ python3 -m homekit.pair -d 1A:EA:89:9D:04:AF -p ${Device Setup Code} -a Eve -f test_report.json Could not open "test_report.json" because it does not exist

Created the empty file test_report.json

(venv) Christophers-MacBook-Air:HomekitPythonPlaying christopheryoung$ python3 -m homekit.pair -d 1A:EA:89:9D:04:AF -p ${Device Setup Code} -a Eve -f test_report.json Cannot parse "test_report.json" as JSON file (venv) Christophers-MacBook-Air:HomekitPythonPlaying christopheryoung$

jlusiardi commented 5 years ago

hi @netmanchris, ok sorry for the troubles. I updated the wiki with a new step 3 to create the controller storage. The parameters for the commands should be explained in the help for each command (--help).

netmanchris commented 5 years ago

@jlusiardi No trouble at all! I'm happy to be able to go through this and help highlight issues in the docs and help to improve them!

Is there a way for me to contribute to the Wiki? GITHUB doesn't have a great way to contribute to docs through a GIT workflow. :)

There's also a spelling mistake on step

Step 5. Paired Identity

(venv) Christophers-MacBook-Air:HomekitPythonPlaying christopheryoung$ python3 -m homekit.**identitfy -**f test_report.json -a Eve
/Users/christopheryoung/PycharmProjects/HomekitPythonPlaying/venv/bin/python3: No module named homekit.identitfy

Should be homekit.identity.

What's the best way to be able to send device specific test reports back to the project? I'm thinking maybe a device test template that gets filled out by contributors and posted to a tested_devices folder? A completed template looks something like this based on the current test plan.

Step 1 Discovery

Command - (venv) Christophers-MacBook-Air:HomekitPythonPlaying christopheryoung$ python -m homekit.discover > pairingtest.txt

Output

Name: Eve Light Strip 8606._hap._tcp.local. Url: http_impl://10.101.0.175:80 Configuration number (c#): 1 Feature Flags (ff): Supports HAP Pairing (Flag: 1) Device ID (id): 1A:EA:89:9D:04:AF Model Name (md): Eve Light Strip 20EAS9901 Protocol Version (pv): 1.1 State Number (s#): 1 Status Flags (sf): Accessory has not been paired with any controllers. (Flag: 1) Category Identifier (ci): Lightbulb (Id: 5)

Step 2. unpaired identify

(venv) Christophers-MacBook-Air:HomekitPythonPlaying christopheryoung$ python3 -m homekit.identify -d 1A:EA:89:9D:04:AF

Result:

Eve Light strip flashes

Step 3. Initialize Controller Storage

Created the file test_report.json Note: Must contain {} in the file to be a valid JSON syntax.

Step 4. Pairing

Note ${Device Setup Code} is the Homekit pairing code in format XXX-XX-XXX where X is a digit between 0 and 9.

(venv) Christophers-MacBook-Air:HomekitPythonPlaying christopheryoung$ python3 -m homekit.pair -d 1A:EA:89:9D:04:AF -p 126-22-521 -a Eve -f test_report.json Pairing for "Eve" was established.

Contents of test_report.json

{
  "Eve": {
    "AccessoryPairingID": "1A:EA:89:9D:04:AF",
    "AccessoryLTPK": "4abc5178fb710785ea00241cb85e0875e7ec21e3ec0149f6d24469c077c4b329",
    "iOSPairingId": "2d1feb16-1f94-4048-92f3-bae4c3fedd31",
    "iOSDeviceLTSK": "8748696d41490110155fd0f9807ac925f6947d441226581d30b1363b12c9ad76",
    "iOSDeviceLTPK": "9999a484a03687cc3bd1d03d95e940ac9036a891c016f4d17abfcdde558ce91b",
    "AccessoryIP": "10.101.0.175",
    "AccessoryPort": 80,
    "Connection": "IP",
    "accessories": [
      {
        "aid": 1,
        "services": [
          {
            "type": "0000003E-0000-1000-8000-0026BB765291",
            "iid": 1,
            "characteristics": [
              {
                "type": "00000014-0000-1000-8000-0026BB765291",
                "iid": 2,
                "perms": [
                  "pw"
                ],
                "format": "bool"
              },
              {
                "type": "00000020-0000-1000-8000-0026BB765291",
                "iid": 3,
                "value": "Eve Systems",
                "perms": [
                  "pr"
                ],
                "format": "string"
              },
              {
                "type": "00000021-0000-1000-8000-0026BB765291",
                "iid": 4,
                "value": "Eve Light Strip 20EAS9901",
                "perms": [
                  "pr"
                ],
                "format": "string"
              },
              {
                "type": "00000023-0000-1000-8000-0026BB765291",
                "iid": 5,
                "value": "Eve Light Strip 8606",
                "perms": [
                  "pr"
                ],
                "format": "string"
              },
              {
                "type": "00000030-0000-1000-8000-0026BB765291",
                "iid": 6,
                "value": "HV40H1A00544",
                "perms": [
                  "pr"
                ],
                "format": "string"
              },
              {
                "type": "00000052-0000-1000-8000-0026BB765291",
                "iid": 7,
                "value": "1.0.5",
                "perms": [
                  "pr"
                ],
                "format": "string"
              },
              {
                "type": "00000053-0000-1000-8000-0026BB765291",
                "iid": 8,
                "value": "1",
                "perms": [
                  "pr"
                ],
                "format": "string"
              }
            ]
          },
          {
            "type": "000000A2-0000-1000-8000-0026BB765291",
            "iid": 9,
            "characteristics": [
              {
                "type": "00000037-0000-1000-8000-0026BB765291",
                "iid": 10,
                "value": "1.1.0",
                "perms": [
                  "pr"
                ],
                "format": "string"
              }
            ]
          },
          {
            "type": "00000043-0000-1000-8000-0026BB765291",
            "iid": 11,
            "primary": true,
            "characteristics": [
              {
                "type": "00000025-0000-1000-8000-0026BB765291",
                "iid": 12,
                "perms": [
                  "pw",
                  "pr",
                  "ev"
                ],
                "format": "bool",
                "value": true
              },
              {
                "type": "00000023-0000-1000-8000-0026BB765291",
                "iid": 13,
                "perms": [
                  "pr"
                ],
                "format": "string",
                "value": "Eve Light Strip"
              },
              {
                "type": "00000008-0000-1000-8000-0026BB765291",
                "iid": 14,
                "perms": [
                  "pw",
                  "pr",
                  "ev"
                ],
                "format": "int",
                "value": 100,
                "unit": "percentage",
                "minValue": 0,
                "maxValue": 100,
                "minStep": 1
              },
              {
                "type": "00000013-0000-1000-8000-0026BB765291",
                "iid": 15,
                "perms": [
                  "pw",
                  "pr",
                  "ev"
                ],
                "format": "float",
                "value": 3,
                "unit": "arcdegrees",
                "minValue": 0,
                "maxValue": 360,
                "minStep": 1
              },
              {
                "type": "0000002F-0000-1000-8000-0026BB765291",
                "iid": 16,
                "perms": [
                  "pw",
                  "pr",
                  "ev"
                ],
                "format": "float",
                "value": 6,
                "unit": "percentage",
                "minValue": 0,
                "maxValue": 100,
                "minStep": 1
              },
              {
                "type": "E863F11A-079E-48FF-8F27-9C2605A29F52",
                "iid": 17,
                "perms": [
                  "pr"
                ],
                "format": "uint32",
                "value": 0,
                "minValue": 0,
                "maxValue": 4294967295,
                "minStep": 1
              }
            ]
          },
          {
            "type": "E863F007-079E-48FF-8F27-9C2605A29F52",
            "iid": 18,
            "hidden": true,
            "characteristics": [
              {
                "type": "E863F131-079E-48FF-8F27-9C2605A29F52",
                "iid": 19,
                "perms": [
                  "pr"
                ],
                "format": "tlv8",
                "value": ""
              },
              {
                "type": "E863F11D-079E-48FF-8F27-9C2605A29F52",
                "iid": 20,
                "perms": [
                  "pw"
                ],
                "format": "tlv8"
              },
              {
                "type": "E863F11C-079E-48FF-8F27-9C2605A29F52",
                "iid": 21,
                "perms": [
                  "pw"
                ],
                "format": "data"
              },
              {
                "type": "E863F116-079E-48FF-8F27-9C2605A29F52",
                "iid": 22,
                "perms": [
                  "pr"
                ],
                "format": "data",
                "value": ""
              },
              {
                "type": "E863F117-079E-48FF-8F27-9C2605A29F52",
                "iid": 23,
                "perms": [
                  "pr"
                ],
                "format": "data",
                "value": ""
              },
              {
                "type": "E863F121-079E-48FF-8F27-9C2605A29F52",
                "iid": 24,
                "perms": [
                  "pw"
                ],
                "format": "data"
              }
            ]
          }
        ]
      }
    ]
  }
}

Step 5. Paired Identity

(venv) Christophers-MacBook-Air:HomekitPythonPlaying christopheryoung$ python3 -m homekit.identitfy -f test_report.json -a Eve
/Users/christopheryoung/PycharmProjects/HomekitPythonPlaying/venv/bin/python3: No module named homekit.identitfy

Fixed spelling mistake of command from Wiki

(venv) Christophers-MacBook-Air:HomekitPythonPlaying christopheryoung$ python3 -m homekit.identify -f test_report.json -a Eve

Results

Eve Light Strip flashes

Step 6. Get Accessories

(venv) Christophers-MacBook-Air:HomekitPythonPlaying christopheryoung$ python3 -m homekit.get_accessories -f test_report.json -a Eve
1.1: >accessory-information<
  1.2:  () >identify< [pw]
  1.3: Eve Systems () >manufacturer< [pr]
  1.4: Eve Light Strip 20EAS9901 () >model< [pr]
  1.5: Eve Light Strip 8606 () >name< [pr]
  1.6: HV40H1A00544 () >serial-number< [pr]
  1.7: 1.0.5 () >firmware.revision< [pr]
  1.8: 1 () >hardware.revision< [pr]
1.9: >service<
  1.10: 1.1.0 () >version< [pr]
1.11: >lightbulb<
  1.12: True () >on< [pw,pr,ev]
  1.13: Eve Light Strip () >name< [pr]
  1.14: 100 () >brightness< [pw,pr,ev]
  1.15: 3 () >hue< [pw,pr,ev]
  1.16: 6 () >saturation< [pw,pr,ev]
  1.17: 0 () >Unknown Characteristic E863F11A-079E-48FF-8F27-9C2605A29F52< [pr]
1.18: >Unknown Service: E863F007-079E-48FF-8F27-9C2605A29F52<
  1.19:  () >Unknown Characteristic E863F131-079E-48FF-8F27-9C2605A29F52< [pr]
  1.20:  () >Unknown Characteristic E863F11D-079E-48FF-8F27-9C2605A29F52< [pw]
  1.21:  () >Unknown Characteristic E863F11C-079E-48FF-8F27-9C2605A29F52< [pw]
  1.22:  () >Unknown Characteristic E863F116-079E-48FF-8F27-9C2605A29F52< [pr]
  1.23:  () >Unknown Characteristic E863F117-079E-48FF-8F27-9C2605A29F52< [pr]
  1.24:  () >Unknown Characteristic E863F121-079E-48FF-8F27-9C2605A29F52< [pw]
(venv) Christophers-MacBook-Air:HomekitPythonPlaying christopheryoung$

jlusiardi commented 5 years ago

Hey, I copied the wiki contents into the doc folder. Now everyone can make PR like for normal project content. So should be easier to work on that.

netmanchris commented 5 years ago

@jlusiardi Awesome.

1 For submission of individual device tests, How would you like to see that? I'm thinking the following

New directory under doc of "tested_devices"
each device should have a single page with the device output for that specific device.

2 Should we also include the version of the library it was tested with? Thinking this is probably a good way to track changes in device behavior over time.

3 For PR with test data, guessing that we should be PR'ing against a docs branch? Can you confirm? Trying to get involved but totally respecting that I'm very new to the project :)

jlusiardi commented 5 years ago

1) sounds great! 2) yes, including the version is a great idea to track improvement 3) can you elaborate a little what the advantage of a special branch would be?

netmanchris commented 5 years ago

1 and 2. Perfect.

3 - No specific advantage - just guessing based on the PR for the new device categories. :)

jlusiardi commented 5 years ago

3) such a report of a device is passiv data. So it does not change the code and can be merged into master directly IMHO

netmanchris commented 5 years ago

Works for me!

netmanchris commented 5 years ago

@jlusiardi Added PR https://github.com/jlusiardi/homekit_python/pull/137 with test results for Eve Light Strip. Can you please have a look and let me know what else we want to get added to the test_results? If there's more info we want to grab I'm happy to do it but wanted to get that list solid now before so I don't have to go back and add to devices in the future.

jlusiardi commented 5 years ago

@netmanchris I just had a brief look at https://github.com/netmanchris/homekit_python/blob/master/doc/tested_devices/Eve%20Light%20Strip.md. Since there is pretty much input/output stuff involved it would be better readable if all IO related stuff would be surrounded by ```?

Another interesting aspect: since the accessories do not all have the same characteristics, how can a test for put_characteristic.py look like? Same for events? I guess I should update the HomeKit-Accessory-Testing-Guide.md a little

netmanchris commented 5 years ago

Adding the ``` . for proper input/ouput rendering.

I'm still learning the code base, so I'm not sure I have an opinion on what a test for a specific accessory type would look like. I do think that in an ideal world we should be able to capture all charecteristics reported by an individual accessory even if this is a custom characteristic specifically for that device only. I've pushed to https://github.com/netmanchris/homekit_python/blob/master/doc/tested_devices/Eve%20Light%20Strip.md with the updates. Feel free to comment or ask for modifications. Once we get a solid output. I'll send a new PR.

netmanchris commented 5 years ago

Hey @jlusiardi

Can you please take a look at the following? I ended up creating a jupyter notebook which might make it easier to duplicate this. Output gets exported as the following markdown with a bit of manual fixing of some of the formating.

https://github.com/netmanchris/homekit_python/blob/master/doc/tested_devices/Eve%20Light%20Strip.md

Hoping we've got a winner here. :)

jlusiardi commented 5 years ago

look great! Step 2 has some minor issue in one heading: result **✔ ** Can I see the jupyter notebook? I guess that might be a good thing to add?

netmanchris commented 5 years ago

Here's the jupyter notebook.

https://github.com/netmanchris/homekit_python/blob/6aa2a571985557599c7a3550ea9817658adc70e7/doc/tested_devices/Homekit%20Device%20Test%20Template.ipynb

The formating is a little off. Not sure if you're familiar with Jupyter. The "!" in front of the command allows me to run a terminal command from within the python cel. So the actual template is a bit off from the MD rendered version ( hence a bit of manual fixing of the formatting ).

If you look at the response of step 1, you'll also see that there are a LOT more devices showing up in that step than what is shown in the MD version. Essentially just deleted them out. :)

I'm fine with adding the notebook, but I would suggest that anyone contributing test results would still have to manually remove unessential information for a specific device, including removing the HomeKit from the file. Does that make sense?

netmanchris commented 5 years ago

Added Jupyter Notebook and MD file for the Eve Energy Strip as well in the same folder. The nice thing about using the Notebook format is that it should be easy to rerun the test for a given device against new versions of the library.

jlusiardi commented 5 years ago

When I see the lot of devices (most paired) I think we should add filtering to the discover functions. At least a filter to hide all paired devices on demand.

netmanchris commented 5 years ago

That would be useful for sure. I'm actually seeing a ton of paired devices showing up within HomeAssistant as well. I'm guessing that it might be using the same discovery component and showing the same results? If so, the filtering of unpaired devices could be useful beyond testing as well. Should this be opened as a new issue?

netmanchris commented 5 years ago

Looks like we're getting ready to close this issue and start adding more tested device output to the repo. If there are any specific devices that need testing, feel free to let me know and, assuming I have it, I can prioritize.

jlusiardi commented 5 years ago

I opened a new issue (#139) for filtering, merged your request and did some rework (y).

Great work, thank you! Looking forward to the other device reports!

jlusiardi / homekit_python

Device test plan #53

Step 1 Discovery

Output

Step 2. unpaired identify

Result:

Step 3. Pairing

Step 5. Paired Identity

Step 1 Discovery

Output

Step 2. unpaired identify

Result:

Step 3. Initialize Controller Storage

Step 4. Pairing

Contents of test_report.json

Step 5. Paired Identity

Results

Step 6. Get Accessories