Virgin Media (GB) recording capacity

[callumeveratt]

I'm not sure this ever worked on the original Arris integration (certainly for me the value was always unavailable for recording_capacity) but the following appears in the log for getting recording capacity in GB:

Logger: lghorizon.lghorizon_api Source: /usr/local/lib/python3.10/site-packages/backoff/_common.py:120 First occurred: 10:39:10 (1 occurrences) Last logged: 10:39:10 Giving up _do_api_call(...) after 3 tries (lghorizon.exceptions.LGHorizonApiConnectionError: Unable to call https://prod.spark.virginmedia.com/eng/web/recording-service/customers/xxx_gb/quota. Error:501 Server Error: Not Implemented for url: https://prod.spark.virginmedia.com/eng/web/recording-service/customers/xxx_gb/quota)

[Sholofly]

@callumeveratt I think this works as intended. For me there's no way to determine if cloud recording is available but to query that url. If it gives me a 501 i know it isn't avaiable. I presume you don't have a sensor at all anymore for recording capacity?

[caraar12345]

It looks like VM sends this over MQTT as a percentage --

Topic: {household_id}/{box_id}/localRecordings/capacity Payload: {"messageType":"CPE.capacity","version":"1.0.7","used":97}

[Sholofly]

Ah m8 thats brilliant information. Until now i was only supporting cloud recordings. In this way I can add a sensor for local recordings. How did you find out?

[caraar12345]

All the investigation I've been doing has been just by logging into https://virgintvgo.virginmedia.com with the Chrome dev tools open and watching the calls it makes.

In particular, in the call to mqtt, which is the websocket, you can see the MQTT messages either way!

[Sholofly]

Cool, I'm going to try to add this to the api. Thanks!

[Sholofly]

BTW, can you find out of anything in your account data states that recording is enabled and if this is network or local?

[callumeveratt]

Ignore that - I imagine it's not specified anywhere for Virgin GB as channel owners have cold feet over cloud recording - so it's never been enabled in the UK from what I gather

[callumeveratt]

Would definitely be cool to use the local recording data where applicable though :-)

[caraar12345]

Will also have a dig around -- but with Virgin, I can confidently say it'll always be local as we don't have Cloud DVR!

@callumeveratt I think remote recording in that context is the ability to remotely instruct your box to record something rather than recording to VM's servers

[callumeveratt]

@caraar12345 - Yeah you're right, I got ahead of myself for a second. It is the ability to remote record, not cloud record.

[caraar12345]

Ha! Impressive timing from both of us 😁

Does look like there's something about local DVR though -- localDvrAvailable. I believe networkDvrAvailable refers to being able to play back local recordings on a mobile device.

Unrelated tidbit, I do wonder where they get 'gender' from, as I may be somewhat effeminate, but I've definitely never identified myself as female to anyone 😆

{
    "oespToken": "",
    "accessToken": "",
    "householdId": "{customer_id}_gb",
    "username": "{customer_id}",
    "locationId": "18",
    "originalLocationId": "18",
    "timeToIdleSeconds": 7200,
    "customer": {
        "givenName": "Aaron",
        "countryCode": "GB",
        "gender": "female",
        "status": [],
        "smartCardId": "",
        "physicalDeviceId": "{box_id}",
        "stbType": "EOS",
        "networkDvrAvailable": true,
        "replayOTTOnlyAvailable": false,
        "localDvrAvailable": true,
        "householdId": "{customer_id}_gb",
        "sharedProfileId": "",
        "replayTvEntitled": true,
        "replayTvAvailable": true,
        "replayTvMinimumBroadcastTime": 0,
        "replayTvOptedIn": true,
        "offlineViewingEntitled": true,
        "profileIds": [
            ""
        ],
        "oboProvisioned": true,
        "customerId": "{customer_id}",
        "purchaseEntitlementsToken": "",
        "isSkLite": false
    },

[Sholofly]

From what call is this data coming from?

[caraar12345]

Oops - https://prod.oesp.virginmedia.com/oesp/v4/GB/eng/web/session

[Sholofly]

Are you guys (or guy and lady) able to update and can you check if every box have an attribute with recording capacity?

[caraar12345]

Will give it a shot; unfortunately I'm still contending with the config flow issue (hopefully this restart will sort it!?)

[ChrisThomas42]

@Sholofly On VirginMedia GB, I have 1 device and 1 entity. No sign of recording capacity anywhere.

[caraar12345]

If you click the box entity, does it show in attributes?

[Sholofly]

No you should check the developer tools

[callumeveratt]

Yep - there for me. Null though - is that expected at the mo?

[Sholofly]

Depends, do you have local recordings?

[callumeveratt]

Yep - have recordings on the box and on TVGo I can see the MQTT request @caraar12345 gave

{"messageType":"CPE.capacity","version":"1.0.7","used":72}

[Sholofly]

ok, i that case it doesn't work yet. I have to add more logging to find out where the problem is. Do you have debugging enabled in your component?

[callumeveratt]

Nope - but if you need to add some more logging I can get it turned on in the meantime

[Sholofly]

OK stupid. I didnt't commit the part to subscribe to the actual mqtt topic :( Can you manually update your manifest.json to

"requirements": [
    "lghorizon>=0.5.13"
  ]

And then reboot of course ;)

[Sholofly]

@Sholofly On VirginMedia GB, I have 1 device and 1 entity. No sign of recording capacity anywhere.

The local recordings are an attribute of the box. It's not an account thing like the cloud recordings.

[ChrisThomas42]

After changing the entry 0.5.12 to 0.5.13 and restarting, I no longer have an entry for recording capacity listed in states. I can see all my local recodings in the integration.

[Sholofly]

Do you see recording capacity of each box in the box attributes?

[callumeveratt]

I updated the manifest also - the recording_capacity did disappear originally but it came back but sadly still null

[Sholofly]

Can you check the logs for mqtt messages if you set it to debug?

[callumeveratt]

Just trawling though, so far found these

2022-12-21 15:05:41.002 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Sending SUBSCRIBE (d0, m7) [(b'201434117_gb/+/localRecordings', 0)]

2022-12-21 15:05:41.003 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Sending SUBSCRIBE (d0, m8) [(b'201434117_gb/+/localRecordings/capacity', 0)]

[callumeveratt]

I see a SUBACK shortly after - but not much else

[Sholofly]

If you visit the web page now and force the webpage to retrieve your capacity. Is your HA instance getting an update too?

[ChrisThomas42]

I've just re-enabled debug and restarted, SOOO much stuff in the logs I need to go through more fully, but this is a start

2022-12-21 15:16:54.468 INFO (SyncWorker_2) [lghorizon.lghorizon_api] Executing API call to https://prod.spark.virginmedia.com/eng/web/recording-service/customers/105210861_gb/quota 2022-12-21 15:16:54.567 DEBUG (SyncWorker_2) [lghorizon.lghorizon_api] Step 1 - Get Authorization data 2022-12-21 15:16:54.739 DEBUG (SyncWorker_2) [lghorizon.lghorizon_api] Step 2 - Get Authorization cookie 2022-12-21 15:16:54.961 DEBUG (SyncWorker_2) [lghorizon.lghorizon_api] Step 3 - Login 2022-12-21 15:16:55.194 DEBUG (SyncWorker_2) [lghorizon.lghorizon_api] Step 4 - Follow redirect 2022-12-21 15:16:55.305 DEBUG (SyncWorker_2) [lghorizon.lghorizon_api] Step 5 - Extract auth code 2022-12-21 15:16:55.305 DEBUG (SyncWorker_2) [lghorizon.lghorizon_api] Auth code: UBoZFR, Auth state: f00a129d-1974-44e4-842c-3f1b478c5ab4 2022-12-21 15:16:55.305 DEBUG (SyncWorker_2) [lghorizon.lghorizon_api] Step 6 - Post auth data with valid code 2022-12-21 15:16:56.001 INFO (SyncWorker_2) [lghorizon.lghorizon_api] Backing off _do_api_call(...) for 1.1s (lghorizon.exceptions.LGHorizonApiConnectionError: Unable to call https://prod.spark.virginmedia.com/eng/web/recording-service/customers/105210861_gb/quota. Error:501 Server Error: Not Implemented for url: https://prod.spark.virginmedia.com/eng/web/recording-service/customers/105210861_gb/quota)

[callumeveratt]

@ChrisThomas42 I believe that's the cloud recording rather than local - and that error is expected with GB as no cloud recording

@Sholofly - I forced a refresh of my TV Go (if that's what you meant) and I couldn't see anything in the logs around MQTT and the localCapacity topic

[Sholofly]

Ok just one last try for today: can you update your manifest to v0.5.14?

[callumeveratt]

Tried - didn't work originally but thought I'd relink as a test also but actually seems the integration can't connect now:

Logger: custom_components.lghorizon.config_flow Source: custom_components/lghorizon/config_flow.py:81 Integration: LG Horizon (documentation, issues) First occurred: 15:38:17 (4 occurrences) Last logged: 15:38:47 'LGHorizonAuth' object has no attribute 'accessToken'

[callumeveratt]

2022-12-21 15:40:24.721 DEBUG (SyncWorker_11) [lghorizon.lghorizon_api] Obtain Virgin GB mqtt token... 2022-12-21 15:40:24.725 ERROR (MainThread) [custom_components.lghorizon.config_flow] 'LGHorizonAuth' object has no attribute 'accessToken'

[caraar12345]

Oh crap, I’d bet you’ve maxed your session count. I had to leave it an hour and some started clearing out so I could log in again 😅

On Wed, 21 Dec 2022 at 15:40, callumeveratt @.***> wrote:

Tried - didn't work originally but thought I'd relink as a test also but actually seems the integration can't connect now:

Logger: custom_components.lghorizon.config_flow Source: custom_components/lghorizon/config_flow.py:81 Integration: LG Horizon (documentation https://github.com/Sholofly/lghorizon, issues https://github.com/Sholofly/lghorizon/issues) First occurred: 15:38:17 (4 occurrences) Last logged: 15:38:47 'LGHorizonAuth' object has no attribute 'accessToken'

— Reply to this email directly, view it on GitHub https://github.com/Sholofly/lghorizon/issues/31#issuecomment-1361517485, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOYOFGR6K6NMMUXP2VJBQDWOMQFZANCNFSM6AAAAAATFPNJ4A . You are receiving this because you were mentioned.Message ID: @.***>

[callumeveratt]

Aha of course - I'll give it a little

[ChrisThomas42]

so this is a small extract from my log,

2022-12-21 15:38:22.873 INFO (SyncWorker_0) [lghorizon.lghorizon_api] Retrieving recordingcapacity... 2022-12-21 15:38:22.873 INFO (SyncWorker_0) [lghorizon.lghorizon_api] Executing API call to https://prod.spark.virginmedia.com/eng/web/recording-service/customers/*****_gb/quota 2022-12-21 15:38:22.945 DEBUG (SyncWorker_0) [lghorizon.lghorizon_api] Step 1 - Get Authorization data 2022-12-21 15:38:23.119 DEBUG (SyncWorker_0) [lghorizon.lghorizon_api] Step 2 - Get Authorization cookie 2022-12-21 15:38:23.142 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:23.146 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:23.399 DEBUG (SyncWorker_0) [lghorizon.lghorizon_api] Step 3 - Login 2022-12-21 15:38:23.579 DEBUG (SyncWorker_0) [lghorizon.lghorizon_api] Step 4 - Follow redirect 2022-12-21 15:38:23.699 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:23.707 DEBUG (SyncWorker_0) [lghorizon.lghorizon_api] Step 5 - Extract auth code 2022-12-21 15:38:23.707 DEBUG (SyncWorker_0) [lghorizon.lghorizon_api] Auth code: PJ363e, Auth state: 3fed465e-20e8-44a1-86ee-f190c6f72618 2022-12-21 15:38:23.707 DEBUG (SyncWorker_0) [lghorizon.lghorizon_api] Step 6 - Post auth data with valid code 2022-12-21 15:38:23.792 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:23.793 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:23.793 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:24.097 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:24.115 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:24.116 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:24.116 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:24.117 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:24.118 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:24.118 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received SUBACK 2022-12-21 15:38:24.266 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received PUBLISH (d0, q0, r0, m0), '******_gb/0QhZJ30DAj/status', ... (72 bytes) 2022-12-21 15:38:24.266 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received MQTT message. Topic: ******_gb/0QhZJ30DAj/status 2022-12-21 15:38:24.266 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Message: {'source': '0QhZJ30DAj', 'state': 'ONLINE_RUNNING', 'deviceType': 'HGO'} 2022-12-21 15:38:24.285 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received PUBREC (Mid: 14) 2022-12-21 15:38:24.286 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Sending PUBREL (Mid: 14) 2022-12-21 15:38:24.322 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received PUBCOMP (Mid: 14) 2022-12-21 15:38:24.402 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received PUBLISH (d0, q0, r1, m0), '******_gb/6d3aaf2f-721a-4163-a638-ffcbc46e3cbc/status', ... (110 bytes) 2022-12-21 15:38:24.403 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received MQTT message. Topic: ******_gb/6d3aaf2f-721a-4163-a638-ffcbc46e3cbc/status 2022-12-21 15:38:24.403 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Message: {'mac': '', 'ipAddress': '', 'state': 'OFFLINE', 'source': '6d3aaf2f-721a-4163-a638-ffcbc46e3cbc', 'deviceType': 'HGO'} 2022-12-21 15:38:24.421 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received PUBLISH (d0, q0, r1, m0), '******_gb/3C36E4-EOSSTB-003469169803/status', ... (152 bytes) 2022-12-21 15:38:24.421 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received MQTT message. Topic: ******_gb/3C36E4-EOSSTB-003469169803/status 2022-12-21 15:38:24.422 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Message: {'deviceType': 'STB', 'source': '3C36E4-EOSSTB-003469169803', 'state': 'ONLINE_STANDBY', 'mac': '**:**:', 'ipAddress': '192.168.*.*/255.255.255.0'} 2022-12-21 15:38:24.428 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Sending PUBLISH (d0, q2, r0, m15), 'b'******_gb/3C36E4-EOSSTB-003469169803'', ... (66 bytes) 2022-12-21 15:38:24.487 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received PUBREC (Mid: 15) 2022-12-21 15:38:24.488 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Sending PUBREL (Mid: 15) 2022-12-21 15:38:24.584 DEBUG (Thread-3 (_thread_main)) [lghorizon.models] Received PUBCOMP (Mid: 15) 2022-12-21 15:38:24.890 INFO (SyncWorker_0) [lghorizon.lghorizon_api] Backing off _do_api_call(...) for 0.8s (lghorizon.exceptions.LGHorizonApiConnectionError: Unable to call https://prod.spark.virginmedia.com/eng/web/recording-service/customers/******_gb/quota. Error:501 Server Error: Not Implemented for url: https://prod.spark.virginmedia.com/eng/web/recording-service/customers/******_gb/quota) 2022-12-21 15:38:25.686 INFO (SyncWorker_0) [lghorizon.lghorizon_api] Executing API call to https://prod.spark.virginmedia.com/eng/web/recording-service/customers/******_gb/quota

I think(hope) I've removed any potentially sensitive info???? There's more before and after.

[caraar12345]

Woohoo - I finally managed to get the config flow to work so I can chime in!

source_list: 
  - <many_channels>
media_content_type: episode
media_title: BBC Launcher
source: BBC Launcher
entity_picture_local: >-
  /api/media_player_proxy/media_player.virgin_tv?token=
play_mode: app
channel: BBC Launcher
title: BBC Launcher
image: >-
  https://cdn.metrological.com/lightning/apps/com.bbc.app.launcher/1.0.0-a646d9e/static/icon.png
recording_capacity: null
device_class: tv
entity_picture: >-
  https://cdn.metrological.com/lightning/apps/com.bbc.app.launcher/1.0.0-a646d9e/static/icon.png
friendly_name: Virgin TV
supported_features: 154497

No MQTT message is coming through with the recording_capacity which is just... weird.

Interesting note: https://prod.spark.virginmedia.com/eng/web/recording-service/customers/{household_id}/recordings?sort=time&sortOrder=desc&language=en does work. And the data there contains diskSpace for every item. And for me, adding them together gets 96.05% which is reasonable based on the MQTT message reporting 97% (always round up?).

So worst case: iterate through the recordings and add the disk spaces!???

[callumeveratt]

Apologies as been out this evening, just got back and managed to readd. Still no luck and either no value reported for recording or a null value

[Sholofly]

https://prod.spark.virginmedia.com/eng/web/recording-service/customers/{household_id}/recordings?sort=time&sortOrder=desc&language=en

@caraar12345 We have this call too. I results in a list AND a summary of quota at the end. Isn't that the case here?

[callumeveratt]

@Sholofly - just realized we all went quiet on this. I have a bit of time this evening so I can have a look into some of the calls ect and see if I can tweak it to work on my HA if you want :)

[Sholofly]

@callumeveratt Sorry, I'm very busy lately. Please let me know when you have time. Maybe we can figure this out soon :)

[ColinRobbins]

I've started to look at this, and think my findings are the same as above. In GB, the quota URL does not exists, but the recordings one works just fine.

Using the Virgin web interface, I've used both Chrome developer tools and burp to capture HTTP traffic and there is no call to quota, but the webpage does get the data somehow.

Looking at the JS pages (hard because of the obfuscation) there is code that looks for .quota and .occupied and so I assume it is getting in from somewhere.

I can see some MQTT calls in Chrome dev tools, but the messages are only 2 bytes of binary data as far as I can see. Maybe I'm not capturing the MQTT correctly - not sure, I not that experienced with MQTT.

I can't see any indication the code is iterating the recordings and adding it up.

Should we (at least) put some code that says "If GB don't make the quota call"?

[ColinRobbins]

Ahh, just found the following MQTT message...

00000000: aaaa aaaa  aaaa aaaa aaaa aaaa aaaa          XXXX-XXXXXX
00000001: aaaa aaaa  aaaa aaaa aaaa aaaa aaaa          XXXX-XXXXXX
00000002: aaaa aaaa  aaaa aaaa aaaa aaaa aaaa           XXXXX/loca
00000003: 6c52 6563 6f72 6469 6e67 732f 6361 7061  lRecordings/capa
00000004: 6369 7479 0066 7b22 6d65 7373 6167 6554  city.f{"messageT
00000005: 7970 6522 3a22 4350 452e 6361 7061 6369  ype":"CPE.capaci
00000006: 7479 222c 2276 6572 7369 6f6e 223a 2231  ty","version":"1
00000007: 2e30 2e37 222c 2275 7365 6422 3a36 7d    .0.7","used":6}

which contains the "used:6" - which is the correct - 6% disk used.

@Sholofly if you can give me a few pointers on what this might mean, and where to modify the code, I'll give it a go.

[Sholofly]

Sorry my time is limited nowadays so I try to respond as quickly as possible. Let me start to thank you for your dedication!

I'm from NL and the Ziggo boxes don't use local recordings but cloud recordings. That's why my capacity is exposed using the API. If the box want's to cummincate to the outside world it uses the MQTT protocol. I'm not sure if you're familiar with this protocol but its a massaging protocol where you can subscribe on topics to receive messages (or send messages with a topic of course)

More info: https://mqtt.org/

What you see in your example is the fact that a message is received on the /localRecordings/capacity topic. With the used percentage of your box.

I think the subscription already is in place (https://github.com/Sholofly/lghorizon-python/blob/2fa57cc4ea5decff549db3fa7f952ef9403bd018/lghorizon/models.py#L363)

And the processing is already in place too: https://github.com/Sholofly/lghorizon-python/blob/2fa57cc4ea5decff549db3fa7f952ef9403bd018/lghorizon/lghorizon_api.py#L349

So probably my cloud update is frustrating the localrecording, or there's something wrong in my processing.

Probably you can figure that one out. I'd really like to hep you, but since I don't have a UK account with local recordings, that's kind of difficult for me...

[ColinRobbins]

@Sholofly thank you for the quick reply, I think I can see how to make that work. I just have one more quick question if I can.

The MQTT subscription to local/capacity will presumably only send a message when the capacity changes.

Is there a way to force a "read" of the current status via MQTT?

I appreciate your help.

[Sholofly]

I'm not sure, probably you can inspect your browser for the answer. Since you saw the message appear, probably the previous message is the trigger? Or it's send automatically on subscription?