`KeyError: 'signed'` when trying to update example app from a separate Github repo

[Samlant]

Only posting here because I couldn't add "tufup" tag to my stack overflow question, located here.

0

I'm just trying to get my hands around a simple version of updating an app that I have its repo targets & metadata hosted on a separate Github repo. I got it working great when locally hosted per the tufup_example guide... The issue I'm running into is that whenever I start main.exe (created by following tufup_example on windows) is that (I think) the timestamp.json isn't correctly being deserialized...

I've ran through tuf's & tufup's docs and have been at this for a week, so now I'm earnestly asking for help or guidance on this. Am I overlooking something? Do I need to do something special to sign metadata now that it's hosted on Github?

Copy of my error is below, I hope it's just something simple I'm overlooking:

PS C:\Users\username\AppData\Local\Programs\my_app> .\main.exe
INFO:__main__:my_app 1.0
INFO:myapp:Trusted root metadata copied to cache.
DEBUG:tuf.ngclient._internal.trusted_metadata_set:Updating initial trusted root
DEBUG:tuf.ngclient._internal.trusted_metadata_set:Loaded trusted root v1
DEBUG:tuf.ngclient.fetcher:Downloading: https://github.com/username/updater/blob/main/metadata/2.root.json
DEBUG:tuf.ngclient._internal.requests_fetcher:Made new session ('https', 'github.com')
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): github.com:443
DEBUG:urllib3.connectionpool:https://github.com:443 "GET /username/updater/blob/main/metadata/2.root.json HTTP/1.1" 404 None
DEBUG:tuf.ngclient.updater:Local timestamp not valid as final: [Errno 2] No such file or directory: 'C:\\Users\\username\\AppData\\Local\\my_app\\update_cache\\metadata\\timestamp.json'
DEBUG:tuf.ngclient.fetcher:Downloading: https://github.com/username/updater/blob/main/metadata/timestamp.json
DEBUG:tuf.ngclient._internal.requests_fetcher:Reusing session ('https', 'github.com')
DEBUG:urllib3.connectionpool:Resetting dropped connection: github.com
DEBUG:urllib3.connectionpool:https://github.com:443 "GET /username/updater/blob/main/metadata/timestamp.json HTTP/1.1" 200 2323
DEBUG:tuf.ngclient.fetcher:Downloaded 6205 out of 16384 bytes
Traceback (most recent call last):
  File "tuf\api\serialization\json.py", line 37, in deserialize
  File "tuf\api\metadata.py", line 172, in from_dict
KeyError: 'signed'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "main.py", line 12, in <module>
  File "myapp\__init__.py", line 75, in main
  File "myapp\__init__.py", line 36, in update
  File "tufup\client.py", line 154, in check_for_updates
  File "tuf\ngclient\updater.py", line 133, in refresh
  File "tuf\ngclient\updater.py", line 346, in _load_timestamp
  File "tuf\ngclient\_internal\trusted_metadata_set.py", line 211, in update_timestamp
  File "tuf\api\metadata.py", line 263, in from_bytes
  File "tuf\api\serialization\json.py", line 40, in deserialize
tuf.api.serialization.DeserializationError: Failed to deserialize JSON
[11720] Failed to execute script 'main' due to unhandled exception!
PS C:\Users\username\AppData\Local\Programs\my_app> .\main.exe
INFO:__main__:my_app 1.0
DEBUG:tuf.ngclient._internal.trusted_metadata_set:Updating initial trusted root
DEBUG:tuf.ngclient._internal.trusted_metadata_set:Loaded trusted root v1
DEBUG:tuf.ngclient.fetcher:Downloading: https://github.com/username/updater/blob/main/metadata/2.root.json
DEBUG:tuf.ngclient._internal.requests_fetcher:Made new session ('https', 'github.com')
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): github.com:443
DEBUG:urllib3.connectionpool:https://github.com:443 "GET /username/updater/blob/main/metadata/2.root.json HTTP/1.1" 404 None
DEBUG:tuf.ngclient._internal.trusted_metadata_set:Updated timestamp v3
DEBUG:tuf.ngclient.fetcher:Downloading: https://github.com/username/updater/blob/main/metadata/timestamp.json
DEBUG:tuf.ngclient._internal.requests_fetcher:Reusing session ('https', 'github.com')
DEBUG:urllib3.connectionpool:Resetting dropped connection: github.com
DEBUG:urllib3.connectionpool:https://github.com:443 "GET /username/updater/blob/main/metadata/timestamp.json HTTP/1.1" 200 2321
DEBUG:tuf.ngclient.fetcher:Downloaded 6205 out of 16384 bytes
Traceback (most recent call last):
  File "tuf\api\serialization\json.py", line 37, in deserialize
  File "tuf\api\metadata.py", line 172, in from_dict
KeyError: 'signed'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "main.py", line 12, in <module>
  File "myapp\__init__.py", line 75, in main
  File "myapp\__init__.py", line 36, in update
  File "tufup\client.py", line 154, in check_for_updates
  File "tuf\ngclient\updater.py", line 133, in refresh
  File "tuf\ngclient\updater.py", line 346, in _load_timestamp
  File "tuf\ngclient\_internal\trusted_metadata_set.py", line 211, in update_timestamp
  File "tuf\api\metadata.py", line 263, in from_bytes
  File "tuf\api\serialization\json.py", line 40, in deserialize
tuf.api.serialization.DeserializationError: Failed to deserialize JSON
[5048] Failed to execute script 'main' due to unhandled exception!

I created a working app that would auto-update itself from a localhost server as described in the tufup_example, however when I created a new repo from github and hosted fresh files that had the metadata and target files pointing to that Github repo, I received the above error(s).

I tried creating several "clean" repos and clients but have not been able to figure out this error.

[dennisvang]

Hi @Samlant , thanks for taking an interest in tufup.

To answer this, we'll need a bit more detail regarding your configuration and workflow.

However, there is one thing you could check first, based on the KeyError: 'signed' above:

Did you inspect the contents of the timestamp.json file you are hosting on github? It should look similar to this:

{
 "signatures": [
  {
   "keyid": "****",
   "sig": "****"
  }
 ],
 "signed": {
  "_type": "timestamp",
  "expires": "2023-10-17T08:02:36Z",
  "meta": {
   "snapshot.json": {
    "version": 1
   }
  },
  "spec_version": "1.0.29",
  "version": 1
 }
}

Based on your stack trace, it looks like the "signed" field is missing.

If not, could you confirm that the "expires" value is in the future?

Could you also confirm that you are using the tufup-example app with all default settings? (except for the github url)

[Samlant]

Thanks @dennisvang , much appreciated!

Sometimes, pressure to do good by others is all you need. In this case, in my pursuit to give you a well-rounded answer, I found my issue (when debugging line by line via VS code).

When the app checks for updates and JSONDeserializer.deserialize() is called for the timestamp.json file, the raw data is the entire Github webpage, so using this stack overflow post about using https://raw.githubusercontent.com rather than github.com..., this grabbed solely the contents of the JSON file, rather than a response containing it and other elements of the page!

Using the https://raw.githubusercontent.com address path makes the app run successfully in its production env; thank you very much for all your work on tufup, I'll reach out if I have any other questions!

To just answer your questions since it was already written out:

Confirming that the timestamp.json hosted on Github is similar to your above example, and that the "expires" value reads future at the time the client ran.

Double-checked & confirming also that I'm using default code per the example. However, note that I use VS Code, so I need to add "src" to myapp imports, as well as to the app_version_attr in repo_init.py (so it reads app_version_attr = "src.myapp.__version__").

I've received the above "failed to deserialize JSON" error both when ran via main.exe and within VS code.

[dennisvang]

@Samlant Thanks for coming back with such a detailed explanation, that will certainly help others as well. :-)

I'll close the issue now.

[dennisvang]

When the app checks for updates and JSONDeserializer.deserialize() is called for the timestamp.json file, the raw data is the entire Github webpage, so using this stack overflow post about using https://raw.githubusercontent.com rather than github.com..., this grabbed solely the contents of the JSON file, rather than a response containing it and other elements of the page!

Using the https://raw.githubusercontent.com/ address path makes the app run successfully in its production env

@Samlant It would also be great if you could post this as an answer to your question on stackoverflow.