How to serve repository in production, on remote server.

[dennisvang]

Could you elaborate? The question is not exactly clear to me.

[its-monotype]

It turns out that we have a repository that the client fetch to get updates, during development it is served on the localhost, we run it: python -m http.server -d temp/repository. But what to do when we need to deploy this repository to a remote server, such as Heroku?

[its-monotype]

@dennisvang

[dennisvang]

@its-monotype Your remote server would need to serve the tuf metadata and targets content, just like the local server does. So that would be the content of the temp/repository folder in the example.

Your client app would then need to know the urls at which the metadata and targets are served. In the example, these urls are specified in myapp.settings.

Depending on the expiration dates you choose for your tuf metadata files, your remote server may need to be configured so it can automatically sign the snapshot and timestamp metadata at regular intervals. You could use e.g. a cron-job for this. Read the TUF docs for more details on metadata signing, expiration, etc.

How this should be implemented on Heroku is an implementation detail, that depends on your own preferences.

[its-monotype]

@its-monotype Your remote server would need to serve the tuf metadata and targets content, just like the local server does. So that would be the content of the temp/repository folder in the example.

Your client app would then need to know the urls at which the metadata and targets are served. In the example, these urls are specified in myapp.settings.

Depending on the expiration dates you choose for your tuf metadata files, your remote server may need to be configured so it can automatically sign the snapshot and timestamp metadata at regular intervals. You could use e.g. a cron-job for this. Read the TUF docs for more details on metadata signing, expiration, etc.

How this should be implemented on Heroku is an implementation detail, that depends on your own preferences.

Thanks for feedback ❤️

[its-monotype]

@dennisvang In my project I use NestJS as a back-end api and the web front-end also communicates with it, and I have a lot of Python desktop applications that also communicate with the NestJS back-end api, and in each, I need to implement auto-updates

If I will use the implementation as in tufup-example, it turns out that I will have 1 repository for each application and all the logic for creating a TUF repository will be stored in the separate Github repository of each desktop application and when a new update is released, I will need to upload the updated metadata to S3, and a new target from the temp folder, but how can I then separate all these TUF repositories on S3?

Suppose I will return update files and metadata with the NestJS backend api to which the desktop app will access these files. Then let's say I make a worker that periodically downloads snapshot and timestamp metadata and private keys for signing them and will re-sign timestamp and snapshot metadata, to do this, the worker must use python and it must have tufup installed and it must also have .tufup- repo-config, but where can I get it if I create a repository not in a worker (github repo) but in a desktop app (github repo)?

And then I don’t understand how the key re-signing process takes place, what do I need for this do in code?

Also, I'm worried about where to store the keys for re-signing the snapshot and timestamp metadata that the worker needs, also on S3 along with metadata and targets?

Also in tufup-example it says

"For convenience, this example uses a single key pair for all TUF roles, and the private key is unencrypted and stored locally. This approach is not safe and should not be used in production."

But what decision do I need to make for production the most important question is how to do it, how to create keys, and how to encrypt private keys?

I understand that in production you need to use a pair of private and public keys for each of the root, targets, snapshot, and timestamp roles. How should this work and how best to organize it?

I would like to hear answers/advice, maybe it's obvious to you, but it's not clear to me at all 🙏

[dennisvang]

I would like to hear answers/advice, maybe it's obvious to you, but it's not clear to me at all

@its-monotype No, you're right, these things can be quite difficult to understand. I'll try to cook up an answer later, because I'm a little short on time right now.

[dennisvang]

@dennisvang In my project I use NestJS as a back-end api and the web front-end also communicates with it, and I have a lot of Python desktop applications that also communicate with the NestJS back-end api, and in each, I need to implement auto-updates

@its-monotype As I'm not familiar with NestJS, I'll try to give a generally applicable answer.

If I will use the implementation as in tufup-example, it turns out that I will have 1 repository for each application and all the logic for creating a TUF repository will be stored in the separate Github repository of each desktop application and when a new update is released, I will need to upload the updated metadata to S3, and a new target from the temp folder, but how can I then separate all these TUF repositories on S3?

You could create a separate folder on S3 for each desktop app, and put the corresponding metadata and targets folders in there. Each desktop app then needs to know the urls for their own metadata and targets folders. These urls are passed into the desktop app's tufup.client.Client when you initialize it.

Suppose I will return update files and metadata with the NestJS backend api to which the desktop app will access these files. Then let's say I make a worker that periodically downloads snapshot and timestamp metadata and private keys for signing them and will re-sign timestamp and snapshot metadata, to do this, the worker must use python and it must have tufup installed and it must also have .tufup- repo-config, but where can I get it if I create a repository not in a worker (github repo) but in a desktop app (github repo)?

To re-sign the timestamp and snapshot metadata, you only need the following:

• python with the latest tufup installed
• the latest metadata files for all roles (e.g. from S3), in a local metadata directory
• the private keys for the timestamp and snapshot roles should be available to your worker, perhaps on the instance itself, or in a private S3 bucket (the other roles should use offline keys, according to the TUF faq)
• the .tufup-repo-config file (note that the repo_dir in the config file should point to the local parent directory of your metadata dir)

So, you don't need to have the actual app source code, and you don't need the content of the targets dir.

The repo config file should hardly ever change, so you can copy it from your local development system and deploy to your worker, together with the other required components.

And then I don’t understand how the key re-signing process takes place, what do I need for this do in code?

On the worker, you can use the tufup command line interface (e.g. in a bash script), from the directory that contains the .tufup-repo-config, to re-sign the files:

tufup sign snapshot <path to dir containing your private keys>

This will sign snapshot and timestamp in one go, using the expiration periods specified in the repo config file.

Note: use tufup -h or tufup <command> -h for command line instructions.

Alternatively, you could write a python script to do the re-signing. Have a look at the cli implementation for inspiration: _cmd_sign.

Also, I'm worried about where to store the keys for re-signing the snapshot and timestamp metadata that the worker needs, also on S3 along with metadata and targets?

You could put your keys in a private S3 bucket, and automatically copy them to your worker upon deployment.

Alternatively, perhaps it would be possible to set up an action in a private github repo, using a schedule event and encrypted secrets. However, I haven't tried this myself, so I'm not entirely sure it can be done.

Also in tufup-example it says

"For convenience, this example uses a single key pair for all TUF roles, and the private key is unencrypted and stored locally. This approach is not safe and should not be used in production."

But what decision do I need to make for production the most important question is how to do it, how to create keys, and how to encrypt private keys?

I understand that in production you need to use a pair of private and public keys for each of the root, targets, snapshot, and timestamp roles. How should this work and how best to organize it?

Most importantly: keep the private keys private. See the web for more information on handling public/private keys in general.

Decisions about the number of keys to use, and how to store them, are up to you. However, you can see the TUF faq and the rest of the TUF docs for guidance.

To get an impression of the decisions involved (albeit for a much more complex use-case), you could have a look at PEP-458.

Encrypting a key means it is password protected. Note that the online keys for your worker should probably not be encrypted, because you cannot (easily) enter a password on the worker.

The keys for the example app are created automatically by repo_init.py, so you can have a look at how it's done. The key names and encryption settings are defined in the settings.py. You can modify that to meet your needs.

Alternatively, you can create/add/replace keys manually, using the tufup keys command line tool.

As with all the tufup command line tools, you can use -h to see instructions, so e.g. tufup keys -h will tell you what to do.

I would like to hear answers/advice, maybe it's obvious to you, but it's not clear to me at all pray

[dennisvang]

One more point that may not be immediately clear:

The update-repository files and folders do not have to be placed inside your app's project directory. In fact, it may be more convenient to create a separate repository for your app update (meta)data, as long as you tell tufup where to find everything. This would create a clear separation between the client-side and the repo-side.

In fact, I guess you could even choose to serve your metadata files from a github repo, and serve the targets files from e.g. S3.

In the tufup-example, the update-repo scripts/tools/directories are placed in the same project directory as the dummy app, but this is just for convenience, to have a self contained example.

[its-monotype]

Thanks for the explanation.

The update-repository files and folders do not have to be placed inside your app's project directory. In fact, it may be more convenient to create a separate repository for your app update (meta)data, as long as you tell tufup where to find everything. This would create a clear separation between the client-side and the repo-side.

To separate the repository from the source code of the app, do you mean targets/metadata/scripts to be placed in a separate dir and for each app to make such a dir?

But, I think, the solution with a repo in temp dir which is located inside the app's dir (as it is done in tufup-example) looks more convenient, because if we separate the repo from the app source code, then in order to create a new target we will have to do an extra step, namely, move the files compiled by pyinstaller to a separate TUF repo dir and then create a new target here. And also in my case, if I have many apps, then I will have to create many TUF repo dirs for all the apps.

In fact, I guess you could even choose to serve your metadata files from a github repo, and serve the target files from e.g. S3.

By serving metadata files from github repo, you mean that I need to init TUF repo and store all stuff in separate github repo isolated from the app source code, and to serve metadata from this repo and only upload targets to S3? Like I would have like 2 github repo one for the app (I'll name it, for example, app-1) and the other for TUF repo (I'll name it, for example, app-1-repo), and I also want to clarify that github repo for TUF repo must be public in order to serve metadata from it?

So, the idea of the separation of the app code and TUF repository confused me even more so I lean more towards the implementation as in tufup-example.

[its-monotype]

Here's an example of how I see the upgrade process using the approach as in tufup-example (that is, the application source code and the TUF repository are together):

I have a project dir in which I have the source code of the app and all TUF repo stuff.

I have S3 there with the public [1] folder (where will be stored metadata and target and distributed) and the private [2] folder (where will be stored the .tufup-repo-config and keys for the worker that periodically re-signs the snapshot and timestamp metadata) for that specific app.

After I initialized the repository, I deploy .tufup-repo-config and keys to the private [2] folder on S3.

Then, to release a new update, I create a pyinstaller bundle, call script repo_add_bundle.py to create the target, and deploy this new target and modified metadata from temp/repo to the public [1] S3 folder.

[dennisvang]

So, the idea of the separation of the app code and TUF repository confused me even more so I lean more towards the implementation as in tufup-example.

@its-monotype Sorry about that, I was just trying to list your options. You can organize your repositories however you like.

[dennisvang]

Here's an example of how I see the upgrade process using the approach as in tufup-example (that is, the application source code and the TUF repository are together):

I have a project dir in which I have the source code of the app and all TUF repo stuff.

I have S3 there with the public [1] folder (where will be stored metadata and target and distributed) and the private [2] folder (where will be stored the .tufup-repo-config and keys for the worker that periodically re-signs the snapshot and timestamp metadata) for that specific app.

After I initialized the repository, I deploy .tufup-repo-config and keys to the private [2] folder on S3.

Then, to release a new update, I create a pyinstaller bundle, call script repo_add_bundle.py to create the target, and deploy this new target and modified metadata from temp/repo to the public [1] S3 folder.

@its-monotype Yes, that sounds about right.

[dennisvang]

FYI It is also possible to serve the update files from a private location requiring authentication. Your app would then need to need to supply credentials via the session_auth argument to the tufup.client.Client. See example in the tests.

[its-monotype]

note that the repo_dir in the config file should point to the local parent directory of your metadata dir

How can I change repo_dir in the .tufup-repo-config using code?

[its-monotype]

I mean, is there a way to do this somehow easier than parsing the entire file, finding the repo_dir line and replacing it with the current parent of metadata dir?

[its-monotype]

@dennisvang And I also ran into a problem when I did not explicitly specify the -e (expiration days) option for the tufup sign snapshot ./temp/keystore command, then my metadata is not updated. I used tufup-example setup, to ensure that I didn't break smth during the developing of my project.

My .tufup-repo-config:

{
    "app_name": "my_app",
    "app_version_attr": "myapp.__version__",
    "encrypted_keys": [],
    "expiration_days": {
        "root": 365,
        "snapshot": 7,
        "targets": 7,
        "timestamp": 1
    },
    "key_map": {
        "root": [
            "my_key"
        ],
        "snapshot": [
            "my_key"
        ],
        "targets": [
            "my_key"
        ],
        "timestamp": [
            "my_key"
        ]
    },
    "keys_dir": "C:\\Users\\itsmo\\Projects\\tufup-example\\temp\\keystore",
    "repo_dir": "C:\\Users\\itsmo\\Projects\\tufup-example\\temp\\repository",
    "thresholds": {
        "root": 1,
        "snapshot": 1,
        "targets": 1,
        "timestamp": 1
    }
}

My temp/repository/metadata/timestamp.json file after running the command tufup sign snapshot ./temp/keystore:

{
    "signatures": [
        {
            "keyid": "6bbf73b70515728e1625745553c1e00879dc10da23d32e26836491fd85ca8016",
            "sig": "c558f99e3de22d03c61b8aca4a2b95ffbb3153f54315c0f9bf01d934135627692c09f72391b01db6a394cae1128985f6b072375fcc7a41ec7ca676d25a53e400"
        }
    ],
    "signed": {
        "_type": "timestamp",
        "expires": "2022-11-20T16:03:44Z",
        "meta": {
            "snapshot.json": {
                "version": 3
            }
        },
        "spec_version": "1.0.30",
        "version": 3
    }
}

"expires": "2022-11-20T16:03:44Z", not changed, although current timestamp: 2022-11-20T22:26:40.413Z

Command output:

tufup version: 0.4.6
Adding signature...
Done.

But if I explicitly specify expiration days, then metadata is updated, and output of the command is:

tufup version: 0.4.6
Setting expiration date 7 days from now...
Publishing changes...
Done.

It feels like tufup can't pick up expiration dates from .tufup-repo-config. Or maybe I'm doing something wrong or did not understand something...

[dennisvang]

I mean, is there a way to do this somehow easier than parsing the entire file, finding the repo_dir line and replacing it with the current parent of metadata dir?

@its-monotype At the moment, no, there is no alternative.

Manually updating the config file is possible using the CLI command tufup init. This will allow you to review all the current settings and modify them where necessary. Alternatively, you could just open the file in a text editor.

Programmatically updating the config file is not considered a common use case. However, the config file is just JSON, so it is relatively easy to parse/update. For example:

import json
import pathlib

config_file_path = pathlib.Path('/path/to/.tufup-repo-config')
config_dict = json.loads(config_file_path.read_text())
config_dict['repo_dir'] = '/new/repo_dir'
config_file_path.write_text(json.dumps(config_dict))

Or, more robust and nicer output:

json.dumps(config_dict, default=str, sort_keys=True, indent=4)

It is also possible to use the Repository class, but that's less flexible and hides what is actually going on:

from tufup.repo import Repository

repo = Repository.from_config()  # expects .tufup-repo-config in current working dir
repo.repo_dir = '/new/repo_dir'
repo.save_config()

[dennisvang]

[...] when I did not explicitly specify the -e (expiration days) option for the tufup sign snapshot ./temp/keystore command, then my metadata is not updated. [...] It feels like tufup can't pick up expiration dates from .tufup-repo-config. [...]

@its-monotype This is actually a design choice. I understand it can be a bit confusing, so I'll try to explain below.

The tufup sign command can be used in two ways:

1. Normal signing: Both the "signatures" and the "signed" part of the metadata file are modified.
2. Threshold signing: Only the "signatures" part is modified. This is used when multiple people need to sign-off on a file, to meet the specified "threshold" number of signatures.

The expiration date is part of the "signed" metadata, so this cannot change when we do a threshold sign. Therefore, the tufup sign command requires an explicit value for the number of days until expiration. If you don't specify -e, it is assumed to be a threshold signature.

The values specified in the config file are used in other commands where "signed" metadata are changed. For example, when you do update the expiration date for snapshot, the expiration date for timestamp is also updated using the value from the config.

[its-monotype]

@dennisvang Thank you a lot for such an informative explanation. I have another question when I want to deploy new app updates after running repo_add_bundle.py, which metadata files should be deployed to S3? And also my metadata in the local dev environment could be outdated because the "cron-like worker" periodically re-signs metadata on S3 to make sure that snapshot/timestamp metadata is always up to date. So, how do I deploy a new app update to S3 from the local dev environment correctly?

[dennisvang]

[...] when I want to deploy new app updates after running repo_add_bundle.py, which metadata files should be deployed to S3?

@its-monotype Typically, after adding a new bundle, the targets, snapshot, and timestamp metadata will have changed, so these need to be deployed.

And also my metadata in the local dev environment could be outdated because the "cron-like worker" periodically re-signs metadata on S3 to make sure that snapshot/timestamp metadata is always up to date. So, how do I deploy a new app update to S3 from the local dev environment correctly?

As long as the local dev environment uses a valid key to sign the metadata (or multiple valid keys), that should not matter. The newly signed files from the local dev environment can simply overwrite the existing ones on S3. The next time your worker comes along, it will simply re-sign the new files.

[wickeat]

Hi @dennisvang, thank you for the informative pointers above. I am also trying to set up an updater with S3 bucket as the remote. This is my first foray into TUF and there are some things that I'm still figuring out of the framework.

Following the tufup-example, I've run the init step, creating the installer bundle step (let's call it v0.1), and then the add bundle step. After creating a subsequent new bundle version (v0.2), and then adding the bundle via repo_add_bundle.py, the snapshot.json, targets.json and timestamp.json files are modified. However, there isn't any new root.json file created. There are only 1.root.json and root.json from the init step.

My question is, should there be a new 2.root.json file created after registering a new version? This is what I gathered from reading up on how TUF performs the update on the client side.

Thank you in advance for any help and guidance.

[dennisvang]

My question is, should there be a new 2.root.json file created after registering a new version? ...

Hi @wickeat,

When you register a new version of your app, you add a new "target" to targets.json, which means snapshot.json needs to be updated, which, in turn, means timestamp.json needs to be updated. This is because snapshot points to a specific version of targets, and timestamp points to a specific version of snapshot (you can open the corresponding json files to see this).

However, root.json should not be updated.

In fact, root.json only needs to be updated on rare occasions, for example, if the file expires (in which case you'll need to set a new expiry date), or when the set of trusted keys needs to be modified.

According to the TUF docs:

Root Metadata (root.json) ... Specifies the other top-level roles. When specifying these roles, the trusted keys for each are listed, along with the minimum number of those keys required to sign the role's metadata.

If you open up the root.json, you'll see something like this:

{
  "signatures": [...],
  "signed": {
    "_type": "root",
    "consistent_snapshot": false,
    "expires": "2023-10-12T08:02:36Z",
    "keys": {...},
    "roles": {...},
    "spec_version": "1.0.29",
    "version": 1
  }
}

In short:

A new version of the root.json file is only necessary when anything in the "signed" section changes (other than "version" itself).

[mdhuzaifapatel]

Hello, I need your help... I was trying to use tufup for making my application auto-update when new updates are available. I tried the sample example you have provided and it's working fine. But I have an existing application in which I want to integrate tufup... how should I do that? also I'm using a linux server as a repository where updates will be pushed. I tried following the steps give in sample-example but It's not creating a exe in linux..

please help me out i just want to achieve this: Updates will be pushed to server My app should fetch updates from there and update the app.

I also want to know how to create temp/repository folder files? should i create them on windows and then upload on server ?

[dennisvang]

Hi @mdhuzaifapatel, thanks for your interest in tufup. I'll try to give you some pointers, although I cannot give advice on the actual server set-up and configuration.

... But I have an existing application in which I want to integrate tufup... how should I do that? ...

The best way to understand how to integrate tufup into your application is to study the example application, specifically myapp/init.py. Some steps your app will need to take (assuming a tufup update-repository has already been created):

• define which directories tufup should use on the client system(see examples in myapp/settings.py, but it's your choice where to put everything)
• ensure the directories mentioned above exist on the client system
• include a copy of root.json in your app bundle
• make sure your app knows the url for your update server
• instantiate a tufup Client, somewhere in your app (it's your choice where and when you want to do this), then ...
  • call client.check_for_updates()
  • if an update is found, call client.download_and_apply_update()

See myapp/init.py for details.

Updates will be pushed to server ...

There are many ways to upload files to your server. This is outside the scope of tufup. Also see e.g. 3

My app should fetch updates from there and update the app.

See explanation above.

I also want to know how to create temp/repository folder files? should i create them on windows and then upload on server ?

It depends on how you distribute your app:

• Is it a Windows, Mac, or Linux app?
• How do you bundle and distribute your app? For example, do you use PyInstaller, or something else?

Bundling the app is also outside the scope of tufup. Tufup just handles collections of files, regardless of how they were bundled. See tufup readme for details. Also see e.g. this discussion.

In any case, to create the repository files (the content of e.g. temp/repository), you use the tufup repository tools. The repository tools take care of creating and updating tuf metadata files and tufup archive files. The repository tools can be used from the command line or from a script. This is all explained in the readme. A summary:

You start by initializing an update-repository This will automatically create a root.json metadata file. See e.g. repo_init.py. IMPORTANT: As the name suggests, you only need to initialize your update-repository once.

A copy of the root.json file must be included in your app bundle. Once you have a bundled app, you can use the tufup repository tools to add the app to your update-repository. See e.g. repo_add_bundle.py. For every new version of your app, you create a new app bundle and add it to the tufup repo.

Once a new app version has been added, you need to upload the files from your update-repository (i.e. the content of temp/repository or its equivalent) to some place where your server has access to them (e.g. on the actual server system, or on something like AWS S3). Then your server should serve the files, just like in the example.

Finally, we always recommend thoroughly reading the following:

• TUF documentation
• tufup-example readme
• tufup readme
• any tufup related issues here: https://github.com/dennisvang/tufup/issues (especially those labeled "question")

[mdhuzaifapatel]

...

Thanks for the reply, I got some idea, still I want to clear few things.. I use pyinstaller to bundle the app and create an exe and it's for windows.

So according to readme, I followed all the steps used that pyinstaller command to bundle then used that set the target command for version 1.0 ,later made some changes and then repeated the same steps and added a target for version 2.0, i did all of this in windows system. Now the next step in readme is to host the repository using command python -m http. server, but it's for local testing so what I did is I copied the tar files from targets folder and contents of metadata to a location at server... (Also I had already added this location's url in settings.py for metadata and targets ).. and hosted it.

Later , i extracted the tar file 1.0 and ran the exe on windows system but it didn't get updated.

Pls help me out. Thanks

[dennisvang]

... so what I did is I copied the tar files from targets folder and contents of metadata to a location at server... (Also I had already added this location's url in settings.py for metadata and targets ).. and hosted it.

Later , i extracted the tar file 1.0 and ran the exe on windows system but it didn't get updated.

@mdhuzaifapatel Just to be sure, this is using the "myapp" from the tufup-example repo?

Could you post the console output from the app?

(make sure to remove any sensitive info)

[mdhuzaifapatel]

Yes, its using myapp from example

[dennisvang]

Could you post the console output from the app?

(make sure to remove any sensitive info)

[mdhuzaifapatel]

Actually I'm using your "myapp" itself, I have just added some print statements, the thing is its not updating from server..

I ran this "create_pyinstaller_bundle_win.bat" it created a bundle "main.exe", then I ran this "tufup targets add 1.0 temp/dist/main temp/keystore" now tar-1.0 was created (This is first version of app)

Later, I did some changes in app's code and repeated the steps above now tar-2.0 was created. Now I uploaded these tar files with patch on my server at some address.

Then according to documentation I extracted the tar-1.0 in windows system and ran "main.exe", I was expecting the app to download the patch from server for 2.0 and update the app, but it didn't.

The same approach when I tried locally, worked fine that is both client and repo on same system.

[dennisvang]

Yes, I understand, but what is the console output when you run main.exe?

[dennisvang]

Without the console output, I cannot help you.

Could you open a cmd window, then run main.exe from the command line, and then post the output?

[mdhuzaifapatel]

import sys
from myapp import main, settings

print('Hello....')

print(f'(settings.APP_NAME) (settings.APP_VERSION}')

X = input('Enter name: ')

main(sys.argv[1:])

this is my "main.py"

Console Output:

Hello....
my_app 1.0
Enter name:

That's it .... but when I used localhost for hosting the app used to get updated with version 2

[dennisvang]

And what happens if you enter a "name" and hit enter?

[dennisvang]

... timestamp was signed by 0/1 keys

@mdhuzaifapatel This is a new problem and looks similar to #13.

Also see this post and the troubleshooting section in the readme.

[mdhuzaifapatel]

... timestamp was signed by 0/1 keys

@mdhuzaifapatel This is a new problem and looks similar to #13.

Also see this post and the troubleshooting section in the readme.

It worked thanks.

I need your help in integrating tufup with an existing python application. I'm using pyinstaller along with eel for bundling my app, so I was want to steps I should follow to implement auto-updates into my existing application.

In my app, once I run the pyinstaller command a folder will be created which has all the files and an exe file. I zip this folder and distribute it to users . So please help me out how should I proceed in implementing tufup. I'm confused.

For now I'm ready with a folder which has bundled exe file with all necessary files..

If possible please give me a detailed explanation.

I tried your given example and it's working fine, I have added a server's url from where the updates are getting downloaded and the example app (my_app) is updating.

Thanks a lot. @dennisvang

[dennisvang]

It worked thanks.

@mdhuzaifapatel thanks, that is good know.

I need your help in integrating tufup with an existing python application. ... If possible please give me a detailed explanation.

First of all, you'll need to initialize a tuf(up) repository. Note that, as far as this repository is concerned, your application is just a bundle of files (any files), to which you assign a name and a version. To initialize a repository, as explained above, you can use e.g. tufup init on the command line, or you can follow the repo_init script from tufup-example.

You'll need to make some informed decisions here, as to the level of security you need. For example, how many distinct keys are you going to require for each role, and will you share keys between roles? Look into TUF roles for guidance.

Then, as explained above, you'll need to incorporate the tufup Client into your existing python application. The tufup-example application shows the simplest way to do this, so I would suggest studying the myapp source in detail, and trying to substitute your own app for myapp, then improve from there.

In my app, once I run the pyinstaller command a folder will be created which has all the files and an exe file. I zip this folder and distribute it to users . So please help me out how should I proceed in implementing tufup. I'm confused.

Once the tufup client has been incorporated into your python app, you can bundle your app, using PyInstaller and whatever, as you would normally do. Do make sure the root.json file from the repository is included in the bundle (again have a look at tufup-example). Consider the resulting bundle of files to be the equivalent of the output from the create_pyinstaller_bundle...-step in tufup-example.

Note there is no need to zip your folder for tufup, because tufup takes care of that internally. However, how you handle your initial distribution is outside the scope of tufup: you could create a zip as you normally do, or you could distribute the .tar.gz archive created by tufup (this would be more cumbersome for the user...), or you could create an actual installer using e.g. NSIS or Inno Setup.

Once you have your bundle, add it to the repository using e.g. tufup targets add ... on the command line, or follow the repo_add_bundle script from tufup-example. This will automatically create a zipped archive (.tar.gz) in your targets folder.

To make the repository available on your server, proceed exactly like you did with tufup-example.

Note, on the command line, you can use --help for more info about any specific command, as in: tufup targets add --help

If there is any specific part that is not clear, please let me know.

[mdhuzaifapatel]

I got this, and followed the same steps. My app is running fine as usual, but I can't see a console like in sample example where it asked for "yes" or "no" to download the updates. I'm using eel for frontend of this app. In your sample example I was able to see if updates are available or not, but when I tried it with my app, I'm unable to get it, i.e., I want to how to display it on the frontend so that user can select yes or no for updating the app. So if I get to know if tufup is working in backend then I can go for frontend, but I think tufup is not updating the app... please help

[dennisvang]

... In your sample example I was able to see if updates are available or not, but when I tried it with my app, I'm unable to get it, i.e., I want to how to display it on the frontend so that user can select yes or no for updating the app. ...

@mdhuzaifapatel To see what tufup is doing, you should set your app's log level to logging.DEBUG (assuming you have enabled logging). See e.g. tufup-example main.py.

To know if an update is available, you use Client.check_for_updates(), as in the tufup-example app. This either returns None, if no new version is available, or it returns an object with information about your new app version.

You can use this information to inform your user, and allow them to decide whether to proceed with the update. However, how you do this is an implementation detail of your app. It is entirely up to you.

You can also have a look at issue #76, which is related.

[mdhuzaifapatel]

tuf.api.exceptions.DownloadLengthMismatchError: Downloaded 21755631 bytes exceeding the maximum allowed length of 21755616

how can we fix this issue? pls help

[mdhuzaifapatel]

Traceback (most recent call last):
  File "app.py", line 150, in <module>
  File "app.py", line 138, in main
  File "app.py", line 107, in update
  File "tufup\client.py", line 131, in download_and_apply_update
  File "tufup\client.py", line 271, in _apply_updates
RuntimeError: input(): lost sys.stdin

I got this error when I ran my "exe". I want to how can I bypass the user's choice i.e., the selection of "y" or "n". The update should be downloaded and applied automatically. Also a console which used to get displayed when running your sample example is not displaying in my app, and I think the above has occurred because I was waiting for 'y' or 'n'. I tried to modify the code to avoid but still no progress. sorry for taking your time.

[dennisvang]

... I want to how can I bypass the user's choice i.e., the selection of "y" or "n". The update should be downloaded and applied automatically. ...

@mdhuzaifapatel Try setting skip_confirmation=True. See example app here.

[mdhuzaifapatel]

xx19.89.236 - - [22/Sep/2023 06:52:55] code 404, message File not found
xx19.89.236 - - [22/Sep/2023 06:52:55] "GET /metadata/2.root.json HTTP/1.1" 404 -
xx19.89.236 - - [22/Sep/2023 06:52:55] "GET /metadata/timestamp.json HTTP/1.1" 200 -
xx19.89.236 - - [22/Sep/2023 06:52:59] code 404, message File not found
xx19.89.236 - - [22/Sep/2023 06:52:59] "GET /metadata/2.root.json HTTP/1.1" 404 -
xx19.89.236 - - [22/Sep/2023 06:52:59] code 404, message File not found
xx19.89.236 - - [22/Sep/2023 06:52:59] "GET /metadata/2.root.json HTTP/1.1" 404 -
xx.19.89.236 - - [22/Sep/2023 06:52:59] code 404, message File not found

getting this in console..it was working fine but suddenly I'm facing this...I even cleared the "update_cache_dir" still the same issue.

It is downloading metadata files not the contents of targets. Pls help me fix it. Thanks..

[dennisvang]

@mdhuzaifapatel The code 404, message File not found looks like a custom message from your own code, is that correct?

To say anything about that, I would need to see the relevant part of your app's source.

Also, it looks like you posted only a small part of the console output. In order to say anything sensible, I would need to see the complete console output.

Did you set the logging level to DEBUG?

Also, using named loggers in your app would be very helpful, as in logger = logging.getLogger(__name__).

[mdhuzaifapatel]

Its working now, thanks a lot @dennisvang .

I have replaced the progress hook with tkinter message box --> it says "app has been updated" but after this when I run the app a "cmd" is getting displayed. Could you please let me know how to get rid if it?

After the update is processed, the next time my app's screen should be displayed and I don't want cmd window to be displayed...

My Code:

import tkinter as tk
from tkinter import messagebox

def progress_hook(bytes_downloaded: int, bytes_expected: int):
    progress_percent = bytes_downloaded / bytes_expected * 100
    if progress_percent >= 100:
        # Update is complete, show a tkinter message
        root = tk.Tk()
        root.withdraw()  # Hide the main tkinter window
        tk.messagebox.showinfo("Update Complete", "The application has been updated.")
        root.destroy()  # Close the tkinter window

def update():
    client = Client(
        app_name=settings.APP_NAME,
        app_install_dir=settings.INSTALL_DIR,
        current_version=settings.APP_VERSION,
        metadata_dir=settings.METADATA_DIR,
        metadata_base_url=settings.METADATA_BASE_URL,
        target_dir=settings.TARGET_DIR,
        target_base_url=settings.TARGET_BASE_URL,
        refresh_required=False,
    )

    # Perform update
    if client.check_for_updates():
        client.download_and_apply_update(
            skip_confirmation=True,
            progress_hook=progress_hook,
            purge_dst_dir=False,
            exclude_from_purge=None,
            log_file_name='install.log'
        )
        return True

    return False

if __name__ == '__main__':

    for dir_path in [settings.INSTALL_DIR, settings.METADATA_DIR, settings.TARGET_DIR]:
        dir_path.mkdir(exist_ok=True, parents=True)

    if not settings.TRUSTED_ROOT_DST.exists():
        shutil.copy(src=settings.TRUSTED_ROOT_SRC, dst=settings.TRUSTED_ROOT_DST)

    # Perform the update and check if it returned True (update was applied)
    if update():
        sys.exit()

        // rest of the code

Rest of the code has my remaining code of the app, so this part gets executed if there are no updates, i.e., when update() return False. I'm following this approach, I'm open for your suggestions on this too.

[dennisvang]

@mdhuzaifapatel Is this on Windows?

The progress hook displays download progress, not update progress. So, 100% progress means the update has been downloaded, but it has not been installed yet.

On windows, by default, the client runs the install script in a new command window. This is where the actual installation happens. This only happens once for each update. If you don't like this, you can provide a custom install callback.

Note, also, that the tufup client will automatically exit the current process before installing a new update, so you don't need to do that yourself.

[dennisvang]

@mdhuzaifapatel Most of your questions above are unrelated to the current issue (#3).

If you have any more questions, could you please post them as separate "Discussion" topics?

click here to start a new Discussion

[its-monotype]

@dennisvang

@its-monotype Your remote server would need to serve the tuf metadata and targets content, just like the local server does. So that would be the content of the temp/repository folder in the example.

Your client app would then need to know the urls at which the metadata and targets are served. In the example, these urls are specified in myapp.settings.

Depending on the expiration dates you choose for your tuf metadata files, your remote server may need to be configured so it can automatically sign the snapshot and timestamp metadata at regular intervals. You could use e.g. a cron-job for this. Read the TUF docs for more details on metadata signing, expiration, etc.

How this should be implemented on Heroku is an implementation detail, that depends on your own preferences.

Why do we need only to re-sign periodically snapshot (snapshot.json) and timestamp (timestamp.json) TUF metadata and not targets (targets.json) or delegated targets (ex: 1.root.json) metadata? When that other metadata is used and what will be if it's expired?

[dennisvang]

@its-monotype All of the metadata files need to be re-signed periodically, viz. before the date specified in "expires" member in each file.

The difference is in the time to expiration, which is typically short for timestamp, and longer for targets, etc. However, in the end, the time to expiration for each file is your own choice. See for example https://github.com/dennisvang/tufup-example/blob/35214b372c40f091efae718af18f63738e5c0f39/repo_settings.py#L36

As to why, that really this is a matter for TUF. See the TUF FAQ, TUF metadata section, TUF specification, and python-tuf for more details.

You can open the metadata files to see the expiration dates.

When that other metadata is used and what will be if it's expired?

If any of the metadata files expire, clients will refuse to update. See detailed client workflow in TUF spec.