search

theupdateframework / python-tuf

Python reference implementation of The Update Framework (TUF)
https://theupdateframework.com/
Apache License 2.0
1.63k stars 272 forks source link

New repository tools should provide option to write consistent snapshots #151

Closed trishankkarthik closed 10 years ago

trishankkarthik commented 10 years ago

Observed

The new repository tools writes metadata with the assumption that the metadata will not be continuously changing.

Problem

This means that a pair of concurrent writing of metadata to the same repository may produce inconsistent metadata on disk.

Expected

The new repository tools should offer the option to produce consistent snapshots as outlined in the PEP.

Issue #152 will outline how the TUF specification should change to produce and recognize consistent snapshots.

vladimir-v-diaz commented 10 years ago

Writing consistent snapshots:

repository.write(consistent_snapshots=True)
$ ls metadata
0bd0704d1caf7bd62bc2780f95ffebf2afa5f51762ec332f8944cf89e8688833.targets.txt
8579fa3ad8b1df5086b94d5cdd0c7f092fcf540ff3c32c56d91e4026ab1ce108.root.txt
aa0cdcd23071486411935435ffa150a564f24546705028cbdc4ce4965a0b8c5f.timestamp.txt
dda4623fef86d03ad0ef28800a7d4820321816d801177f6903462f49598fa370.release.txt
root.txt
targets
timestamp.txt

$ ls metadata/targets/
148858bda6ea25a47e3ad996377fd2b4f8c23dcd77d6a00a7deaf41a9aa49fbc.django.txt
django
JustinCappos commented 10 years ago

Cool. So we decided to ditch the hashalg.hash.filename format?

Justin

On Tue, Jan 14, 2014 at 10:10 AM, Vladimir Diaz notifications@github.comwrote:

Writing consistent snapshots:

repository.write(consistent_snapshots=True)

$ ls metadata 0bd0704d1caf7bd62bc2780f95ffebf2afa5f51762ec332f8944cf89e8688833.targets.txt 8579fa3ad8b1df5086b94d5cdd0c7f092fcf540ff3c32c56d91e4026ab1ce108.root.txt aa0cdcd23071486411935435ffa150a564f24546705028cbdc4ce4965a0b8c5f.timestamp.txt dda4623fef86d03ad0ef28800a7d4820321816d801177f6903462f49598fa370.release.txt root.txt targets timestamp.txt $ ls metadata/targets/ 148858bda6ea25a47e3ad996377fd2b4f8c23dcd77d6a00a7deaf41a9aa49fbc.django.txt django

— Reply to this email directly or view it on GitHubhttps://github.com/theupdateframework/tuf/issues/151#issuecomment-32272068 .

vladimir-v-diaz commented 10 years ago

Most likely. I thought about it some more and concluded it supports a lot of the "hashalg.hash.filename" behavior I imagined (e.g., change the hash alg and not require a single change to parsing code). It's nice and simple.

JustinCappos commented 10 years ago

Okay, how does this work? What is the rationale?

Justin

On Tue, Jan 14, 2014 at 10:31 AM, Vladimir Diaz notifications@github.comwrote:

Most likely. I thought about it some more and concluded it supports a lot of the "hashalg.hash.filename" behavior I imagined (e.g., change the hash alg and not require a single change to parsing code). It's nice and simple.

— Reply to this email directly or view it on GitHubhttps://github.com/theupdateframework/tuf/issues/151#issuecomment-32274157 .

vladimir-v-diaz commented 10 years ago

Support multiple hash algorithms, where the generated digests of metadata and target files is included in metadata (and filenames if 'consistent_snapshots' is True). Previously, only a single hash algorithm was supported, and it was set by default to 'sha256' in code. Repository maintainers may now choose any, and/or multiple, hash algorithms from those supported by TUF. By default, 'sha256' is used when generating digests.

Support the recent change to the TUF specification, where writing consistent snapshots may include N versions of identical metadata and targets, if N hash algorithms is used by the repository when generating metadata.

Update code affected by the recent changes to the specification, such as targets that may include digests in their filename.

Support consistent snapshots of compressed metadata, including repositories that provide multiple versions of metadata with different digests included in filenames.

The repository tools can now load repositories that include consistent snapshots of metadata and targets, including those with multiple (i.e., multiple digests prepended to filenames) consistent snapshots of files.

The client code (i.e., updater.py and interposition) may now read repositories with 'consistent_snapshots': true in Root metadata, and properly request and update files with digests included.

Summary of previous implementation updates: Update Root schema Support writing metadata in rolename.ext and digest.rolename.ext formats. Modify libtuf.py methods that walk and extract metadata to recognize the digest.rolename.ext format. Write root.txt and timestamp.txt if consistent_snapshots is True. Support automatic version increments of delegated metadata.

vladimir-v-diaz commented 10 years ago

Writing consistent snapshots, where metadata and target files may include the digests generated by multiple hash algorithms:

tuf.conf.HASH_ALGORITHMS = ['sha224', 'sha256']                                 
repository.write(consistent_snapshots=True)
$ ls repository/metadata/
1ae87a4ef0029c29e5c4181082d3f6437c35ec8b4b1de1ab216a13ee.targets.txt
c52b8c34466bbdb848cad94b1512489435716d5679462bf15019c6aa4c237e50.targets.txt
1e195f9fe80fd5946607ee637696e62721f313496e92ad4d673ce95d.timestamp.txt
c5ef9cf43bc373c7d17631d3390b504dfcb6f7b98896994ab7358f66b14bb894.timestamp.txt
3b54a66ed16423d84bd6f1333c1206c78b6dd256f875e7f6ec761d37.release.txt
root.txt
93c836cb23a3dded67eae1704443893b9d0874e46ed6a5446851572812d64979.root.txt
targets
9d103c5ba0c2a9582966e979284261216817e8efc6517cd478989a3f5e08ee4e.release.txt
timestamp.txt
aab02b2f7ba0c022e05058a116481be8a7890c3c95a7a4e113c1f29b.root.txt

$ ls repository/metadata/targets/
b269a31bb1786421734148e09b0621a8776282078e70cecec3d4c6d6.django.txt.gz
django
b6214e7f8909f608af3398077c62cabf2f4a9ceaf64b82a140647dcbc5394bb3.django.txt
e5f14273949efe1ad052bd6de21f9dcaee208aa5ad9df7f9fbb23bcb.django.txt
dbfa35c7c2be17ababc1c8fed10d043cb8cd832347579bd9cd7de3caa19eab8a.django.txt.gz
vladimir-v-diaz commented 10 years ago 
configurations = tuf.interposition.configure()

urllib.urlopen('http://localhost/file1.txt')
urllib.urlopen('http://localhost/file2.txt')

tuf.interposition.deconfigure(configurations)

Observed:

localhost - - [17/Jan/2014 11:24:51] "GET /metadata/timestamp.txt HTTP/1.1" 200 -
localhost - - [17/Jan/2014 11:24:51] "GET /metadata/9d103c5ba0c2a9582966e979284261216817e8efc6517cd478989a3f5e08ee4e.release.txt HTTP/1.1" 200 -
localhost - - [17/Jan/2014 11:24:51] "GET /metadata/1ae87a4ef0029c29e5c4181082d3f6437c35ec8b4b1de1ab216a13ee.targets.txt HTTP/1.1" 200 -
localhost - - [17/Jan/2014 11:24:51] "GET /targets/5df2fa9eea382d07d47dd99d6c467c71ef3c520e38c01894fd873966.file1.txt HTTP/1.1" 200 -
localhost - - [17/Jan/2014 11:24:51] "GET /targets/8bba1f160aaed38958655319f7af17f9c1cd7844a30581596537cb13.file2.txt HTTP/1.1" 200 -
vladimir-v-diaz commented 10 years ago

Logger output when 'consistent_snapshots' is True:

[2014-01-17 16:24:51,930 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Adding updater for Configuration(netloc=localhost:80)

[2014-01-17 16:24:51,931 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Refreshing top-level metadata for Configuration(netloc=localhost:80)

[2014-01-17 16:24:51,931 UTC] [tuf.download] [INFO][_download_file:726@download.py]
Downloading: http://localhost:8001/metadata/timestamp.txt

[2014-01-17 16:24:51,936 UTC] [tuf.download] [WARNING][_check_content_length:589@download.py]
reported_length (1264) < required_length (16384)

[2014-01-17 16:24:51,936 UTC] [tuf.download] [WARNING][_check_downloaded_length:656@download.py]
Downloaded 1264 bytes, but expected 16384 bytes. There is a difference of 15120 bytes!

[2014-01-17 16:24:51,939 UTC] [tuf.download] [INFO][_download_file:726@download.py]
Downloading: http://localhost:8001/metadata/9d103c5ba0c2a9582966e979284261216817e8efc6517cd478989a3f5e08ee4e.release.txt

[2014-01-17 16:24:51,940 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: 3b54a66ed16423d84bd6f1333c1206c78b6dd256f875e7f6ec761d37

[2014-01-17 16:24:51,940 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: 9d103c5ba0c2a9582966e979284261216817e8efc6517cd478989a3f5e08ee4e

[2014-01-17 16:24:51,942 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
'root.txt' up-to-date.

[2014-01-17 16:24:51,942 UTC] [tuf.download] [INFO][_download_file:726@download.py]
Downloading: http://localhost:8001/metadata/1ae87a4ef0029c29e5c4181082d3f6437c35ec8b4b1de1ab216a13ee.targets.txt

[2014-01-17 16:24:51,943 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: 1ae87a4ef0029c29e5c4181082d3f6437c35ec8b4b1de1ab216a13ee

[2014-01-17 16:24:51,943 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: c52b8c34466bbdb848cad94b1512489435716d5679462bf15019c6aa4c237e50

[2014-01-17 16:24:51,947 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Found updater for hostname=localhost

[2014-01-17 16:24:51,947 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Interposing for http://localhost/file1.txt

[2014-01-17 16:24:51,949 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
'targets.txt' up-to-date.

[2014-01-17 16:24:51,949 UTC] [tuf.download] [INFO][_download_file:726@download.py]
Downloading: http://localhost:8001/targets/5df2fa9eea382d07d47dd99d6c467c71ef3c520e38c01894fd873966.file1.txt

[2014-01-17 16:24:51,950 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: 5df2fa9eea382d07d47dd99d6c467c71ef3c520e38c01894fd873966

[2014-01-17 16:24:51,950 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: ecdc5536f73bdae8816f0ea40726ef5e9b810d914493075903bb90623d97b1d8

[2014-01-17 16:24:51,950 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Found updater for hostname=localhost

[2014-01-17 16:24:51,950 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Interposing for http://localhost/file2.txt

[2014-01-17 16:24:51,950 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
'targets.txt' up-to-date.

[2014-01-17 16:24:51,950 UTC] [tuf.download] [INFO][_download_file:726@download.py]
Downloading: http://localhost:8001/targets/8bba1f160aaed38958655319f7af17f9c1cd7844a30581596537cb13.file2.txt

[2014-01-17 16:24:51,951 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: 8bba1f160aaed38958655319f7af17f9c1cd7844a30581596537cb13

[2014-01-17 16:24:51,951 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: 67ee5478eaadb034ba59944eb977797b49ca6aa8d3574587f36ebcbeeb65f70e

[2014-01-17 16:24:51,951 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Updater removed for Configuration(netloc=localhost:80).
trishankkarthik commented 10 years ago

Great! Could you test this on a target in a delegated targets role, just to be sure?

vladimir-v-diaz commented 10 years ago

Good idea. I had not tested delegated roles until you mentioned it.

configurations = tuf.interposition.configure()
urllib.urlopen('http://localhost/django/file3.txt')
tuf.interposition.deconfigure(configurations)

Observed:

localhost - - [17/Jan/2014 11:46:31] "GET /metadata/timestamp.txt HTTP/1.1" 200 -
localhost - - [17/Jan/2014 11:46:31] "GET /metadata/targets/b269a31bb1786421734148e09b0621a8776282078e70cecec3d4c6d6.django.txt.gz HTTP/1.1" 200 -
localhost - - [17/Jan/2014 11:46:31] "GET /targets/django/94f6e58bd04a4513b8301e75f40527cf7610c66d1960b26f6ac2e743e108bdac.file3.txt HTTP/1.1" 200 -
[2014-01-17 16:46:31,725 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Adding updater for Configuration(netloc=localhost:80)

[2014-01-17 16:46:31,727 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Refreshing top-level metadata for Configuration(netloc=localhost:80)

[2014-01-17 16:46:31,727 UTC] [tuf.download] [INFO][_download_file:726@download.py]
Downloading: http://localhost:8001/metadata/timestamp.txt

[2014-01-17 16:46:31,732 UTC] [tuf.download] [WARNING][_check_content_length:589@download.py]
reported_length (1264) < required_length (16384)

[2014-01-17 16:46:31,732 UTC] [tuf.download] [WARNING][_check_downloaded_length:656@download.py]
Downloaded 1264 bytes, but expected 16384 bytes. There is a difference of 15120 bytes!

[2014-01-17 16:46:31,735 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
'release.txt' up-to-date.

[2014-01-17 16:46:31,735 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
'root.txt' up-to-date.

[2014-01-17 16:46:31,735 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
'targets.txt' up-to-date.

[2014-01-17 16:46:31,737 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Found updater for hostname=localhost

[2014-01-17 16:46:31,737 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Interposing for http://localhost/django/file3.txt

[2014-01-17 16:46:31,738 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
'targets.txt' up-to-date.

[2014-01-17 16:46:31,739 UTC] [tuf.download] [INFO][_download_file:726@download.py]
Downloading: http://localhost:8001/metadata/targets/b269a31bb1786421734148e09b0621a8776282078e70cecec3d4c6d6.django.txt.gz

[2014-01-17 16:46:31,740 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: b269a31bb1786421734148e09b0621a8776282078e70cecec3d4c6d6

[2014-01-17 16:46:31,740 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: dbfa35c7c2be17ababc1c8fed10d043cb8cd832347579bd9cd7de3caa19eab8a

[2014-01-17 16:46:31,740 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: e5f14273949efe1ad052bd6de21f9dcaee208aa5ad9df7f9fbb23bcb

[2014-01-17 16:46:31,740 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: b6214e7f8909f608af3398077c62cabf2f4a9ceaf64b82a140647dcbc5394bb3

[2014-01-17 16:46:31,742 UTC] [tuf.download] [INFO][_download_file:726@download.py]
Downloading: http://localhost:8001/targets/django/94f6e58bd04a4513b8301e75f40527cf7610c66d1960b26f6ac2e743e108bdac.file3.txt

[2014-01-17 16:46:31,744 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: 561bfec53d3dfda762a2431ff6de32f343e505d68829612d1842e6b4

[2014-01-17 16:46:31,744 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: 94f6e58bd04a4513b8301e75f40527cf7610c66d1960b26f6ac2e743e108bdac

[2014-01-17 16:46:31,744 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Updater removed for Configuration(netloc=localhost:80).
vladimir-v-diaz commented 10 years ago

Still need to update logger messages (not obvious that compressed and uncompressed hashes are verified), documentation, minor issues, and refactor a few things.

trishankkarthik commented 10 years ago

On 01/17/2014 11:55 AM, Vladimir Diaz wrote:

Good idea. I had not tested delegated roles until you mentioned it.

Very good, thanks! Looks like everything works as it should.

vladimir-v-diaz commented 10 years ago

Verifying compressed and uncompressed metadata:

[2014-01-19 01:38:36,679 UTC] [tuf.download] [INFO][_download_file:740@download.py]
Downloading: http://localhost:8001/metadata/targets/d28735bb21e41668cf526413b2e68f683673e50060b34cd4cb53b582491117c8.django.txt.gz

[2014-01-19 01:38:36,681 UTC] [tuf.download] [WARNING][_check_downloaded_length:649@download.py]
Downloaded 1806 bytes out of the expected 1806 bytes.

[2014-01-19 01:38:36,681 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: 20b109dfde808e8f10d239625a2ea7f28624bf00ea1446cac4c3c5e8

[2014-01-19 01:38:36,682 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: d28735bb21e41668cf526413b2e68f683673e50060b34cd4cb53b582491117c8

[2014-01-19 01:38:36,682 UTC] [tuf.client.updater] [INFO][_get_file:1152@updater.py]
Decompressing http://localhost:8001/metadata/targets/d28735bb21e41668cf526413b2e68f683673e50060b34cd4cb53b582491117c8.django.txt.gz

[2014-01-19 01:38:36,682 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: 54f988d7f25456622a1dbe771fe633b36b1445b9a3f23efe886b6393

[2014-01-19 01:38:36,682 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: 4b544a0bd028153be15d93e21717361ccdc93deda21a469e06c65c5eed670ac6
trishankkarthik commented 10 years ago

Why is the second log entry a warning, when expected == observed?

On Sat, Jan 18, 2014 at 8:46 PM, Vladimir Diaz notifications@github.comwrote:

Verifying compressed and uncompressed metadata:

[2014-01-19 01:38:36,679 UTC] [tuf.download] [INFO][_download_file:740@download.py] Downloading: http://localhost:8001/metadata/targets/d28735bb21e41668cf526413b2e68f683673e50060b34cd4cb53b582491117c8.django.txt.gz [2014-01-19 01:38:36,681 UTC] [tuf.download] [WARNING][_check_downloaded_length:649@download.py] Downloaded 1806 bytes out of the expected 1806 bytes. [2014-01-19 01:38:36,681 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py] The file's sha224 hash is correct: 20b109dfde808e8f10d239625a2ea7f28624bf00ea1446cac4c3c5e8 [2014-01-19 01:38:36,682 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]The file's sha256 hash is correct: d28735bb21e41668cf526413b2e68f683673e50060b34cd4cb53b582491117c8 [2014-01-19 01:38:36,682 UTC] [tuf.client.updater] [INFO][_get_file:1152@updater.py] Decompressing http://localhost:8001/metadata/targets/d28735bb21e41668cf526413b2e68f683673e50060b34cd4cb53b582491117c8.django.txt.gz [2014-01-19 01:38:36,682 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py] The file's sha224 hash is correct: 54f988d7f25456622a1dbe771fe633b36b1445b9a3f23efe886b6393 [2014-01-19 01:38:36,682 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]The file's sha256 hash is correct: 4b544a0bd028153be15d93e21717361ccdc93deda21a469e06c65c5eed670ac6

— Reply to this email directly or view it on GitHubhttps://github.com/theupdateframework/tuf/issues/151#issuecomment-32698834 .

vladimir-v-diaz commented 10 years ago

Update docstrings and comments and complete the initial implementation of issue 151. Adjust logger level for tuf.download._check_downloaded_length().