search

theupdateframework / python-tuf

Python reference implementation of The Update Framework (TUF)
https://theupdateframework.com/
Apache License 2.0
1.63k stars 272 forks source link

New repository tools should allow hashed-bin delegations #137

Closed trishankkarthik closed 10 years ago

trishankkarthik commented 10 years ago

@nekt reports that the new repository tools do not allow someone to implement something like the unclaimed role which the PEP recommends.

@JustinCappos : What do you think the interface should look like?

vladimir-v-diaz commented 10 years ago

Example of hashed-bin delegations: https://github.com/theupdateframework/pypi.updateframework.com/blob/master/delegate_unclaimed_targets.py

JustinCappos commented 10 years ago

I thought it is allowed, but there should be a helper function to tell you what the hash of something is so you can select the right bin.

If so, we should add that function.

On Wed, Nov 20, 2013 at 10:37 AM, Trishank Karthik Kuppusamy < notifications@github.com> wrote:

@nekt https://github.com/nekt reports that the new repository tools do not allow someone to implement something like the unclaimed role which the PEP https://github.com/theupdateframework/pep-on-pypi-with-tufrecommends.

@JustinCappos https://github.com/JustinCappos : What do you think the interface should look like?

— Reply to this email directly or view it on GitHubhttps://github.com/theupdateframework/tuf/issues/137 .

trishankkarthik commented 10 years ago

Correct, it is technically allowed, but the entire process should be much sweeter.

JustinCappos commented 10 years ago

I agree. Let's add the hash helper function. (It must already exist for client checking but let's just expose what exists.)

On Wed, Nov 20, 2013 at 12:29 PM, Trishank Karthik Kuppusamy < notifications@github.com> wrote:

Correct, it is technically allowed, but the entire process should be much sweeter.

— Reply to this email directly or view it on GitHubhttps://github.com/theupdateframework/tuf/issues/137#issuecomment-28909949 .

vladimir-v-diaz commented 10 years ago

As requested, hash helper method added: https://github.com/theupdateframework/tuf/commit/2a61a8016264af1c5b64d21b5b37c59b5cc7bd7e

trishankkarthik commented 10 years ago

Hmm. This is useful, but it still does not complete the picture for developers. I vote for reopening this issue.

JustinCappos commented 10 years ago

Can you give a before and after example code so that I can understand what is lacking?

vladimir-v-diaz commented 10 years ago

Initial implementation of issue 137.

vladimir-v-diaz commented 10 years ago

@JustinCappos: Creating hashed bins with the delegate_hashed_bins() method.

targets = repository.get_filepaths_in_directory('repository/targets/packages', recursive_walk=True)
repository.targets('unclaimed').delegate_hashed_bins(list_of_targets=targets, keys_of_hashed_bins=[targets_public], number_of_bins=32)

for delegation in repository.targets('unclaimed').delegations:
  delegation.load_signing_key(targets_private)

repository.write()
$ ls repository/metadata/targets/unclaimed/
00-07.txt  18-1f.txt  30-37.txt  48-4f.txt  60-67.txt  78-7f.txt  90-97.txt  a8-af.txt  c0-c7.txt  d8-df.txt  f0-f7.txt
08-0f.txt  20-27.txt  38-3f.txt  50-57.txt  68-6f.txt  80-87.txt  98-9f.txt  b0-b7.txt  c8-cf.txt  e0-e7.txt  f8-ff.txt
10-17.txt  28-2f.txt  40-47.txt  58-5f.txt  70-77.txt  88-8f.txt  a0-a7.txt  b8-bf.txt  d0-d7.txt  e8-ef.txt

Request a file stored in a hashed bin: Note: The client only has root.txt.

configurations = tuf.interposition.configure()
urllib.urlopen('http://localhost:8080/packages/django/file5.txt')
tuf.interposition.deconfigure(configurations)
[2014-01-22 22:50:04,828 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Adding updater for interposed network location: localhost:8080

[2014-01-22 22:50:04,832 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Refreshing top-level metadata for interposed network location: localhost:8080

[2014-01-22 22:50:04,832 UTC] [tuf.download] [INFO][_download_file:740@download.py]
Downloading: http://localhost:8001/metadata/timestamp.txt

[2014-01-22 22:50:04,840 UTC] [tuf.download] [INFO][_check_downloaded_length:671@download.py]
Downloaded 1264 bytes out of an upper limit of 16384 bytes.

[2014-01-22 22:50:04,840 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py]
Not decompressing http://localhost:8001/metadata/timestamp.txt

[2014-01-22 22:50:04,846 UTC] [tuf.download] [INFO][_download_file:740@download.py]
Downloading: http://localhost:8001/metadata/release.txt

[2014-01-22 22:50:04,848 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py]
Downloaded 9775 bytes out of the expected 9775 bytes.

[2014-01-22 22:50:04,848 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py]
Not decompressing http://localhost:8001/metadata/release.txt

[2014-01-22 22:50:04,848 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: ff7b153423e35ced63e9b1823cf7cf5bd899a41cd28b5e98b329766b

[2014-01-22 22:50:04,848 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: 4975acbc63fc8a80e70d2321c113bf143abc4395821d233f82e7111d3783f8d6

[2014-01-22 22:50:04,857 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
'root.txt' up-to-date.

[2014-01-22 22:50:04,858 UTC] [tuf.download] [INFO][_download_file:740@download.py]
Downloading: http://localhost:8001/metadata/targets.txt

[2014-01-22 22:50:04,859 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py]
Downloaded 2112 bytes out of the expected 2112 bytes.

[2014-01-22 22:50:04,859 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py]
Not decompressing http://localhost:8001/metadata/targets.txt

[2014-01-22 22:50:04,860 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: bc47032e31596371966dcc6bfe0e182fa38f585daebbe188aaa01805

[2014-01-22 22:50:04,860 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: f5625cd4fdc48feb834c0a998adaa3156583cca6713d80b2bd210fd6f926f3ac

[2014-01-22 22:50:04,868 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Found updater for interposed network location: localhost:8080

[2014-01-22 22:50:04,868 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Interposing for http://localhost:8080/packages/django/file5.txt

[2014-01-22 22:50:04,871 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]
'targets.txt' up-to-date.

[2014-01-22 22:50:04,872 UTC] [tuf.download] [INFO][_download_file:740@download.py]
Downloading: http://localhost:8001/metadata/targets/unclaimed.txt.gz

[2014-01-22 22:50:04,874 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py]
Downloaded 2187 bytes out of the expected 2187 bytes.

[2014-01-22 22:50:04,874 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: 924ba00f3834f48945095816bdfac0a63c4320b68a171faf251cbc6c

[2014-01-22 22:50:04,874 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: 41d52d9c367ce921eae11d273f4c7fc26ea4765fc20daebdfb0c4bcef162e446

[2014-01-22 22:50:04,875 UTC] [tuf.client.updater] [INFO][_get_file:1152@updater.py]
Decompressing http://localhost:8001/metadata/targets/unclaimed.txt.gz

[2014-01-22 22:50:04,875 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: fa905dc53afb37df094d86e25b3098285e83f039ea797f8de8e62ad3

[2014-01-22 22:50:04,876 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: 3e18c43c7cd277fb250c0144bad5ddc8879b415af8243d81e9129d0082db2f5e

[2014-01-22 22:50:04,891 UTC] [tuf.download] [INFO][_download_file:740@download.py]
Downloading: http://localhost:8001/metadata/targets/unclaimed/f0-f7.txt

[2014-01-22 22:50:04,893 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py]
Downloaded 1332 bytes out of the expected 1332 bytes.

[2014-01-22 22:50:04,893 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py]
Not decompressing http://localhost:8001/metadata/targets/unclaimed/f0-f7.txt

[2014-01-22 22:50:04,893 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: cf80776de05775844ebae8f5e8a69dfe365c8608eb0e9642d91542db

[2014-01-22 22:50:04,893 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: 53588ca227dbe6d8fbbc34cef7af91e1c4abf3a1ecd5664f144d1e7b7f240051

[2014-01-22 22:50:04,899 UTC] [tuf.download] [INFO][_download_file:740@download.py]
Downloading: http://localhost:8001/targets/packages/django/file5.txt

[2014-01-22 22:50:04,900 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py]
Downloaded 6 bytes out of the expected 6 bytes.

[2014-01-22 22:50:04,901 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py]
Not decompressing http://localhost:8001/targets/packages/django/file5.txt

[2014-01-22 22:50:04,901 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha224 hash is correct: a8e923b5902a13ac83831ac03ba16561616b1306d36e7e54f632bbe2

[2014-01-22 22:50:04,901 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]
The file's sha256 hash is correct: 2dbe9e279b85ee61cab10513c39791a2ce4bad678906832551afd1446c24bbe3

[2014-01-22 22:50:04,901 UTC] [tuf.interposition] [INFO][info:38@utility.py]
Updater removed for interposed network location: localhost:8080.
trishankkarthik commented 10 years ago

Very nice --- this is exciting!

On Wed, Jan 22, 2014 at 5:54 PM, Vladimir Diaz notifications@github.comwrote:

@JustinCappos https://github.com/JustinCappos: Creating hashed bins with the delegate_hashed_bins() method.

targets = repository.get_filepaths_in_directory('repository/targets/packages', recursive_walk=True)repository.targets('unclaimed').delegate_hashed_bins(list_of_targets=targets, keys_of_hashed_bins=[targets_public], number_of_bins=32) for delegation in repository.targets('unclaimed').delegations: delegation.load_signing_key(targets_private) repository.write()

$ ls repository/metadata/targets/unclaimed/ 00-07.txt 18-1f.txt 30-37.txt 48-4f.txt 60-67.txt 78-7f.txt 90-97.txt a8-af.txt c0-c7.txt d8-df.txt f0-f7.txt 08-0f.txt 20-27.txt 38-3f.txt 50-57.txt 68-6f.txt 80-87.txt 98-9f.txt b0-b7.txt c8-cf.txt e0-e7.txt f8-ff.txt 10-17.txt 28-2f.txt 40-47.txt 58-5f.txt 70-77.txt 88-8f.txt a0-a7.txt b8-bf.txt d0-d7.txt e8-ef.txt

Request a file stored in a hashed bin: Note: The client only has root.txt.

configurations = tuf.interposition.configure()urllib.urlopen('http://localhost:8080/packages/django/file5.txt')tuf.interposition.deconfigure(configurations)

[2014-01-22 22:50:04,828 UTC] [tuf.interposition] [INFO][info:38@utility.py] Adding updater for interposed network location: localhost:8080 [2014-01-22 22:50:04,832 UTC] [tuf.interposition] [INFO][info:38@utility.py] Refreshing top-level metadata for interposed network location: localhost:8080 [2014-01-22 22:50:04,832 UTC] [tuf.download] [INFO][_download_file:740@download.py] Downloading: http://localhost:8001/metadata/timestamp.txt [2014-01-22 22:50:04,840 UTC] [tuf.download] [INFO][_check_downloaded_length:671@download.py] Downloaded 1264 bytes out of an upper limit of 16384 bytes. [2014-01-22 22:50:04,840 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py] Not decompressing http://localhost:8001/metadata/timestamp.txt [2014-01-22 22:50:04,846 UTC] [tuf.download] [INFO][_download_file:740@download.py] Downloading: http://localhost:8001/metadata/release.txt [2014-01-22 22:50:04,848 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py] Downloaded 9775 bytes out of the expected 9775 bytes. [2014-01-22 22:50:04,848 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py] Not decompressing http://localhost:8001/metadata/release.txt [2014-01-22 22:50:04,848 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py] The file's sha224 hash is correct: ff7b153423e35ced63e9b1823cf7cf5bd899a41cd28b5e98b329766b [2014-01-22 22:50:04,848 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]The file's sha256 hash is correct: 4975acbc63fc8a80e70d2321c113bf143abc4395821d233f82e7111d3783f8d6 [2014-01-22 22:50:04,857 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]'root.txt' up-to-date. [2014-01-22 22:50:04,858 UTC] [tuf.download] [INFO][_download_file:740@download.py] Downloading: http://localhost:8001/metadata/targets.txt [2014-01-22 22:50:04,859 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py] Downloaded 2112 bytes out of the expected 2112 bytes. [2014-01-22 22:50:04,859 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py] Not decompressing http://localhost:8001/metadata/targets.txt [2014-01-22 22:50:04,860 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py] The file's sha224 hash is correct: bc47032e31596371966dcc6bfe0e182fa38f585daebbe188aaa01805 [2014-01-22 22:50:04,860 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]The file's sha256 hash is correct: f5625cd4fdc48feb834c0a998adaa3156583cca6713d80b2bd210fd6f926f3ac [2014-01-22 22:50:04,868 UTC] [tuf.interposition] [INFO][info:38@utility.py] Found updater for interposed network location: localhost:8080 [2014-01-22 22:50:04,868 UTC] [tuf.interposition] [INFO][info:38@utility.py] Interposing for http://localhost:8080/packages/django/file5.txt [2014-01-22 22:50:04,871 UTC] [tuf.client.updater] [INFO][_update_metadata_if_changed:1465@updater.py]'targets.txt' up-to-date. [2014-01-22 22:50:04,872 UTC] [tuf.download] [INFO][_download_file:740@download.py] Downloading: http://localhost:8001/metadata/targets/unclaimed.txt.gz [2014-01-22 22:50:04,874 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py] Downloaded 2187 bytes out of the expected 2187 bytes. [2014-01-22 22:50:04,874 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py] The file's sha224 hash is correct: 924ba00f3834f48945095816bdfac0a63c4320b68a171faf251cbc6c [2014-01-22 22:50:04,874 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]The file's sha256 hash is correct: 41d52d9c367ce921eae11d273f4c7fc26ea4765fc20daebdfb0c4bcef162e446 [2014-01-22 22:50:04,875 UTC] [tuf.client.updater] [INFO][_get_file:1152@updater.py] Decompressing http://localhost:8001/metadata/targets/unclaimed.txt.gz [2014-01-22 22:50:04,875 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py] The file's sha224 hash is correct: fa905dc53afb37df094d86e25b3098285e83f039ea797f8de8e62ad3 [2014-01-22 22:50:04,876 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]The file's sha256 hash is correct: 3e18c43c7cd277fb250c0144bad5ddc8879b415af8243d81e9129d0082db2f5e [2014-01-22 22:50:04,891 UTC] [tuf.download] [INFO][_download_file:740@download.py] Downloading: http://localhost:8001/metadata/targets/unclaimed/f0-f7.txt [2014-01-22 22:50:04,893 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py] Downloaded 1332 bytes out of the expected 1332 bytes. [2014-01-22 22:50:04,893 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py] Not decompressing http://localhost:8001/metadata/targets/unclaimed/f0-f7.txt [2014-01-22 22:50:04,893 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py] The file's sha224 hash is correct: cf80776de05775844ebae8f5e8a69dfe365c8608eb0e9642d91542db [2014-01-22 22:50:04,893 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]The file's sha256 hash is correct: 53588ca227dbe6d8fbbc34cef7af91e1c4abf3a1ecd5664f144d1e7b7f240051 [2014-01-22 22:50:04,899 UTC] [tuf.download] [INFO][_download_file:740@download.py] Downloading: http://localhost:8001/targets/packages/django/file5.txt [2014-01-22 22:50:04,900 UTC] [tuf.download] [INFO][_check_downloaded_length:649@download.py] Downloaded 6 bytes out of the expected 6 bytes. [2014-01-22 22:50:04,901 UTC] [tuf.client.updater] [INFO][_get_file:1155@updater.py] Not decompressing http://localhost:8001/targets/packages/django/file5.txt [2014-01-22 22:50:04,901 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py] The file's sha224 hash is correct: a8e923b5902a13ac83831ac03ba16561616b1306d36e7e54f632bbe2 [2014-01-22 22:50:04,901 UTC] [tuf.client.updater] [INFO][_check_hashes:675@updater.py]The file's sha256 hash is correct: 2dbe9e279b85ee61cab10513c39791a2ce4bad678906832551afd1446c24bbe3 [2014-01-22 22:50:04,901 UTC] [tuf.interposition] [INFO][info:38@utility.py] Updater removed for interposed network location: localhost:8080.

— Reply to this email directly or view it on GitHubhttps://github.com/theupdateframework/tuf/issues/137#issuecomment-33078477 .

vladimir-v-diaz commented 10 years ago

Final updates to #137.

Add the add_restricted_paths() method. Rename libtuf.py Update README. Update delegate_hashed_bins() docstring. More testing of hashed bins and consistent snapshots.