search

repository-service-tuf / repository-service-tuf-worker

Repository Service for TUF: Worker
MIT License
8 stars 15 forks source link

Bug: Incosistent Metadata after updating the Online Key #491

Closed kairoaraujo closed 4 months ago

kairoaraujo commented 5 months ago

What happened?

The Online Key is used for signing the Targets, Snapshop, Timestamp, and the Target Delegated Roles (succinct roles bins-<id>.json)

The bug occurs when the Online Key is updated, generating an inconsistency while the TUF client tries to download a Target File (artifact).

After an investigation, the root cause of the bug is that we bump the high-level Target roles but don't update the keys, in this case, the Online Key in the Target Metadata (targets.signed.delegations.keys and targets.signed.delegations.succinct_roles.keyids).

The below test shows the failure and the inconsistency. I modified our Functional Tests for Metadata Update to rotate the online key.

INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: d213ca176460459aa0a2c58f32858d1e
INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: 4c974c106e054d758bae664749b6923b
INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: f2050b62642d4be8aeb75b755ae83405
INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: d3a171d70e7f4d53a1155ab3afc9a902
INFO     tests.functional.metadata.test_update:test_update.py:90 [METADATA UPDATE] Submiting Root Metadata Update
INFO     tests.functional.metadata.test_update:test_update.py:116 [METADATA UPDATE]  Metadata Updated by 32ceccb5382e4010a96dda9abad4b383
INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: 7330cc86307c4f64a87ee7b643363185
INFO     tests.functional.metadata.test_update:test_update.py:146 [METADATA UPDATE] {"data":{"task_id":"32ceccb5382e4010a96dda9abad4b383","state":"SUCCESS","result":{"message":"Metadata Update Processed","status":true,"task":"metadata_update","last_update":"2024-04-17T12:05:33.071964Z","details":{"role":"root"}}},"message":"Task state."}
INFO     tests.functional.metadata.test_update:test_update.py:153 [METADATA UPDATE] Update Metadata to 2.root.json finished
INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: 467cd789f552414ca21734de9702dd99
INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: ac1a7ee237844acfbe861975fae02e9e
INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: bb7aff18b7a14d6a9243664bd3b5e39e
INFO     tests.functional.metadata.test_update:test_update.py:22 Adding artifacts
INFO     tests.functional.metadata.test_update:test_update.py:45 Added task_id: 68a28dcbd4e94dc184d3e3d4440348d8
INFO     tests.functional.metadata.test_update:test_update.py:175 [METADATA UPDATE] Metadata Update available (2.root.json)
INFO     tests.functional.metadata.test_update:test_update.py:49 Stop adding artifacts. Total requests: 9
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 1/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 2/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 3/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 4/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 5/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 6/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 7/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 8/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:188 Task 9/9 finshed!
INFO     tests.functional.metadata.test_update:test_update.py:195 Verifying test/condescending_golick-0.tar.gz
============================================================================= warnings summary ==============================================================================
tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed
tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed
tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed
  /usr/local/lib/python3.10/site-packages/pytest_bdd/compat.py:46: PytestDeprecationWarning: A private pytest class or function was used.
    fd = FixtureDef(

tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed
tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed
tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed
  /usr/local/lib/python3.10/site-packages/pytest_html/basereport.py:356: DeprecationWarning: The 'py' module is deprecated and support will be removed in a future release.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------------------------------------------------- generated json file: /rstuf-runner/rstuf-umbrella/test-report.json -----------------------------------------------------
============================================================================= slowest durations =============================================================================
16.34s call     tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed
------------------------------------------------ Generated html report: file:///rstuf-runner/rstuf-umbrella/test-report.html ------------------------------------------------
========================================================================== short test summary info ==========================================================================
FAILED tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed - tuf.api.exceptions.UnsignedMetadataError: bins-1 was signed by 0/1 keys
====================================================================== 1 failed, 6 warnings in 16.42s =======================================================================

Below, I added a simple fix that shows the consistency.


tests/functional/metadata/test_update.py::test_updating_root_metadata_full_signed <- ../../usr/local/lib/python3.10/site-packages/pytest_bdd/scenario.py 
------------------------------------------------------------------------------- live log call -------------------------------------------------------------------------------
2024-04-17 11:58:37 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:37 [    INFO] Added task_id: 5791656a48694886b80d4f8540d3df8d (test_update.py:45)
2024-04-17 11:58:38 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:38 [    INFO] Added task_id: 9c3c85acf42b4713b742162130799da1 (test_update.py:45)
2024-04-17 11:58:38 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:38 [    INFO] Added task_id: d6331645182e417eb5f51aa60276e8bd (test_update.py:45)
2024-04-17 11:58:39 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:39 [    INFO] Added task_id: 79806abceb95462fa42f4056e334a498 (test_update.py:45)
2024-04-17 11:58:39 [    INFO] [METADATA UPDATE] Submiting Root Metadata Update (test_update.py:90)
2024-04-17 11:58:39 [    INFO] [METADATA UPDATE]  Metadata Updated by ed2a1afb04414feb8e39fc876d1e3ae1 (test_update.py:116)
2024-04-17 11:58:40 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:40 [    INFO] Added task_id: 64c6df38a2b2421f8f9b9afe71901f0b (test_update.py:45)
2024-04-17 11:58:40 [    INFO] [METADATA UPDATE] {"data":{"task_id":"ed2a1afb04414feb8e39fc876d1e3ae1","state":"SUCCESS","result":{"message":"Metadata Update Processed","status":true,"task":"metadata_update","last_update":"2024-04-17T11:58:40.200730Z","details":{"role":"root"}}},"message":"Task state."} (test_update.py:146)
2024-04-17 11:58:40 [    INFO] [METADATA UPDATE] Update Metadata to 2.root.json finished (test_update.py:153)
2024-04-17 11:58:40 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:40 [    INFO] Added task_id: 87d6476a0dbd43b4b3562cceac8241af (test_update.py:45)
2024-04-17 11:58:41 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:41 [    INFO] Added task_id: 5859bac4faa5404cb730fb4e139c9d16 (test_update.py:45)
2024-04-17 11:58:41 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:41 [    INFO] Added task_id: 40a514a128ef49d9b40ad8a37be9736f (test_update.py:45)
2024-04-17 11:58:42 [    INFO] Adding artifacts (test_update.py:22)
2024-04-17 11:58:42 [    INFO] Added task_id: 48a7d7ca1c644657bf848d4685147204 (test_update.py:45)
2024-04-17 11:58:42 [    INFO] [METADATA UPDATE] Metadata Update available (2.root.json) (test_update.py:175)
2024-04-17 11:58:42 [    INFO] Stop adding artifacts. Total requests: 9 (test_update.py:49)
2024-04-17 11:58:46 [    INFO] Task 1/9 finshed! (test_update.py:188)
2024-04-17 11:58:47 [    INFO] Task 2/9 finshed! (test_update.py:188)
2024-04-17 11:58:51 [    INFO] Task 3/9 finshed! (test_update.py:188)
2024-04-17 11:58:51 [    INFO] Task 4/9 finshed! (test_update.py:188)
2024-04-17 11:58:52 [    INFO] Task 5/9 finshed! (test_update.py:188)
2024-04-17 11:58:52 [    INFO] Task 6/9 finshed! (test_update.py:188)
2024-04-17 11:58:52 [    INFO] Task 7/9 finshed! (test_update.py:188)
2024-04-17 11:58:52 [    INFO] Task 8/9 finshed! (test_update.py:188)
2024-04-17 11:58:52 [    INFO] Task 9/9 finshed! (test_update.py:188)
2024-04-17 11:58:52 [    INFO] Verifying test/great_kapitsa-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/serene_blackburn-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/amazing_edison-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/adoring_tu-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/optimistic_chatterjee-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/amazing_visvesvaraya-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/optimistic_gould-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/lucid_dijkstra-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/stoic_gates-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/mystifying_colden-9.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/practical_brattain-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/pedantic_dirac-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/modest_joliot-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/youthful_dirac-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/amazing_khayyam-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/vigorous_wu-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/nostalgic_jepsen-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/dazzling_shirley-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/affectionate_edison-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/dazzling_ellis-9.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/sweet_colden-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/vigilant_rosalind-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/busy_saha-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/elated_leavitt-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/goofy_herschel-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/affectionate_brattain-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/tender_wiles-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/hardcore_austin-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/cranky_jang-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/ecstatic_faraday-9.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/beautiful_euler-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/compassionate_johnson-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/loving_chaplygin-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/gracious_feistel-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/xenodochial_buck-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/bold_haibt-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/pedantic_wright-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/quizzical_haslett-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/kind_heyrovsky-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/keen_ritchie-9.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/sad_johnson-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/keen_feistel-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/ecstatic_cannon-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/xenodochial_haslett-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/zen_wu-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/upbeat_jackson-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/boring_ptolemy-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/amazing_darwin-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/elastic_cartwright-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/inspiring_dubinsky-9.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/strange_snyder-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/nostalgic_wescoff-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/eloquent_jang-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/quizzical_bhabha-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/competent_hamilton-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/vigorous_mccarthy-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/pensive_banach-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/charming_bohr-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/lucid_lalande-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/zealous_chatelet-9.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/awesome_rosalind-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/youthful_dijkstra-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/modest_merkle-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/elated_noyce-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/sad_carson-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/quirky_leavitt-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/naughty_varahamihira-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/awesome_darwin-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/recursing_lovelace-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/stupefied_kalam-9.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/kind_solomon-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/naughty_colden-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/suspicious_darwin-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/sleepy_golick-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/brave_nash-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/elegant_benz-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/condescending_brown-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/optimistic_gould-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/boring_vaughan-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/suspicious_torvalds-9.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/condescending_mendeleev-0.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/elastic_bassi-1.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/boring_napier-2.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/elegant_sutherland-3.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/zealous_shaw-4.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/ecstatic_chatterjee-5.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/wonderful_feynman-6.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/xenodochial_dubinsky-7.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/inspiring_robinson-8.tar.gz (test_update.py:195)
2024-04-17 11:58:52 [    INFO] Verifying test/fervent_ride-9.tar.gz (test_update.py:195)

Feature: Update metadata
    Scenario: Updating Root metadata full signed
        Given RSTUF is running and operational
        Then the RSTUF is receiving multiple requests
        When the RSTUF key holders send a fully signed metadata
        Then the API requester should get status code '202' with 'task_id'
        Then the API requester gets from endpoint 'GET /api/v1/task' status 'SUCCESS'
        Then the '2.root.json' will be available in the TUF Metadata
        Then the user downloads will not have inconsistency during this process
    PASSED

Credit for the bug: @matglas

Note: Another bug was identified during the investigation.

https://github.com/repository-service-tuf/repository-service-tuf-worker/blob/70f548f951850f84a4e0677ad87483494533b65f/repository_service_tuf_worker/repository.py#L1304-L1311

What steps did you take?

  1. Generate a Ceremony rstuf admin ceremony
  2. Add targets
  3. Generate a Metadata Update and rotate the online key rstuf admin metadata update
  4. Add targets
  5. Try to download targets using a TUF client

What behavior did you expect?

Update the Online Key without breaking the TUF Metadata consistency for the TUF clients.

Include a Functional Test to avoid regression

Relevant log output

No response

Code of Conduct