*: split dumnati to commons, graph-builder, and policy-engine

[zonggen]

This changes splits dumnati to two binaries and a library crate. No content changes are added and only directory structure changes.

Related: https://github.com/coreos/fedora-coreos-cincinnati/issues/3 Signed-off-by: Allen Bai abai@redhat.com

[zonggen]

The updated directory structure is:

.
├── commons
│   ├── Cargo.toml
│   └── src
│       ├── graph.rs
│       ├── lib.rs
│       ├── metadata.rs
│       └── metrics.rs
├── dumnati
│   ├── Cargo.toml
│   └── src
│       ├── graph.rs
│       ├── main.rs
│       ├── metadata.rs
│       ├── metrics.rs
│       ├── policy.rs
│       └── scraper.rs
├── graph-builder
│   ├── Cargo.toml
│   └── src
│       ├── main.rs
│       ├── policy.rs
│       └── scraper.rs
└──policy-engine
    ├── Cargo.toml
    └── src
        ├── main.rs
        ├── policy.rs
        └── scraper.rs

The dumnati directory remains untouched and its content has been split into commons, graph-builder and policy-engine. @lucab is this overall structure somehow reasonable?

[lucab]

@zonggen yes the hierarchy looks ok. I'd just call the two binaries fcos-graph-builder and fcos-policy-engine so that it makes it easier to slowly move in the other repository without impacting existing code upfront.

[lucab]

By the way you'll need to bring https://github.com/coreos/fedora-coreos-cincinnati/pull/18 into this.

[zonggen]

• rebased on top of master
• renamed to fcos-graph-builder and fcos-policy-engine, also for the variables mentioned in previous reviews
• moved scraper logic out of fcos-policy-engine so that it turns an empty graph now
• re-adjusted structure a little bit by moving policy under ./commons. So now the structure looks like:

  ├── commons
  │   ├── Cargo.toml
  │   └── src
  │       ├── graph.rs
  │       ├── lib.rs
  │       ├── metadata.rs
  │       ├── policy.rs
  │       └── metrics.rs
  ├── dumnati
  │   ├── Cargo.toml
  │   └── src
  │       ├── graph.rs
  │       ├── main.rs
  │       ├── metadata.rs
  │       ├── metrics.rs
  │       ├── policy.rs
  │       └── scraper.rs
  ├── fcos-graph-builder
  │   ├── Cargo.toml
  │   └── src
  │       ├── main.rs
  │       └── scraper.rs
  └──fcos-policy-engine
     ├── Cargo.toml
      └── src
          └── main.rs

[zonggen]

Verify the behaviors:

fcos-graph-builder:

GET http://127.0.0.1:8080/v1/graph?basearch=x86_64&stream=stable

```json { "nodes": [ { "version": "31.20200108.3.0", "metadata": { "org.fedoraproject.coreos.releases.age_index": "0", "org.fedoraproject.coreos.scheme": "checksum" }, "payload": "113aa27efe1bbcf6324af7423f64ef7deb0acbf21b928faec84bf66a60a5c933" }, { "version": "31.20200113.3.1", "metadata": { "org.fedoraproject.coreos.releases.age_index": "1", "org.fedoraproject.coreos.scheme": "checksum" }, "payload": "f480038412cba26ab010d2cd5a09ddec736204a6e9faa8370edaa943cf33c932" }, { "version": "31.20200118.3.0", "metadata": { "org.fedoraproject.coreos.releases.age_index": "2", "org.fedoraproject.coreos.scheme": "checksum" }, "payload": "093f7da6ffa161ae1648a05be9c55f758258ab97b55c628bea5259f6ac6e370e" }, { "version": "31.20200127.3.0", "metadata": { "org.fedoraproject.coreos.releases.age_index": "3", "org.fedoraproject.coreos.scheme": "checksum" }, "payload": "2ba7bb9678d86fc469f1920a03c270d7057e0b93d06fcd1a437f1f79bdc26d83" }, { "version": "31.20200210.3.0", "metadata": { "org.fedoraproject.coreos.releases.age_index": "4", "org.fedoraproject.coreos.scheme": "checksum" }, "payload": "4ea6beed22d0adc4599452de85820f6e157ac1750e688d062bfedc765b193505" }, { "version": "31.20200223.3.0", "metadata": { "org.fedoraproject.coreos.scheme": "checksum", "org.fedoraproject.coreos.releases.age_index": "5" }, "payload": "3477082298eb5e2213433415c78f0919a991b40e62726b8fde434d244c1ec1b6" }, { "version": "31.20200310.3.0", "metadata": { "org.fedoraproject.coreos.releases.age_index": "6", "org.fedoraproject.coreos.scheme": "checksum" }, "payload": "436592e6eb93e899bebab8dbd17514c85be683390ef8bbce8c6d96069ce4c543" }, { "version": "31.20200323.3.2", "metadata": { "org.fedoraproject.coreos.scheme": "checksum", "org.fedoraproject.coreos.releases.age_index": "7" }, "payload": "9b322b9cead87ebf80881b75560e481b6e5a26966814b01bf0c12b61fa363708" }, { "version": "31.20200407.3.0", "metadata": { "org.fedoraproject.coreos.scheme": "checksum", "org.fedoraproject.coreos.releases.age_index": "8" }, "payload": "89e17cc21b6aa3bea8959d1e6957fda157168d57ba6805d8a36142184edc2901" }, { "version": "31.20200420.3.0", "metadata": { "org.fedoraproject.coreos.scheme": "checksum", "org.fedoraproject.coreos.releases.age_index": "9" }, "payload": "b3fc3a3e8513d7e424d0ced1e2517484cb766d238951f2fdec3da2fed3522efb" }, { "version": "31.20200505.3.0", "metadata": { "org.fedoraproject.coreos.releases.age_index": "10", "org.fedoraproject.coreos.scheme": "checksum" }, "payload": "01f074cc6cd88d8d2b43f821da692f2367c101eb4377802cb35092bde0ef02f7" }, { "version": "31.20200517.3.0", "metadata": { "org.fedoraproject.coreos.releases.age_index": "11", "org.fedoraproject.coreos.scheme": "checksum" }, "payload": "967b7b8d624e6d10ff51c2e81ef198fae966c567ac2e9b479771c693d0987949" }, { "version": "32.20200601.3.0", "metadata": { "org.fedoraproject.coreos.scheme": "checksum", "org.fedoraproject.coreos.updates.rollout": "true", "org.fedoraproject.coreos.updates.start_value": "1", "org.fedoraproject.coreos.releases.age_index": "12" }, "payload": "b51037798e93e5aae5123633fb596c80ddf30302b5110b0581900dbc5b2f0d24" }, { "version": "32.20200615.3.0", "metadata": { "org.fedoraproject.coreos.updates.barrier_reason": "https://github.com/coreos/fedora-coreos-tracker/issues/484", "org.fedoraproject.coreos.updates.duration_minutes": "4320", "org.fedoraproject.coreos.updates.rollout": "true", "org.fedoraproject.coreos.updates.start_value": "0", "org.fedoraproject.coreos.updates.barrier": "true", "org.fedoraproject.coreos.releases.age_index": "13", "org.fedoraproject.coreos.updates.start_epoch": "1593540000", "org.fedoraproject.coreos.scheme": "checksum" }, "payload": "abb4b978999bd689ff36114e99c737be16026bc154a4428d4c84a340758c5b01" } ], "edges": [ [ 0, 13 ], [ 1, 13 ], [ 2, 13 ], [ 3, 13 ], [ 4, 13 ], [ 5, 13 ], [ 6, 13 ], [ 7, 13 ], [ 8, 13 ], [ 9, 13 ], [ 10, 13 ], [ 11, 13 ], [ 12, 13 ], [ 0, 12 ], [ 1, 12 ], [ 2, 12 ], [ 3, 12 ], [ 4, 12 ], [ 5, 12 ], [ 6, 12 ], [ 7, 12 ], [ 8, 12 ], [ 9, 12 ], [ 10, 12 ], [ 11, 12 ] ] } ```

GET http://127.0.0.1:9080/metrics

``` # HELP fcos_cincinnati_gb_scraper_graph_final_edges Number of edges in the cached graph, after processing # TYPE fcos_cincinnati_gb_scraper_graph_final_edges gauge fcos_cincinnati_gb_scraper_graph_final_edges{stream="next"} 9 fcos_cincinnati_gb_scraper_graph_final_edges{stream="stable"} 25 fcos_cincinnati_gb_scraper_graph_final_edges{stream="testing"} 29 # HELP fcos_cincinnati_gb_scraper_graph_final_releases Number of releases in the cached graph, after processing # TYPE fcos_cincinnati_gb_scraper_graph_final_releases gauge fcos_cincinnati_gb_scraper_graph_final_releases{stream="next"} 10 fcos_cincinnati_gb_scraper_graph_final_releases{stream="stable"} 14 fcos_cincinnati_gb_scraper_graph_final_releases{stream="testing"} 30 # HELP fcos_cincinnati_gb_scraper_graph_last_refresh_timestamp UTC timestamp of last graph refresh # TYPE fcos_cincinnati_gb_scraper_graph_last_refresh_timestamp gauge fcos_cincinnati_gb_scraper_graph_last_refresh_timestamp{stream="next"} 1594327749 fcos_cincinnati_gb_scraper_graph_last_refresh_timestamp{stream="stable"} 1594327749 fcos_cincinnati_gb_scraper_graph_last_refresh_timestamp{stream="testing"} 1594327749 # HELP fcos_cincinnati_gb_scraper_upstream_scrapes_total Total number of upstream scrapes # TYPE fcos_cincinnati_gb_scraper_upstream_scrapes_total counter fcos_cincinnati_gb_scraper_upstream_scrapes_total{stream="next"} 8 fcos_cincinnati_gb_scraper_upstream_scrapes_total{stream="stable"} 8 fcos_cincinnati_gb_scraper_upstream_scrapes_total{stream="testing"} 8 # HELP process_start_time_seconds Start time of the process since unix epoch in seconds. # TYPE process_start_time_seconds gauge process_start_time_seconds 1594327539 ```

fcos-policy-engine:

GET http://127.0.0.1:8081/v1/graph?basearch=x86_64&stream=stable

```json { "nodes": [], "edges": [] } ```

GET http://127.0.0.1:9081/metrics

``` # HELP fcos_cincinnati_pe_v1_graph_incoming_requests_total Total number of incoming HTTP client request to /v1/graph # TYPE fcos_cincinnati_pe_v1_graph_incoming_requests_total counter fcos_cincinnati_pe_v1_graph_incoming_requests_total 1 # HELP fcos_cincinnati_pe_v1_graph_rollout_wariness Per-request rollout wariness. # TYPE fcos_cincinnati_pe_v1_graph_rollout_wariness histogram fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0"} 0 fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.1"} 0 fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.2"} 1 fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.30000000000000004"} 1 fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.4"} 1 fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.5"} 1 fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.6000000000000001"} 1 fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.7000000000000001"} 1 fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.8"} 1 fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="0.9"} 1 fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="1"} 1 fcos_cincinnati_pe_v1_graph_rollout_wariness_bucket{le="+Inf"} 1 fcos_cincinnati_pe_v1_graph_rollout_wariness_sum 0.18848315746051456 fcos_cincinnati_pe_v1_graph_rollout_wariness_count 1 # HELP process_start_time_seconds Start time of the process since unix epoch in seconds. # TYPE process_start_time_seconds gauge process_start_time_seconds 1594328133 ```

, which look sane to me.

[lucab]

Thanks! Both output and patch looks fine. Travis is red due to a few minor linting errors. Other than that, this should be ok to land.

[lucab]

@zonggen while rebasing, you may also want to include https://github.com/coreos/fedora-coreos-cincinnati/pull/20 here.

[zonggen]

Re-based on top of master. Also adjust according to Travis CI https://travis-ci.com/github/zonggen/fedora-coreos-cincinnati/jobs/360895796 where failing at cargo clippy -- -D warnings