search

m-lab / etl-schema

All schema and views related to the etl pipeline and public bigquery tables.
Apache License 2.0
7 stars 2 forks source link

Automatically exclude OAM and other MLab IPs #44

Open mattmathis opened 5 years ago

mattmathis commented 5 years ago

The list of excluded IPs need to be managed more dynamically than a source file.

Is a union over all time the right answer? Is a union across prod and staging the right answer?

(This is a long term solution to #6)

pboothe commented 5 years ago

Matt, could you say more here?

mattmathis commented 3 years ago

I see three plausible interim solutions: 1) add a primitive macro expansion to create_dataset_views.sh to update IaOAM in 6 places. 2) Keep an OAM list in a table in a temporary location (accessed by the extended views). Temporarily manage the list in the same scope as the the views themselves (e.g. by create_dataset_views.sh) 3) Create an OAM list in a TBD permanent location. where it can be managed by k8s. Also an interim method to update it.

My preference would be 2, which has the least stranded engineering cost, and would be a huge step towards 3. Only the table location needs to change.

mattmathis commented 3 years ago

We can also automatically detect tests run from IP addresses similar tt our clusters, to help seed a manually curated OAM list, It can be updated retroactively.