Weekly document

[yuanlii]

Oct. 14th, 2019: started to expand the number of hosts running traceroutes to include latency hosts. Account for Open Science Grid ?

Github page: https://github.com/orgs/sand-ci/people Ask Shawn to add as member

TODO: Add github repo | *ask Shawn Improve on statistics ML Front - wait till fix

[yuanlii]

Oct.27,2019: Have finished plotting heatmap and distribution plot for owd_mean. Check code => also with rationale of why using log scale

• TODO: networkX mining
• TODO: explore relationship between delay_sd, delay_mean, delay_median
  • still will be using log-transform

owd suggestions: 尽量看某个src-dest -> owd是如何变化的 (几周、几个月)

• bad: negative => 可以找到哪些sites with unsynchromized clocks
• good: positive
• drifted: ? => assume src is the issue | dest -> 看这个src的其他的destination是否也是problem -> dest同理

about trace: good: complete without self loop

[yuanlii]

Nov.3rd: Questions: (Nov.3rd)

• should I separate those above-zero values vs. below-zero values (e.g., create new index in db?) => best way?
• try to plot several src-dest pair => but how to choose which pairs to plot? --> e.g., focus on those with higher sd?
  • 现在做法: choose 30 days as period (1 day as interval) => not sure if reasonable or not

feedback: BML - CREN => run conclusions

labeled: *drifted: negative values (labeled these) => wrong measurement | synchronization or other problem => problems

可以做的方向： 1.10 sites: positive, negative 同时plot

2. find the ones go up vs. go down (packet go opposite ways)
3. BIP => geometric distance vs. one-way delay （3 times must be error, 1.25 times)

-> write the function (找到negative): labeling function & well-explained

[yuanlii]

Nov.11,2019

“simulated annealing” — a common physics algorithm (measure which part of route is drifted, and then find a time delta trying to correct the src_host and dest_host <— on the physicians’ side

pick one source => check the top 10 dest_host that are associated with the source —> able to find out which part the route is drifted *also can check those src_host vs. dest_host with ups and downs

one server can have multiple hosts(source_host, dest_host)

• can also automate the process <= currently is manual work to detect which host is problematic
• also can check ps_meta to take a look at the hosts with drifted (a certain field inside?)

main goal: understand the general cases + understand the exceptional cases

—— general understanding —— each site can have several perfSONAR configured (NTP: —> tell which path is drifted, and how much does it drift)

Ipv6 and Ipv4 should be treated differently

[yuanlii]

Nov.17,2019: Questions: automation => write a function (src_host, dest_host, from_date, to_date, interval): currently can get trends for each specified src_host, dest_host and time range (1 days); what else needed or good to be added? -- e.g., more precise time range (seconds, minutes, hours?)

[Feedback]: 1.hours would be better as scale

2. pick the src_host and the ten most frequent dest hosts (ElasticSearch API can directly achieve that) the goal is that we hope to have dest_hosts that have enough measurements we want more general cases: assume mtp is working just fine

3. think of some correlation measurements (check out packages in Numpy) --> if the source_host is in common, then can find out which dest_hosts & clocks are drifted