[storage] Add storage consistency semantics

[mimir-d]

• add checks for the consistency model in JobStorageManager, TestEventFetcher, FrameworkEventFetcher
• remove _Async methods, use the storage consistency model to hint to the storage layer as to how to retrieve data
• add storage_test, job_test unit tests; refactor existing tests in storage to use fixtures

testing:

• unit tests:

  % go test ./... -count=1
  ok      github.com/facebookincubator/contest/pkg/api    1.635s
  ?       github.com/facebookincubator/contest/pkg/cerrors    [no test files]
  ?       github.com/facebookincubator/contest/pkg/config [no test files]
  ?       github.com/facebookincubator/contest/pkg/event  [no test files]
  ?       github.com/facebookincubator/contest/pkg/event/frameworkevent   [no test files]
  ?       github.com/facebookincubator/contest/pkg/event/internal/querytools  [no test files]
  ?       github.com/facebookincubator/contest/pkg/event/internal/reflecttools    [no test files]
  ok      github.com/facebookincubator/contest/pkg/event/testevent    0.931s
  ok      github.com/facebookincubator/contest/pkg/job    0.638s
  ?       github.com/facebookincubator/contest/pkg/jobmanager [no test files]
  ok      github.com/facebookincubator/contest/pkg/lib/comparison 0.778s
  ?       github.com/facebookincubator/contest/pkg/logging    [no test files]
  ok      github.com/facebookincubator/contest/pkg/pluginregistry 2.356s
  ok      github.com/facebookincubator/contest/pkg/runner 3.985s
  ok      github.com/facebookincubator/contest/pkg/storage    1.819s
  ok      github.com/facebookincubator/contest/pkg/storage/limits 1.994s
  ok      github.com/facebookincubator/contest/pkg/target 1.078s
  ok      github.com/facebookincubator/contest/pkg/test   1.607s
  ?       github.com/facebookincubator/contest/pkg/transport  [no test files]
  ?       github.com/facebookincubator/contest/pkg/transport/http [no test files]
  ?       github.com/facebookincubator/contest/pkg/types  [no test files]
  ?       github.com/facebookincubator/contest/pkg/userfunctions/donothing    [no test files]
  ?       github.com/facebookincubator/contest/pkg/userfunctions/ocp  [no test files]
  ok      github.com/facebookincubator/contest/pkg/xcontext   0.492s
  ?       github.com/facebookincubator/contest/pkg/xcontext/buildinfo [no test files]
  ?       github.com/facebookincubator/contest/pkg/xcontext/bundles   [no test files]
  ok      github.com/facebookincubator/contest/pkg/xcontext/bundles/logrusctx 2.521s
  ?       github.com/facebookincubator/contest/pkg/xcontext/bundles/zapctx    [no test files]
  ok      github.com/facebookincubator/contest/pkg/xcontext/fields    0.325s
  ok      github.com/facebookincubator/contest/pkg/xcontext/logger    1.207s
  ok      github.com/facebookincubator/contest/pkg/xcontext/logger/internal   1.441s
  ok      github.com/facebookincubator/contest/pkg/xcontext/logger/logadapter/logrus  2.621s
  ?       github.com/facebookincubator/contest/pkg/xcontext/logger/logadapter/zap [no test files]
  ?       github.com/facebookincubator/contest/pkg/xcontext/metrics   [no test files]
  ok      github.com/facebookincubator/contest/pkg/xcontext/metrics/prometheus    2.540s
  ok      github.com/facebookincubator/contest/pkg/xcontext/metrics/simplemetrics 2.561s
  ?       github.com/facebookincubator/contest/pkg/xcontext/metrics/test  [no test files]
  ok      github.com/facebookincubator/contest/pkg/xcontext/metrics/tsmetrics 2.595s

• test a status command

  
  % go run . status 3 | head -20
  Requesting URL http://localhost:8080/status with requestor ID 'contestcli-http'
  with params:
  jobID: [3]
  requestor: [contestcli-http]

The server responded with status 200 OK { "ServerID": "mbp", "Data": { "Status": { "Name": "status test", "State": "JobStateCompleted", "StateErrMsg": "", "StartTime": "2021-06-23T20:39:56Z", "EndTime": "2021-06-23T20:40:02Z", "RunStatus": { "JobID": 3, "RunID": 3, "StartTime": "0001-01-01T00:00:00Z", "TestStatuses": [ { "JobID": 3, "RunID": 3, "TestName": "Literal test", "TestStepStatuses": [ {

[mimir-d]

The xcontext storage consistency value could be better documented I think. I'm open to comments.

[mimir-d]

closed by playing with the github CLI...

[mimir-d]

why yes.. it should. I seem to have ignored that because it didn't use the context copy to begin with

[codecov[bot]]

Codecov Report

Merging #290 (e0d96c4) into master (ceebc35) will increase coverage by 0.30%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #290      +/-   ##
==========================================
+ Coverage   67.24%   67.54%   +0.30%     
==========================================
  Files         136      136              
  Lines        6607     6613       +6     
==========================================
+ Hits         4443     4467      +24     
+ Misses       1658     1644      -14     
+ Partials      506      502       -4     

Flag	Coverage Δ
e2e	45.46% <89.28%> (+0.51%)	:arrow_up:
integration	57.20% <96.42%> (+0.32%)	:arrow_up:
unittests	50.20% <100.00%> (+1.10%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/event/frameworkevent/framework.go	73.33% <ø> (ø)
pkg/jobmanager/status.go	60.46% <100.00%> (ø)
pkg/runner/job_runner.go	73.12% <100.00%> (+3.55%)	:arrow_up:
pkg/storage/events.go	58.33% <100.00%> (+8.33%)	:arrow_up:
pkg/storage/job.go	100.00% <100.00%> (+28.57%)	:arrow_up:
pkg/storage/storage.go	59.25% <100.00%> (+17.15%)	:arrow_up:
pkg/xcontext/context.go	63.84% <0.00%> (+1.12%)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update ceebc35...e0d96c4. Read the comment docs.