read and write trace data from cloud providers

drmorr0 commented 2 weeks ago

[x] I certify that this PR does not contain any code that has been generated with GitHub Copilot or any other AI-based code generation tool, in accordance with this project's policies.

Description

Update sk-tracer to import and export data from cloud providers instead of just local storage.

Once https://github.com/apache/arrow-rs/pull/5912 is resolved and a new version released, we can delete some of the code in here.

Docs have also been updated with the new trace information, and also some hooks info.

Testing done

manual testing
new tests added for objectstore, tracer export

Additional info

Resolves #61

codecov[bot] commented 2 weeks ago

Codecov Report

Attention: Patch coverage is 61.18421% with 59 lines in your changes missing coverage. Please review.

Project coverage is 73.14%. Comparing base (5dd669c) to head (44963de).

:exclamation: Current head 44963de differs from pull request most recent head d4747b6

Please upload reports for the commit d4747b6 to get more accurate results.

Files	Patch %	Lines
tracer/errors.rs	0.00%	14 Missing :warning:
lib/store/external_storage.rs	52.00%	12 Missing :warning:
cli/export.rs	0.00%	11 Missing :warning:
tracer/main.rs	50.00%	11 Missing :warning:
ctrl/objects.rs	88.31%	9 Missing :warning:
ctrl/main.rs	0.00%	1 Missing :warning:
driver/main.rs	0.00%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #120 +/- ## ========================================== - Coverage 73.71% 73.14% -0.58% ========================================== Files 39 39 Lines 2184 2245 +61 ========================================== + Hits 1610 1642 +32 - Misses 574 603 +29 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

drmorr0 commented 1 week ago

Kubernetes Object DAG

%%{init: {'themeVariables': {'mainBkg': '#ddd'}}}%%
graph LR

classDef default color:#000
subgraph global
  direction LR
  global/simkube[<b>Namespace</b><br>simkube]
%% DELETED OBJECTS START
%% DELETED OBJECTS END
end

subgraph sk-tracer
  direction LR
  simkube/sk-tracer-svc[<b>Service</b><br>sk-tracer-svc]
  simkube/sk-tracer-depl[<b>Deployment</b><br>sk-tracer-depl]
  simkube/sk-tracer-sa[<b>ServiceAccount</b><br>sk-tracer-sa]
  sk-tracer/sk-tracer-crb[<b>ClusterRoleBinding</b><br>sk-tracer-crb]
  simkube/sk-tracer-tracer-config[<b>ConfigMap</b><br>sk-tracer-tracer-config]
  simkube/sk-tracer-sa--->simkube/sk-tracer-depl
  sk-tracer/sk-tracer-crb--->simkube/sk-tracer-depl
  simkube/sk-tracer-tracer-config--->simkube/sk-tracer-depl
%% DELETED OBJECTS START
%% DELETED OBJECTS END
end

subgraph sk-ctrl
  direction LR
  simkube/sk-ctrl-depl[<b>Deployment</b><br>sk-ctrl-depl]
  simkube/sk-ctrl-sa[<b>ServiceAccount</b><br>sk-ctrl-sa]
  sk-ctrl/sk-ctrl-crb[<b>ClusterRoleBinding</b><br>sk-ctrl-crb]
  simkube/sk-ctrl-sa--->simkube/sk-ctrl-depl
  sk-ctrl/sk-ctrl-crb--->simkube/sk-ctrl-depl
%% DELETED OBJECTS START
%% DELETED OBJECTS END
end

global--->sk-tracer
global--->sk-ctrl

%% STYLE DEFINITIONS START
  style simkube/sk-ctrl-depl fill:#cb4
%% STYLE DEFINITIONS END

New object Deleted object Updated object Updated object (causes pod recreation)

Detailed Diff

#### simkube/sk-ctrl-depl: ChangedWithPodRecreate

``` root['spec']['template']['spec']['containers'][0]['args'][1]: not present --> "--driver-secrets" ``` ``` root['spec']['template']['spec']['containers'][0]['args'][2]: not present --> "simkube" ```

acrlabs / simkube