Build a Databricks Query Exporter for Component Details

[pkudinov]

Reuse the standard databricks query runner and JSON data exporter to generate a daily snapshot of each component detail, including the number of times it was viewed (using RPC data)

[eduohe]

Done 🟢

1) Notebook: https://4221960800361869.9.gcp.databricks.com/?o=4221960800361869#notebook/390215550281597/command/390215550281600 Scheduled to run every Monday at 4am PST ( It takes around 3hours to finish )

2) Gold table:

SELECT count(*) 
FROM near_uc_prod.rpc_logs.gold_component_rpc_loads 

3) Each component is written in this GCS folder: gs://databricks-near-query-runner/output/near_bos_component_details/component_rpc_loads Sample content of https://storage.googleapis.com/databricks-near-query-runner/output/near_bos_component_details/component_rpc_loads/calebjacob.near/widget/AccountProfile.json:

{
  "data": {
    "component": "calebjacob.near/widget/AccountProfile",
    "total_rpc_loads": 12886,
    "rpc_loads": [
      {
        "week": "2023-10-16T00:00:00.000Z",
        "number_of_rpc_loads": 12
      },
      {
        "week": "2023-10-09T00:00:00.000Z",
        "number_of_rpc_loads": 65
      },
      {
        "week": "2023-10-02T00:00:00.000Z",
        "number_of_rpc_loads": 30
      },
      {
        "week": "2023-09-25T00:00:00.000Z",
        "number_of_rpc_loads": 91
      },
      {
        "week": "2023-09-18T00:00:00.000Z",
        "number_of_rpc_loads": 114
      },
      {
        "week": "2023-09-11T00:00:00.000Z",
        "number_of_rpc_loads": 57
      },
      {
        "week": "2023-09-04T00:00:00.000Z",
        "number_of_rpc_loads": 64
      },
      {
        "week": "2023-08-28T00:00:00.000Z",
        "number_of_rpc_loads": 960
      },
      {
        "week": "2023-08-21T00:00:00.000Z",
        "number_of_rpc_loads": 209
      },
      {
        "week": "2023-08-14T00:00:00.000Z",
        "number_of_rpc_loads": 1529
      },
      {
        "week": "2023-08-07T00:00:00.000Z",
        "number_of_rpc_loads": 4645
      },
      {
        "week": "2023-07-31T00:00:00.000Z",
        "number_of_rpc_loads": 4748
      },
      {
        "week": "2023-07-24T00:00:00.000Z",
        "number_of_rpc_loads": 210
      },
      {
        "week": "2023-07-17T00:00:00.000Z",
        "number_of_rpc_loads": 135
      },
      {
        "week": "2023-07-10T00:00:00.000Z",
        "number_of_rpc_loads": 17
      }
    ]
  },
  "executed_at": 1697593519319
}