feat: ingest platform stats from new spark-evaluate schema

[PatrickNercessian]

Links:

• https://github.com/filecoin-station/spark-evaluate/pull/311
• https://github.com/space-meridian/roadmap/issues/132

[PatrickNercessian]

The current error in build-stats step seems to be that there is no day column in the materialized view, which seems like an issue with using an older version of spark-evaluate. However, #207 was supposed to handle this. Any ideas? @bajtos

[bajtos]

The current error in build-stats step seems to be that there is no day column in the materialized view, which seems like an issue with using an older version of spark-evaluate. However, #207 was supposed to handle this. Any ideas? @bajtos

IIRC, we run the "refresh" queries every 12 hours. Maybe you hit the window when the change was deployed, but the materialised view has not been updated yet.

When I query the materialized view now, I see that it includes the day column.

Let me re-run the CI.

[PatrickNercessian]

When I query the materialized view now, I see that it includes the day column.

I woke up this morning realizing that the PR that mattered for this error was actually https://github.com/filecoin-station/spark-evaluate/pull/341, which was just now merged, so the error was expected actually. My bad!

[bajtos]

When I query the materialized view now, I see that it includes the day column.

I woke up this morning realizing that the PR that mattered for this error was actually filecoin-station/spark-evaluate#341, which was just now merged, so the error was expected actually. My bad!

Makes sense 👍🏻

I misunderstood your original comment. Now I understand this issue is the local database used by npm test, which makes sense.

Can you update spark-evaluate version as part of this PR (similarly to what I did in #207)?

[PatrickNercessian]

Can you update spark-evaluate version as part of this PR (similarly to what I did in #207)?

Sure. Do we also want to update the other mentions in package-lock.json? Or just the one from #207 ? There are also https://github.com/filecoin-station/spark-stats/blob/28fbe3a9b10c907f11052651d0f939f286397eec/package-lock.json#L5522 and https://github.com/filecoin-station/spark-stats/blob/28fbe3a9b10c907f11052651d0f939f286397eec/package-lock.json#L5560 which are pointing to an old commit.

[bajtos]

Can you update spark-evaluate version as part of this PR (similarly to what I did in #207)?

Sure. Do we also want to update the other mentions in package-lock.json? Or just the one from #207 ? There are also

https://github.com/filecoin-station/spark-stats/blob/28fbe3a9b10c907f11052651d0f939f286397eec/package-lock.json#L5522

and https://github.com/filecoin-station/spark-stats/blob/28fbe3a9b10c907f11052651d0f939f286397eec/package-lock.json#L5560

which are pointing to an old commit.

Thanks. I don't know which references to update, to be honest. In my experience, the combination of a git dependency and a package-lock file is not supported well by npm 😢 Perhaps you can update both, just to be sure.

[juliangruber]

Once this lands, please follow up with https://github.com/filecoin-station/spark-stats/pull/215/files#diff-4236f3e7e6666ef758a371bb627b6810c7356fc347e81c1c197c5cb7b385bace

[bajtos]

Once this lands, please follow up with https://github.com/filecoin-station/spark-stats/pull/215/files#diff-4236f3e7e6666ef758a371bb627b6810c7356fc347e81c1c197c5cb7b385bace

It would be best to re-enable those tests as part of this pull request. I think the necessary changes in the production code are already part of this PR.

[bajtos]

@PatrickNercessian the CI is red, PTAL.

[bajtos]

I think the dry-run script is failing because it does not have access to GLIF_TOKEN. Nothing to worry about.

Querying scheduled rewards from impact evaluator
JsonRpcProvider failed to detect network and cannot start up; retry in 1s (perhaps the URL is wrong or the node is not started)
file:///home/runner/work/spark-stats/spark-stats/node_modules/ethers/lib.esm/utils/errors.js:124
            error = new Error(message);
                    ^
Error: server response 401 Unauthorized (request={  }, response={  }, error=null, info={ "requestUrl": "https://api.node.glif.io/rpc/v0", "responseBody": "<html>\r\n<head><title>401 Authorization Required</title></head>\r\n<body>\r\n<center><h1>401 Authorization Required</h1></center>\r\n</body>\r\n</html>\r\n", "responseStatus": "401 Unauthorized" }, code=SERVER_ERROR, version=6.13.2)