Fix total storage metric per user account in console

[prodalex]

Further background in this Slack message: https://filecoinproject.slack.com/archives/C027XP5BTGB/p1703109484136799

Steps to reproduce:

1. Upload a bunch of files
2. Check total which shows a wrong result

Expected result:

• correct amount of data stored is shown as metric

[hannahhoward]

old spaces are hypothesized root cause

possible solution is migrating spaces

[prodalex]

Feedback from a user who create a new account and new space:

Problem: I have created a Space and uploaded the collection inside aprox 2.6gb but when I check my account inside the console I still see 0B of 5.0GB used

Steps taken:

1. I have created the account a couple of hours ago
2. after providing my debit card details I was taken to the Console where I first created a Space
3. then proceeded to upload the files via Directory > UI - browsing for the folder Not drag/drop.
4. Also I browsed all the way to the designated folder for upload and clicked inside itself - example of tree: Final/Collection/Upload <- click inside this folder instead of just selecting "Upload" from the tree

[hannahhoward]

OK.

I have reproduced this in my account using console.

1. I have two spaces. I can see both report that they have zero usage (initial & final values)
2. I am testing uploads using space did is did:key:z6MksH9NqkqEkUynxps29CePvUd1Db1C9iEKVRazXeHd1fuj
3. When I upload my file, I can see there is a store/add receipt containing my space did & car cid. (found digging through 'stream-log-store-prod-0') The out value is ok and has a positive non-zero allocated value
4. However, I cannot find any value added to prod-w3infra-space-diff with a partition key for my space
5. There are no logs generated by the ucan-stream function in the billing infra code that tell me anything, I am going to try adding those.
6. This may reflect a larger concern -- because it isn't neccesarily that the usage is reported in correctly but that we may not be succcessfully recording space usage at all in some cases

[alanshaw]

Yes, sounds like you're on the right track.

This is the UCAN stream consumer that should be inserting into the space diffs table: https://github.com/storacha-network/w3infra/blob/main/billing/functions/ucan-stream.js

This is the function that filters stream messages to things we're interested in. Perhaps there is a bug here: https://github.com/storacha-network/w3infra/blob/bc4cb6b75d0164d69660ee30aec48dd85001d9ba/billing/lib/ucan-stream.js#L13-L55

[hannahhoward]

• reproduced in my account

• add data works

• kinesis stream puts the ucan into the firehose

• I can find invocation in the UCAN S3 bucket

• I cannot find it in any invocations of the billing ucanstream handler (which has logging)

• Not in the space diff table

• need to reproduce in cli

[hannahhoward]

Summary of what I know currently:

• when I upload, it succeeds
• I am able to find both the invocation and the receipt for the upload in the kinesis stream
• However, the receipt, while in the kenisis stream, does not appear to get picked up by the billing handler. There is no record of the billing ucan stream handler being invoked with my space did.
• However, if I simply manually create an event in the format of the kinesis stream, and run the lambda in test with the exact receipt I have in the kinesis stream, it works. It stores a space usage delta, and my account total updates.
• It seems this is an issue of not all kinesis stream events getting picked up by the lambda, but I don't know why.
• trying to dig into why this is happening: https://repost.aws/knowledge-center/kinesis-data-streams-lambda-invocation

[alanshaw]

I found this error in the billing ucan stream handler - not sure if helps:

2024-07-03T05:37:02.062Z    0fed54ff-23a7-47e0-a487-388e90deff04    ERROR   Invoke Error    
{
    "errorType": "SyntaxError",
    "errorMessage": "Unexpected non-whitespace character after JSON at position 69",
    "stack": [
        "SyntaxError: Unexpected non-whitespace character after JSON at position 69",
        "    at JSON.parse (<anonymous>)",
        "    at <anonymous> (/billing/functions/ucan-stream.js:65:23)",
        "    at Array.map (<anonymous>)",
        "    at parseUcanStreamEvent (/billing/functions/ucan-stream.js:64:16)",
        "    at <anonymous> (/billing/functions/ucan-stream.js:35:22)",
        "    at processResult (/node_modules/src/awslambda.ts:326:50)",
        "    at <anonymous> (/node_modules/src/awslambda.ts:370:50)",
        "    at <anonymous> (/node_modules/@sentry/src/tracing/trace.ts:163:15)",
        "    at handleCallbackErrors (/node_modules/@sentry/src/utils/handleCallbackErrors.ts:25:26)",
        "    at <anonymous> (/node_modules/@sentry/src/tracing/trace.ts:162:14)"
    ]
}

[hannahhoward]

no that's just my mistake from when I put in the data incorrectly while testing the handler :)

[alanshaw]

I think we're behind on processing the stream. Latest receipt was issued at 2024-06-15T05:48:49.300Z

Your uploads happened on the 20th, 21st and 25th I believe.

[alanshaw]

Iterator age for the ucan stream handler:

For DynamoDB, Kinesis, and Amazon DocumentDB event sources, the age of the last record in the event. This metric measures the time between when a stream receives the record and when the event source mapping sends the event to the function.