Billing sometimes happens for 2 or 3 hour blocks and wrong discount is applied

[scottyeager]

A user noticed that they weren't getting the discount level they expected for some contracts, starting on December 13, 2023.

Upon inspecting some bill reports from the users contracts, I found some odd behavior. The discount level is fluctuating between Silver and Gold with each subsequent billing. Here's contract 42762:

If we run some math on the timestamps, we can see that the Silver discounts apply to periods where two hours have elapsed between billing events, while the Gold discounts happen when billing is executed every hour as expected. Here I've annotated each timestamp with the difference from the last timestamp:

     {
        "amountBilled": "2796847",
        "discountReceived": "Silver",
        "timestamp": "1705487190" (7200)
      },
      {
        "amountBilled": "932282",
        "discountReceived": "Gold",
        "timestamp": "1705479990" (3600)
      },
      {
        "amountBilled": "2796847",
        "discountReceived": "Silver",
        "timestamp": "1705476390"  (7200)
      },
      {
        "amountBilled": "932282",
        "discountReceived": "Gold",
        "timestamp": "1705469190" (3600)
      },

Looking further into the past, we can also spot some cases where there were three hours between billing events and the discount level thus dropped to Bronze. This is where this behavior appears to have started, and we can see at that time billing was consistently done for 2-3 hours at a time:

      {
        "amountBilled": "5486123",
        "discountReceived": "Silver",
        "timestamp": "1702555470"
      },
      {
        "amountBilled": "5094257",
        "discountReceived": "Silver",
        "timestamp": "1702544670"
      },
      {
        "amountBilled": "5943300",
        "discountReceived": "Bronze",
        "timestamp": "1702533870"
      },
      {
        "amountBilled": "5943300",
        "discountReceived": "Bronze",
        "timestamp": "1702523070"
      },
      {
        "amountBilled": "5943300",
        "discountReceived": "Bronze",
        "timestamp": "1702512270"
      },
      {
        "amountBilled": "5943300",
        "discountReceived": "Bronze",
        "timestamp": "1702501470"
      },
      {
        "amountBilled": "1133943",
        "discountReceived": "Gold",
        "timestamp": "1702490670"
      },

Here are a few other contract ids with the same issue: 48127, 48111, 48095 (not an exclusive list).

What to do

There's two issues I see here:

1. Billing can sometimes get delayed by an hour or two
2. Discount levels are calculated based on the amount being billed without consideration for how much time has elapsed

While item 1 seems to be happening quite a lot here and I think that deserves some attention, I think we can never guarantee that billing will happen every hour. Therefore, we also need to address item 2 separately and ensure that the discount level is calculated correctly regardless of the period of time the user is being billed for.

[sameh-farouk]

While item 1 seems to be happening quite a lot here and I think that deserves some attention, I think we can never guarantee that billing will happen every hour.

Thanks for letting us know, @scottyeager! I can confirm that we’re aware of the issue you mentioned in point no.1, and we’re already working on fixing it. In fact, we’ve already fixed it on devnet, and we’re gradually applying the same process to affected validators on main net. It could be done by today or tomorrow.

As for point 2, @renauter will take a look at it and update the issue accordingly.

[scottyeager]

Thanks @sameh-farouk. Good to know that work is already underway :+1:

[renauter]

Thank you for this useful and detailed feedback !

2. Discount levels are calculated based on the amount being billed without consideration for how much time has elapsed

I guess I found the bug. It is considering the elapsed time but in the wrong way. I ll fix it 😉

[sameh-farouk]

Here is a summary of the progress

Billing can sometimes get delayed by an hour or two

Fixed already on all networks

Discount levels are calculated based on the amount being billed without consideration for how much time has elapsed

@renauter has patched this bug and the fix has been released to devnet today (runtime 149)

Please @scottyeager let us know if you still facing the same issue

[A-Harby]

Verified, Devnet. I deployed a VM and waited for 2 days, and the billing value and duration haven't changed for the past two days. I always tried to change the discount package but still wasn't able to reproduce the bug.

[scottyeager]

Thanks all. Seems issue is indeed resolved by the first fix, though also good to know the second fix is there to ensure correct calculation if billing is ever delayed again.

I'll follow up if we notice anything further issue related to this.