submitter: metrics break if chain not pending

trajan0x commented 6 days ago

Chain metrics never get called if no txes are pending at book so metrics are just dead. See:

https://github.com/synapsecns/sanguine/blob/1d6c328f5eb3264ec7ce4a4a774fdc613d46dc84/ethergo/submitter/queue.go#L69-L76

This results in this non-intuitive behavior, especially on restarts:

greptile-apps[bot] commented 6 days ago

To ensure metrics are updated even when no transactions are pending, modify the processQueue method in queue.go to call the metrics recording functions regardless of the pending transactions status. Specifically, add calls to recordNumPending, recordNonces, and recordBalance before returning from the method. Here is the updated code snippet:

func (t *txSubmitterImpl) processQueue(parentCtx context.Context) (err error) {
    // TODO: this might be too short of a deadline depending on the number of pendingTxes in the queue
    deadlineCtx, cancel := context.WithTimeout(parentCtx, time.Second*60)
    defer cancel()

    ctx, span := t.metrics.Tracer().Start(deadlineCtx, "submitter.ProcessQueue")
    defer func() {
        metrics.EndSpanWithErr(span, err)
    }()

    // Record metrics even if no pending transactions
    t.recordNumPending(ctx, nil)
    t.recordNonces(ctx, nil)
    t.recordBalance(ctx, nil)

    // Existing code...
    var wg sync.WaitGroup
    wg.Add(1)

    go func() {
        defer wg.Done()
        err := t.processConfirmedQueue(ctx)
        if err != nil {
            span.AddEvent("processConfirmedQueue error", trace.WithAttributes(
                attribute.String("error", err.Error())))
        }
    }()

    pendingChainIDs, err := t.db.GetChainIDsByStatus(ctx, t.signer.Address(), db.Stored, db.Pending, db.FailedSubmit, db.Submitted)
    if err != nil {
        return fmt.Errorf("could not get pendingChainIDs: %w", err)
    }

    pendingChainIDs64 := make([]int64, len(pendingChainIDs))
    for i, chainID := range pendingChainIDs {
        pendingChainIDs64[i] = chainID.Int64()
    }
    span.SetAttributes(attribute.Int64Slice("pending_chain_ids", pendingChainIDs64))

    wg.Add(len(pendingChainIDs))

    for _, chainID := range pendingChainIDs {
        go func(chainID *big.Int) {
            defer wg.Done()

            // get all the pendingTxes in the queue
            pendingTxes, err := t.db.GetTXS(ctx, t.signer.Address(), chainID, db.Stored, db.Pending, db.FailedSubmit, db.Submitted)
            if err != nil {
                span.AddEvent("could not get pendingTxes", trace.WithAttributes(
                    attribute.String("error", err.Error()), attribute.Int64("chainID", chainID.Int64()),
                ))
                return
            }

            err = t.chainPendingQueue(ctx, chainID, pendingTxes)
            if err != nil {
                span.AddEvent("chainPendingQueue error", trace.WithAttributes(
                    attribute.String("error", err.Error()), attribute.Int64("chainID", chainID.Int64())))
            }
        }(chainID)
    }
    wg.Wait()

    return nil
}

References

/ethergo/submitter/queue.go /ethergo/submitter /ethergo/submitter/submitter.go /ethergo/submitter/chain_queue.go

#### About Greptile

This response provides a starting point for your research, not a precise solution. Help us improve! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. [Ask Greptile](https://app.greptile.com/chat/github/synapsecns/sanguine/master) · [Edit Issue Bot Settings](https://app.greptile.com/apps/github)

greptile-apps[bot] commented 6 days ago

To ensure chain metrics are updated even when no transactions are pending, modify the processQueue method in queue.go to always call the metric recording functions. Specifically, add calls to recordNumPending, recordNonces, and recordBalance before returning from the method.

func (t *txSubmitterImpl) processQueue(parentCtx context.Context) (err error) {
    // Existing code...

    // Record metrics even if no transactions are pending
    t.recordNumPending(ctx, nil)
    t.recordNonces(ctx, nil)
    t.recordBalance(ctx, nil)

    return nil
}

This ensures that the metrics are updated regardless of the presence of pending transactions.

References

/ethergo/submitter/queue.go /ethergo/submitter /ethergo/submitter/submitter.go /ethergo/submitter/chain_queue.go

#### About Greptile

This response provides a starting point for your research, not a precise solution. Help us improve! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. [Ask Greptile](https://app.greptile.com/chat/github/synapsecns/sanguine/master) · [Edit Issue Bot Settings](https://app.greptile.com/apps/github)

synapsecns / sanguine

submitter: metrics break if chain not pending #2823

References

References