baking-bad / tzkt

😼 Awesome Tezos blockchain indexer and API
https://tzkt.io
MIT License
183 stars 35 forks source link

Invalid head [*******]. Reverting... #121

Closed matyushkins closed 2 years ago

matyushkins commented 2 years ago

Hi. v1.9.3 One of the two sync modules crashes with this error. Last night, two falls at different heights. Restoring from a snapshot helps but not for long.

fail: Microsoft.EntityFrameworkCore.Database.Command[20102]
      Failed executing DbCommand (30,008ms) [Parameters=[@__contract_Id_0='?' (DbType = Int32), @__storage_Id_1='?' (DbType = Int32)], CommandType='Text', CommandTimeout='30']
      SELECT s."Id", s."ContractId", s."Current", s."JsonValue", s."Level", s."MigrationId", s."OriginationId", s."RawValue", s."TransactionId"
      FROM "Storages" AS s
      WHERE (s."ContractId" = @__contract_Id_0) AND (s."Id" < @__storage_Id_1)
      ORDER BY s."Id" DESC
      LIMIT 1
fail: Microsoft.EntityFrameworkCore.Query[10100]
      An exception occurred while iterating over the results of a query for context type 'Tzkt.Data.TzktContext'.
      System.InvalidOperationException: An exception has been raised that is likely due to a transient failure.
       ---> Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
       ---> System.TimeoutException: Timeout during reading attempt
         at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
      System.InvalidOperationException: An exception has been raised that is likely due to a transient failure.
       ---> Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
       ---> System.TimeoutException: Timeout during reading attempt
         at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
fail: Tzkt.Sync.Services.Observer[0]
      Failed to rebase branch. An exception has been raised that is likely due to a transient failure.
      System.InvalidOperationException: An exception has been raised that is likely due to a transient failure.
       ---> Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
       ---> System.TimeoutException: Timeout during reading attempt
         at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken)
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
         at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
         at Tzkt.Sync.Protocols.Proto1.TransactionsCommit.RevertStorage(TransactionOperation transaction) in /app/Tzkt.Sync/Protocols/Handlers/Proto1/Commits/Operations/TransactionsCommit.cs:line 614
         at Tzkt.Sync.Protocols.Proto1.TransactionsCommit.RevertInternal(Block block, TransactionOperation transaction) in /app/Tzkt.Sync/Protocols/Handlers/Proto1/Commits/Operations/TransactionsCommit.cs:line 488
         at Tzkt.Sync.Protocols.Proto13Handler.Revert() in /app/Tzkt.Sync/Protocols/Handlers/Proto13/Proto13Handler.cs:line 449
         at Tzkt.Sync.ProtocolHandler.RevertLastBlock(String predecessor) in /app/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 141
         at Tzkt.Sync.ProtocolHandler.RevertLastBlock(String predecessor) in /app/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 161
         at Tzkt.Sync.Services.Observer.RebaseLocalBranchAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 157
         at Tzkt.Sync.Services.Observer.ExecuteAsync(CancellationToken cancelToken) in /app/Tzkt.Sync/Services/Observer/Observer.cs:line 74
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply block: Validation exception - invalid block predecessor. Rebase local branch...
fail: Tzkt.Sync.Services.Observer[0]
      Invalid head [2564100:BMJqWTFCoqpCruWhBzqCPHPW5Zppk947cYb42ehrtVA5Av1qmth]. Reverting...
Groxan commented 2 years ago

Hi! The logs

Failed executing DbCommand (30,008ms)
Timeout during reading attempt

mean that your DB is unable to process that SQL query in 30 sec and fails with timeout. You need to increase the command timeout in the appsettings.json (see "command timeout=xxx;") or via env vars or command line args.