search

JasperFx / marten

.NET Transactional Document DB and Event Store on PostgreSQL
https://martendb.io
MIT License
2.8k stars 442 forks source link

Marten / Oakton swallows exceptions when rebuilding projections #2682

Closed Richard87 closed 1 year ago

Richard87 commented 1 year ago

Hi!

I am troubleshooting why a Projections is hanging

During debugging and exploring the stack I found this exceptions that is thrown:

Projection for Happydogs.ReadModels.ContactListView should either have the Create Method or Constructor for event of type Marten.Events.IEvent<Happydogs.Domain.Member.Event.MemberChangedName>, or Happydogs.ReadModels.ContactListView should have a Default Constructor.

Easy fix when I found it :)

Skjermbilde 2023-08-16 kl 22 05 35

When running, it hangs here waiting for something, before it eventually times out:

Skjermbilde 2023-08-16 kl 22 06 09
Richard87 commented 1 year ago

Oftopic, also, I have no idea why/how, but the error is pretty misleading:

Projection for Happydogs.ReadModels.ContactListView should 
either have the Create Method or Constructor for event of type 
Marten.Events.IEvent<Happydogs.Domain.Member.Event.MemberChangedName>,
 or Happydogs.ReadModels.ContactListView should have a Default Constructor.

But the actuall problem was some unrelated events from a different stream (type/aggregate) (I was trying to get MultiStreamProjection to work!).

jeremydmiller commented 1 year ago

Do you happen to have a suggestion for what that error says? Maybe a hint I guess for "is this event supposed to apply to this projection?"

I know folks really, really want to use records for aggregates or have the validated constructors, and we accommodate that, but it leads you vulnerable to exactly this error if you don't get the stream order correct or in your case, don't filter/group the multi-stream aggregate exactly right.

Richard87 commented 1 year ago

Thanks, hehe, the record type feels perfect, but classes might still be better :)

the actuall error is a bit oftopic, and I think this error actually is a duplicate of the other error I'm experiencing over at https://github.com/JasperFx/marten/issues/2683 ...

jeremydmiller commented 1 year ago

Using the record for the aggregation is a cool syntax, sets you up to hang out with the cool FP kids, and maybe makes your application slow from all the extra object allocations. We'll certainly always support it though

Richard87 commented 1 year ago

hehe true, but I love the with syntax 🤩

Richard87 commented 1 year ago

I'm closing the other issue, and adding the examples of unpredictable log output here:

Example 1

``` fail: Marten.IDocumentStore[0] Marten encountered an exception executing System.InvalidOperationException: This NpgsqlTransaction has completed; it is no longer usable. at Npgsql.NpgsqlTransaction.CheckReady() at Npgsql.NpgsqlTransaction.Rollback(Boolean async, CancellationToken cancellationToken) at Marten.Internal.Sessions.MartenControlledConnectionTransaction.RollbackAsync(CancellationToken token) at Marten.Internal.Sessions.DocumentSessionBase.ExecuteBatchAsync(IUpdateBatch batch, CancellationToken token) fail: Marten.IDocumentStore[0] Marten encountered an exception executing System.InvalidOperationException: This NpgsqlTransaction has completed; it is no longer usable. at Npgsql.NpgsqlTransaction.CheckReady() at Npgsql.NpgsqlTransaction.Rollback(Boolean async, CancellationToken cancellationToken) at Marten.Internal.Sessions.MartenControlledConnectionTransaction.RollbackAsync(CancellationToken token) at Marten.Internal.Sessions.DocumentSessionBase.ExecuteBatchAsync(IUpdateBatch batch, CancellationToken token) MembersView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% ⣷ ContactListView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% ⣷ UniqueMemberEmailView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% ⣷ fail: Marten.IDocumentStore[0] Marten encountered an exception executing delete from public.mt_doc_deadletterevent as d where d.data ->> 'ProjectionName' = :p0;truncate table public.mt_doc_breederview CASCADE;delete from public.mt_event_progression where name = :p1; p0: BreederView p1: BreederView:All System.OperationCanceledException: The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at System.Threading.CancellationToken.ThrowIfCancellationRequested() at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) at Marten.Internal.Sessions.QuerySession.ExecuteReaderAsync(NpgsqlCommand command, CancellationToken token) fail: Marten.IDocumentStore[0] Marten encountered an exception executing System.ObjectDisposedException: NpgsqlTransaction at Npgsql.NpgsqlTransaction.CheckDisposed() at Npgsql.NpgsqlTransaction.CheckReady() at Npgsql.NpgsqlTransaction.Rollback(Boolean async, CancellationToken cancellationToken) at Marten.Internal.Sessions.MartenControlledConnectionTransaction.RollbackAsync(CancellationToken token) at Marten.Internal.Sessions.DocumentSessionBase.ExecuteBatchAsync(IUpdateBatch batch, CancellationToken token) ```

Example 2

``` ── IDocumentStore ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── fail: Marten.IDocumentStore[0] Marten encountered an exception executing System.InvalidOperationException: This NpgsqlTransaction has completed; it is no longer usable. at Npgsql.NpgsqlTransaction.CheckReady() at Npgsql.NpgsqlTransaction.Rollback(Boolean async, CancellationToken cancellationToken) at Marten.Internal.Sessions.MartenControlledConnectionTransaction.RollbackAsync(CancellationToken token) at Marten.Internal.Sessions.DocumentSessionBase.ExecuteBatchAsync(IUpdateBatch batch, CancellationToken token) fail: Marten.IDocumentStore[0] Marten encountered an exception executing delete from public.mt_doc_deadletterevent as d where d.data ->> 'ProjectionName' = :p0;truncate table public.mt_doc_dogmemberview CASCADE;delete from public.mt_event_progression where name = :p1; p0: DogMemberView p1: DogMemberView:All Npgsql.PostgresException (0x80004005): 40001: could not serialize access due to read/write dependencies among transactions DETAIL: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information. at Npgsql.Internal.NpgsqlConnector.g__ReadMessageLong|234_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage) at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken) at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken) at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) at Marten.Internal.Sessions.QuerySession.ExecuteReaderAsync(NpgsqlCommand command, CancellationToken token) Exception data: Severity: ERROR SqlState: 40001 MessageText: could not serialize access due to read/write dependencies among transactions Detail: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information. Hint: The transaction might succeed if retried. File: predicate.c Line: 4815 Routine: OnConflict_CheckForSerializationFailure fail: Marten.IDocumentStore[0] Marten encountered an exception executing delete from public.mt_doc_deadletterevent as d where d.data ->> 'ProjectionName' = :p0;truncate table public.mt_doc_contactlistview CASCADE;delete from public.mt_event_progression where name = :p1; p0: ContactListView p1: ContactListView:All Npgsql.PostgresException (0x80004005): 40001: could not serialize access due to read/write dependencies among transactions DETAIL: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information. at Npgsql.Internal.NpgsqlConnector.g__ReadMessageLong|234_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage) at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken) at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken) at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) at Marten.Internal.Sessions.QuerySession.ExecuteReaderAsync(NpgsqlCommand command, CancellationToken token) Exception data: Severity: ERROR SqlState: 40001 MessageText: could not serialize access due to read/write dependencies among transactions Detail: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information. Hint: The transaction might succeed if retried. File: predicate.c Line: 4815 Routine: OnConflict_CheckForSerializationFailure fail: Marten.IDocumentStore[0] Marten encountered an exception executing System.ObjectDisposedException: NpgsqlTransaction at Npgsql.NpgsqlTransaction.CheckDisposed() at Npgsql.NpgsqlTransaction.CheckReady() at Npgsql.NpgsqlTransaction.Rollback(Boolean async, CancellationToken cancellationToken) at Marten.Internal.Sessions.MartenControlledConnectionTransaction.RollbackAsync(CancellationToken token) at Marten.Internal.Sessions.DocumentSessionBase.ExecuteBatchAsync(IUpdateBatch batch, CancellationToken token) fail: Marten.IDocumentStore[0] Marten encountered an exception executing System.ObjectDisposedException: NpgsqlTransaction at Npgsql.NpgsqlTransaction.CheckDisposed() at Npgsql.NpgsqlTransaction.CheckReady() at Npgsql.NpgsqlTransaction.Rollback(Boolean async, CancellationToken cancellationToken) at Marten.Internal.Sessions.MartenControlledConnectionTransaction.RollbackAsync(CancellationToken token) at Marten.Internal.Sessions.DocumentSessionBase.ExecuteBatchAsync(IUpdateBatch batch, CancellationToken token) BreederView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% ⣷ DogView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% ⣷ fail: Marten.IDocumentStore[0] Marten encountered an exception executing delete from public.mt_doc_deadletterevent as d where d.data ->> 'ProjectionName' = :p0;truncate table public.mt_doc_memberbreederview CASCADE;delete from public.mt_event_progression where name = :p1; p0: MemberBreederView p1: MemberBreederView:All System.OperationCanceledException: The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at System.Threading.CancellationToken.ThrowIfCancellationRequested() at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) at Marten.Internal.Sessions.QuerySession.ExecuteReaderAsync(NpgsqlCommand command, CancellationToken token) fail: Marten.IDocumentStore[0] Marten encountered an exception executing System.ObjectDisposedException: NpgsqlTransaction at Npgsql.NpgsqlTransaction.CheckDisposed() at Npgsql.NpgsqlTransaction.CheckReady() at Npgsql.NpgsqlTransaction.Rollback(Boolean async, CancellationToken cancellationToken) at Marten.Internal.Sessions.MartenControlledConnectionTransaction.RollbackAsync(CancellationToken token) at Marten.Internal.Sessions.DocumentSessionBase.ExecuteBatchAsync(IUpdateBatch batch, CancellationToken token) ```

As a Sidenote Symfony's Console tool is pretty awesome (PHP Framework), the have a ConsoleOutput abstraction that gives you a section method which splits the current console into different parts, where you can have one part for the progressbars, and one part for the streaming log output (I have used it with great success a few times!) https://symfony.com/doc/current/console.html#output-sections

jeremydmiller commented 1 year ago

So what are you wanting to happen here? We've gone through this with other users about having the rebuild blow up on errors during the projection. You're hitting something unusual here by the way it's blowing up on the first step of clearing out progress. That's new.

Richard87 commented 1 year ago

First, I want a way to stop the progressbars from clearing out the screen (missing error log statements obviously makes it really hard to figure out what the root cause is)

In a perfect world I would love Oakton to split the Console into to outputs, one with a streaming log, and one with the progressbars :)

In the end the PostgreSQL error is probably my fault somewhere and I can fix it easier :D

And in a different universe, using stderr would also be amazing :D dotnet run -- projections --rebuild 2> errror.log But I need to learn more about Dotnet Logging for that :D

Richard87 commented 1 year ago

Just read the helptext again and fount --log log.txt, but it didn't create any files:

dotnet run -- projections --rebuild --log log.txt
Building...
Searching 'JasperFx.CodeGeneration.Commands, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' for commands
Searching 'Weasel.CommandLine, Version=6.0.1.0, Culture=neutral, PublicKeyToken=null' for commands
Searching 'Marten.CommandLine, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' for commands
Searching 'Wolverine, Version=1.6.0.0, Culture=neutral, PublicKeyToken=null' for commands
Searching 'Wolverine.Http.FluentValidation, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' for commands

info: Wolverine.Runtime.WolverineRuntime[0]
      Exporting Open Telemetry metrics from Wolverine with name Wolverine:Web, version 1.6.0.0

Overwriting the minimum log level to Error

── IDocumentStore ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

          MembersView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ⣷
              DogView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%  
        DogMemberView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%  
   UniqueBreedersView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ⣷
      ContactListView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ⣷
UniqueMemberEmailView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ⣷
                                                                         fail: Marten.IDocumentStore[0]
      Marten encountered an exception executing 
      delete from public.mt_doc_deadletterevent as d where d.data ->> 'ProjectionName' = :p0;truncate table public.mt_doc_breederview CASCADE;delete from public.mt_event_progression where name = :p1;
        p0: BreederView
        p1: BreederView:All
      System.OperationCanceledException: The operation was canceled.
         at System.Threading.CancellationToken.ThrowOperationCanceledException()
         at System.Threading.CancellationToken.ThrowIfCancellationRequested()
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Marten.Internal.Sessions.QuerySession.ExecuteReaderAsync(NpgsqlCommand command, CancellationToken token)
fail: Marten.IDocumentStore[0]
      Marten encountered an exception executing 

      System.ObjectDisposedException: NpgsqlTransaction
         at Npgsql.NpgsqlTransaction.CheckDisposed()
         at Npgsql.NpgsqlTransaction.CheckReady()
         at Npgsql.NpgsqlTransaction.Rollback(Boolean async, CancellationToken cancellationToken)
         at Marten.Internal.Sessions.MartenControlledConnectionTransaction.RollbackAsync(CancellationToken token)
         at Marten.Internal.Sessions.DocumentSessionBase.ExecuteBatchAsync(IUpdateBatch batch, CancellationToken token)

^C%                                                                                                                                                                    (base) ➜  Web git:(backend/include-member-details-in-contact-list) ✗
jeremydmiller commented 1 year ago

Just for fun, can you run one projection at a time? Or use the -i flag to pick and choose? Maybe you could zero in on a single, offending projection?

Richard87 commented 1 year ago

Yup, usually one at a time works flawlessly, but when running 2 or more at the same time I'm getting errors.

But my last run was interresting, 2 Projections managed to complete, but I got a new error: Npgsql.PostgresException (0x80004005): 40001: could not serialize access due to read/write dependencies among transactions

It seems to be a error when 2 or more connections are trying to write to the same table at the same time (according this answer: https://stackoverflow.com/questions/21706858/org-postgresql-util-psqlexception-error-could-not-serialize-access-due-to-read)

From this snippet:

Marten.IDocumentStore[0]
      Marten encountered an exception executing 
      delete from public.mt_doc_deadletterevent as d where d.data ->> 'ProjectionName' = :p0;truncate table public.mt_doc_dogmemberview CASCADE;delete from public.mt_event_progression where name = :p1;
        p0: DogMemberView
        p1: DogMemberView:All
      Npgsql.PostgresException (0x80004005): 40001: could not serialize access due to read/write dependencies among transactions

My output:

(base) ➜  Web git:(backend/include-member-details-in-contact-list) ✗ dotnet run -- projections --rebuild --log log.txt --log-level Trace
Building...
Searching 'JasperFx.CodeGeneration.Commands, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' for commands
Searching 'Weasel.CommandLine, Version=6.0.1.0, Culture=neutral, PublicKeyToken=null' for commands
Searching 'Marten.CommandLine, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' for commands
Searching 'Wolverine, Version=1.6.0.0, Culture=neutral, PublicKeyToken=null' for commands
Searching 'Wolverine.Http.FluentValidation, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' for commands

info: Wolverine.Runtime.WolverineRuntime[0]
      Exporting Open Telemetry metrics from Wolverine with name Wolverine:Web, version 1.6.0.0

Overwriting the minimum log level to Trace

── IDocumentStore ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

fail: Marten.IDocumentStore[0]
      Marten encountered an exception executing 
      delete from public.mt_doc_deadletterevent as d where d.data ->> 'ProjectionName' = :p0;truncate table public.mt_doc_dogmemberview CASCADE;delete from public.mt_event_progression where name = :p1;
        p0: DogMemberView
        p1: DogMemberView:All
      Npgsql.PostgresException (0x80004005): 40001: could not serialize access due to read/write dependencies among transactions

      DETAIL: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information.
         at Npgsql.Internal.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|234_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Marten.Internal.Sessions.QuerySession.ExecuteReaderAsync(NpgsqlCommand command, CancellationToken token)
        Exception data:
          Severity: ERROR
          SqlState: 40001
          MessageText: could not serialize access due to read/write dependencies among transactions
          Detail: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information.
          Hint: The transaction might succeed if retried.
          File: predicate.c
          Line: 4815
          Routine: OnConflict_CheckForSerializationFailure
fail: Marten.IDocumentStore[0]
      Marten encountered an exception executing 

      System.ObjectDisposedException: NpgsqlTransaction
         at Npgsql.NpgsqlTransaction.CheckDisposed()
         at Npgsql.NpgsqlTransaction.CheckReady()
         at Npgsql.NpgsqlTransaction.Rollback(Boolean async, CancellationToken cancellationToken)
         at Marten.Internal.Sessions.MartenControlledConnectionTransaction.RollbackAsync(CancellationToken token)
         at Marten.Internal.Sessions.DocumentSessionBase.ExecuteBatchAsync(IUpdateBatch batch, CancellationToken token)

          DogView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%  
      BreederView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ⣷
MemberBreederView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ⣷
      MembersView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ⣷
                                                                     fail: Marten.IDocumentStore[0]
      Marten encountered an exception executing 
      delete from public.mt_doc_deadletterevent as d where d.data ->> 'ProjectionName' = :p0;truncate table public.mt_doc_contactlistview CASCADE;delete from public.mt_event_progression where name = :p1;
        p0: ContactListView
        p1: ContactListView:All
      System.OperationCanceledException: The operation was canceled.
         at System.Threading.CancellationToken.ThrowOperationCanceledException()
         at System.Threading.CancellationToken.ThrowIfCancellationRequested()
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Marten.Internal.Sessions.QuerySession.ExecuteReaderAsync(NpgsqlCommand command, CancellationToken token)
fail: Marten.IDocumentStore[0]
      Marten encountered an exception executing 

      System.ObjectDisposedException: NpgsqlTransaction
Marten.Exceptions.ConcurrentUpdateException: Write collision detected while commiting the transaction.
     Npgsql.PostgresException: 40001: could not serialize access due to read/write dependencies among transactions                                                     

     DETAIL: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information.                    
       at async ValueTask<IBackendMessage> Npgsql.Internal.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|234_0(NpgsqlConnector connector, bool async,                
          DataRowLoadingMode dataRowLoadingMode, bool readingNotifications, bool isReadingPrependedMessage)                                                            
       at async Task<bool> Npgsql.NpgsqlDataReader.NextResult(bool async, bool isConsuming, CancellationToken cancellationToken)                                       
       at async Task<bool> Npgsql.NpgsqlDataReader.NextResult(bool async, bool isConsuming, CancellationToken cancellationToken)                                       
       at async ValueTask<NpgsqlDataReader> Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, bool async, CancellationToken cancellationToken)              
       at async ValueTask<NpgsqlDataReader> Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, bool async, CancellationToken cancellationToken)              
       at async Task<DbDataReader> Marten.Internal.Sessions.QuerySession.ExecuteReaderAsync(NpgsqlCommand command, CancellationToken token)                            
  at Exception Marten.Exceptions.MartenExceptionTransformer.<>c.<.cctor>b__1_1(PostgresException e)                                                                    
  at bool JasperFx.Core.Exceptions.ExceptionTransform`1.TryTransform(Exception original, out Exception transformed)                                                    
  at void JasperFx.Core.Exceptions.ExceptionTransformExtensions.TransformAndThrow(IEnumerable<IExceptionTransform> transforms, Exception ex)                           
  at void JasperFx.Core.Exceptions.ExceptionTransforms.TransformAndThrow(Exception ex)                                                                                 
  at void Marten.Exceptions.MartenExceptionTransformer.WrapAndThrow(NpgsqlCommand command, Exception exception)                                                        
  at void Marten.Internal.Sessions.QuerySession.handleCommandException(NpgsqlCommand cmd, Exception e)                                                                 
  at async Task<DbDataReader> Marten.Internal.Sessions.QuerySession.ExecuteReaderAsync(NpgsqlCommand command, CancellationToken token)                                 
  at async Task Marten.Internal.UpdateBatch.ApplyChangesAsync(IMartenSession session, CancellationToken token)                                                         
  at void JasperFx.Core.Exceptions.ExceptionTransformExtensions.TransformAndThrow(IEnumerable<IExceptionTransform> transforms, Exception ex)                           
  at async Task Marten.Internal.UpdateBatch.ApplyChangesAsync(IMartenSession session, CancellationToken token)                                                         
  at async Task Marten.Internal.Sessions.DocumentSessionBase.ExecuteBatchAsync(IUpdateBatch batch, CancellationToken token)                                            
  at async Task Marten.Internal.Sessions.DocumentSessionBase.ExecuteBatchAsync(IUpdateBatch batch, CancellationToken token)                                            
  at async Task Marten.Internal.Sessions.DocumentSessionBase.SaveChangesAsync(CancellationToken token)                                                                 
  at async Task Marten.Events.Daemon.ProjectionDaemon.teardownExistingProjectionProgress(IProjectionSource source, CancellationToken token,                            
     IReadOnlyList<AsyncProjectionShard> shards)                                                                                                                       
  at async Task Marten.Events.Daemon.ProjectionDaemon.teardownExistingProjectionProgress(IProjectionSource source, CancellationToken token,                            
     IReadOnlyList<AsyncProjectionShard> shards)                                                                                                                       
  at async Task Marten.Events.Daemon.ProjectionDaemon.rebuildProjection(IProjectionSource source, TimeSpan shardTimeout, CancellationToken token)                      
  at void Marten.CommandLine.Commands.Projection.ProjectionHost.<>c__DisplayClass8_0.<<TryRebuildShards>b__1>d.MoveNext()                                              
  at void System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()                                                                                
  at async Task<RebuildStatus> Marten.CommandLine.Commands.Projection.ProjectionHost.TryRebuildShards(IProjectionDatabase database, IReadOnlyList<AsyncProjectionShard>
     asyncProjectionShards, TimeSpan? shardTimeout)                                                                                                                    
  at async Task<bool> Marten.CommandLine.Commands.Projection.ProjectionController.Execute(ProjectionInput input)                                                       
  at async Task<bool> Marten.CommandLine.Commands.Projection.ProjectionsCommand.Execute(ProjectionInput input)                                                         
  at async Task<int> Oakton.CommandExecutor.execute(CommandRun run)                                                                                                    

          DogView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%  
      BreederView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ⣷
MemberBreederView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ⣷
      MembersView:All ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ⣷
                                                                     %                                                                                                 (base) ➜  Web git:(backend/include-member-details-in-contact-list) ✗

Heading to bed now, so I will pick this up tomorrow night or the next night!

jeremydmiller commented 1 year ago

First, I want a way to stop the progressbars from clearing out the screen (missing error log statements obviously makes it really hard to figure out what the root cause is)

In a perfect world I would love Oakton to split the Console into to outputs, one with a streaming log, and one with the progressbars :)

In the end the PostgreSQL error is probably my fault somewhere and I can fix it easier :D

And in a different universe, using stderr would also be amazing :D dotnet run -- projections --rebuild 2> errror.log But I need to learn more about Dotnet Logging for that :D

Dude, that's one giant "I take pull requests" :-)

jeremydmiller commented 1 year ago

@oskardudycz I'm closing this with yet more information in that exception message. I think this was a one off that was specific to this particular usage. I also added a little better resiliency/retry for writing dead letter events w/ some code stolen from Wolverine.