Unified Beta. Cannot login anymore after adding 2FA

[kmatasfp]

Steps To Reproduce

1. Run unified beta behind reverse proxy, in my case it is nginx. Happens both with beta and dev tag
2. Add 2FA, does not matter if it is regular TOTP or WebAuthn.
3. Try to login

Expected Result

Two-step login screen appears like this:

Actual Result

I get red unexpected error occured from all the clients - web, desktop, android.

When I look at the response that web client returns I see:

Request URL: https://bitwarden.my-domain.com/identity/connect/token
Request Method: POST
Status Code: 405 
Remote Address: 172.67.154.126:443
Referrer Policy: same-origin

Sent request headers:

:authority: bitwarden.my-domain.com
:method: POST
:path: /identity/connect/token
:scheme: https
accept: application/json
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
auth-email: ZHdasdashhhhhhhU //retracted
bitwarden-client-name: web
bitwarden-client-version: 2023.1.1
content-length: 229
content-type: application/x-www-form-urlencoded; charset=utf-8
device-type: 9
origin: https://bitwarden.my-domain.com
referer: https://bitwarden.my.domain.com
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-origin
sec-gpc: 1
user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36

Screenshots or Videos

No response

Additional Context

Removing 2FA from a user fixes the login. Not sure if there is something wrong with my nginx setup, any guidance would be helpful here.

Githash Version

34544f22-dirty

Environment Details

Ubuntu 22.04 VM

Database Image

postgres: 15

Issue-Link

https://github.com/bitwarden/server/issues/2480

Issue Tracking Info

• [X] I understand that work is tracked outside of Github. A PR will be linked to this issue should one be opened to address it, but Bitwarden doesn't use fields like "assigned", "milestone", or "project" to track progress.

[kmatasfp]

Only difference I could see is that the cloud version is sending some cookie as well, which does not seem to happen with my on prem installation. And I was not able to see any errors in the container logs. The errors comes from zone.js

[kmatasfp]

Tried few more things, if I removed reverse proxy and set BW_DOMAIN to the external address of the container I got lot of JS errors that looked like this when I tried to log in

[justindbaur]

Getting a 405 Method Not Allowed from a POST on /connect/token sounds like a reverse proxy issues but the fact that removing 2FA fixes the issue it sounds like our problem. Can you post any errors you find in /var/log/bitwarden/identity.log?

[kmatasfp]

Yeah it seems to be completely reverse proxy issue as I suspected, I was not able to figure it out how to get it working with nginx, probably some middleware I have in place for "mitigating exploits" is blocking this 🤷. If I tried with vanilla traefik it worked out of the box, mostly - app notifications/automatic vault sync does not work but othewise I am able to log in with 2fa and update and add passwords. Be great if somewhere it was documented what headers need to be set/passed for all the functions to work in case of self hosting behind reverse proxy

[roberto-sartori-gl]

Can you post your nginx configuration? I'm running nginx and the latest bitwarden unified server (2023.2) and I don't have issues with 2FA.

[kmatasfp]

ok let me see what I can find out regarding the nginx conf, it was not a regular nginx it was ingress-nginx as I am running it in kubernetes, currently swapped ingress controller for traefik as it gets me mostly working experience, apps still need manual sync

[kmatasfp]

And I imagine the automatic sync does not work because I see these in the logs

 Bit.Core.Services.RelayPushRegistrationService[0]
      => SpanId:fb36ca2c575e92bc, TraceId:f9f133865393b0fe95f2d2f2604c5806, ParentId:0000000000000000 => ConnectionId:0HMONT99CRKBK => RequestPath:/identity/connect/token RequestId:0HMONT99CRKBK:00000002
      Request to https://push.bitwarden.com/push/register is unsuccessful with status of BadRequest-Bad Request

[roberto-sartori-gl]

In my case, sync only works on my Android phone: if I change a password from my browser , on a computer, the sync is automatic con the phone. The opposite is not true, the Chrome extension needs a manual sync.

[kmatasfp]

Also my troubles might have been related to this, was going thought the logs recommended by @justindbaur and saw following:

 Microsoft.AspNetCore.Server.Kestrel[13]
      => SpanId:4e57971fd1e30424, TraceId:f84dff81850ab141f6d818ab496a335e, ParentId:0000000000000000 => ConnectionId:0HMOP3I379T6T => RequestPath:/identity/connect/token RequestId:0HMOP3I379T6T:00000002
      Connection id "0HMOP3I379T6T", Request id "0HMOP3I379T6T:00000002": An unhandled exception was thrown by the application.
      Npgsql.PostgresException (0x80004005): 42703: column u.UnknownDeviceVerificationEnabled does not exist

      POSITION: 802
         at Npgsql.Internal.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|215_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
         at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
         at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
         at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
         at Bit.Infrastructure.EntityFramework.Repositories.UserRepository.GetByEmailAsync(String email) in /source/src/Infrastructure.EntityFramework/Repositories/UserRepository.cs:line 21
         at Bit.Core.Identity.UserStore.FindByEmailAsync(String normalizedEmail, CancellationToken cancellationToken) in /source/src/Core/Identity/UserStore.cs:line 52
         at Microsoft.AspNetCore.Identity.UserManager`1.FindByEmailAsync(String email)
         at Bit.Identity.IdentityServer.ResourceOwnerPasswordValidator.ValidateAsync(ResourceOwnerPasswordValidationContext context) in /source/src/Identity/IdentityServer/ResourceOwnerPasswordValidator.cs:line 62
         at IdentityServer4.Validation.TokenRequestValidator.ValidateResourceOwnerCredentialRequestAsync(NameValueCollection parameters)
         at IdentityServer4.Validation.TokenRequestValidator.RunValidationAsync(Func`2 validationFunc, NameValueCollection parameters)
         at IdentityServer4.Validation.TokenRequestValidator.ValidateRequestAsync(NameValueCollection parameters, ClientSecretValidationResult clientValidationResult)
         at IdentityServer4.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context)
         at IdentityServer4.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context)
         at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events, IBackChannelLogoutService backChannelLogoutService)
         at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events, IBackChannelLogoutService backChannelLogoutService)
         at IdentityServer4.Hosting.MutualTlsEndpointMiddleware.Invoke(HttpContext context, IAuthenticationSchemeProvider schemes)
         at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
         at IdentityServer4.Hosting.BaseUrlMiddleware.Invoke(HttpContext context)
         at Bit.Core.Utilities.CurrentContextMiddleware.Invoke(HttpContext httpContext, ICurrentContext currentContext, GlobalSettings globalSettings) in /source/src/Core/Utilities/CurrentContextMiddleware.cs:line 19
         at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
         at Bit.SharedWeb.Utilities.ServiceCollectionExtensions.<>c__DisplayClass11_0.<<UseDefaultMiddleware>b__1>d.MoveNext() in /source/src/SharedWeb/Utilities/ServiceCollectionExtensions.cs:line 503
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Builder.Extensions.UsePathBaseMiddleware.InvokeCore(HttpContext context, PathString matchedPath, PathString remainingPath)
         at Bit.Identity.Startup.<>c__DisplayClass10_1.<<Configure>b__2>d.MoveNext() in /source/src/Identity/Startup.cs:line 174
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
        Exception data:
          Severity: ERROR
          SqlState: 42703
          MessageText: column u.UnknownDeviceVerificationEnabled does not exist
          Position: 802
          File: parse_relation.c
          Line: 3651
          Routine: errorMissingColumn

I was moving between beta and dev tags when I was troubleshooting my issue and playing around with different headers. Also the version I got working with "traefik" was clean install, dev tag -> dropped DB and recreated. When I was testing with nginx ingress I relied on db migrations when I swithced between versions, so probably I had 2 compunding issues going on - issue with bitwarden itself and me not having proper reverse proxy headers 🤷.

Long story short I was able reproduce my issue with "traefik" now, when I moved to beta tag (dev tag felt like too bleeding edge for me 😆 ). I saw the same error popping up as above, User.UnknownDeviceVerificationEnabled missing. After I ran following commands things seem to be working again:

alter table "User" alter column "UnknownDeviceVerificationEnabled" set default true;
update existing users to have `UnknownDeviceVerificationEnabled` = true;
alter table "User" alter column "UnknownDeviceVerificationEnabled" set not null;

Used https://github.com/bitwarden/server/blob/master/src/Sql/dbo/Tables/User.sql as reference