Send file has filesize of 0 bytes, likely due to failed authentication

When adding a file via the new "send" feature, I am able to upload a file successfully. However, when going to download it, it says that it has 0 bytes, and clicking on the download button says that the file associated with the Send could not be found.

I've looked at the logs in docker-compose, and I'm not seeing a ton of interesting stuff. This it he log from when I upload the file.

bitwarden-api    | info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
bitwarden-api    |       Request starting HTTP/1.0 POST http://redacted/sends/file multipart/form-data; boundary=---------------------------296041502342539414651855084487 148314
bitwarden-api    | info: Microsoft.AspNetCore.Cors.Infrastructure.CorsService[4]
bitwarden-api    |       CORS policy execution successful.
bitwarden-api    | info: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[2]
bitwarden-api    |       Successfully validated the token.
bitwarden-api    | info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[1]
bitwarden-api    |       Authorization was successful.
bitwarden-api    | info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
bitwarden-api    |       Executing endpoint 'Bit.Api.Controllers.SendsController.PostFile (Api)'
bitwarden-api    | info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[3]
bitwarden-api    |       Route matched with {action = "PostFile", controller = "Sends"}. Executing controller action with signature System.Threading.Tasks.Task`1[Bit.Core.Models.Api.SendResponseModel] PostFile() on controller Bit.Api.Controllers.SendsController (Api).
bitwarden-identity | info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
bitwarden-identity |       Request starting HTTP/1.1 POST http://identity:5000/connect/token application/x-www-form-urlencoded 132
bitwarden-identity | info: IdentityServer4.Hosting.IdentityServerMiddleware[0]
bitwarden-identity |       Invoking IdentityServer endpoint: IdentityServer4.Endpoints.TokenEndpoint for /connect/token
bitwarden-identity | warn: Bit.Core.IdentityServer.ResourceOwnerPasswordValidator[12482444]
bitwarden-identity |       Failed login attempt. ::ffff:172.30.0.2
bitwarden-identity | info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
bitwarden-identity |       Request starting HTTP/1.1 POST http://identity:5000/connect/token application/x-www-form-urlencoded 132
bitwarden-identity | info: IdentityServer4.Hosting.IdentityServerMiddleware[0]
bitwarden-identity |       Invoking IdentityServer endpoint: IdentityServer4.Endpoints.TokenEndpoint for /connect/token
bitwarden-identity | warn: Bit.Core.IdentityServer.ResourceOwnerPasswordValidator[12482444]
bitwarden-identity |       Failed login attempt. ::ffff:172.30.0.2
bitwarden-api    | info: Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor[1]
bitwarden-api    |       Executing ObjectResult, writing value of type 'Bit.Core.Models.Api.SendResponseModel'.
bitwarden-api    | info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2]
bitwarden-api    |       Executed action Bit.Api.Controllers.SendsController.PostFile (Api) in 242.9628ms
bitwarden-api    | info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
bitwarden-api    |       Executed endpoint 'Bit.Api.Controllers.SendsController.PostFile (Api)'
bitwarden-api    | info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
bitwarden-api    |       Request finished in 265.547ms 200 application/json; charset=utf-8

The most likely suspect is

bitwarden-identity | info: IdentityServer4.Hosting.IdentityServerMiddleware[0]
bitwarden-identity |       Invoking IdentityServer endpoint: IdentityServer4.Endpoints.TokenEndpoint for /connect/token
bitwarden-identity | warn: Bit.Core.IdentityServer.ResourceOwnerPasswordValidator[12482444]
bitwarden-identity |       Failed login attempt. ::ffff:172.30.0.2
bitwarden-identity | info: Microsoft.AspNetCore.Hosting.Diagnostics[1]

Upon further inspection of the, I also see this occasionally, and I'm not really sure what to make of it

bitwarden-identity | fail: IdentityServer4.Validation.TokenRequestValidator[0]
bitwarden-identity |       Custom token request validator{ error = invalid_grant }, details: {
bitwarden-identity |         "ClientId": "internal.Api",
bitwarden-identity |         "GrantType": "client_credentials",
bitwarden-identity |         "Scopes": "internal",
bitwarden-identity |         "Raw": {
bitwarden-identity |           "grant_type": "client_credentials",
bitwarden-identity |           "scope": "internal",
bitwarden-identity |           "client_id": "internal.Api",
bitwarden-identity |           "client_secret": "***REDACTED***"
bitwarden-identity |         }
bitwarden-identity |       }

My googling attempts of this error have come up empty. I'm not sure if this is a bug or an issue with my self-hosted setup. Is there anything I can poke at?

Hi @ollien , if you check your storage location mapped to the Attachments container for storing Attachments, you should see a /Send directory under there now, where items would be stored. If you're not seeing that, perhaps there's a missing permission or other issue there. We've done some extensive testing with self-hosted and haven't run into an issue like this, however perhaps there's something else going on.

As far as the failed login attempts, I wouldn't think the Send create would actually be successful in that instance, but not entirely sure. If there's anything specific about your rig/environment in addition to what you have here and can share that would be great to help pinpoint this a little further...

Are you able to upload AND download normal attachments on vault items?
Which client are you using (just the web vault)?
What size file are you attempting to attach?
What is the file name "length" (perhaps a header issue)?
Can you find the Send file in the file system (attachments/Send) and see if the file actually exists there (it'll be encrypted, but should match the Send ID in a sub-folder) and has a file length?
If you pop open your browser developer tools console, are you seeing any CSP or other security errors when attempting to download the file?

It looks like I have just an empty temp directory in core/attachments

# tree attachments
attachments
└── temp

777ing both of these dirs didn't fix the problem, so that rules out a perms issue.

I am able to upload, in that it lets me perform the upload process, but there seems to be no indication that the data is going anywhere. Is there anywhere I could check?
Just the web vault, yeah
It's a 144k image.
The filename is just "IMG_20210311_2341589.jpg" so I doubt it's anything like a header issue...
I think this is answered above, but there's no attachments/Send dir. Making the directory by hand and setting the perms to be the same as

Nope, no networking/CSP issues. I get a response back with this (statuscode 200)

{"Object":"send-fileDownload","Id":"upz6uazfrdlrrwa7pqp23ea1qc3romln","Url":null}

As for special things about the setup: All of my containers are on a separate network that I've added in my global overrides, but this has been true for ages and hasn't caused an issue before.

Here's my docker-compose.override.yml

version: '3'
services:
    mssql:
        restart: unless-stopped
        networks:
            - urlredacted.gg
    web:
        restart: unless-stopped
        networks:
            - urlredacted.gg
    attachments:
        restart: unless-stopped
        networks:
            - urlredacted.gg
    api:
        restart: unless-stopped
        networks:
            - urlredacted.gg
    identity:
        restart: unless-stopped
        networks:
            - urlredacted.gg

    sso:
        restart: unless-stopped
        networks:
            - urlredacted.gg
    admin:
        restart: unless-stopped
        networks:
            - urlredacted.gg
    portal:
        restart: unless-stopped
        networks:
            - urlredacted.gg
    icons:
        restart: unless-stopped
        networks:
            - urlredacted.gg
    notifications:
        restart: unless-stopped
        networks:
            - urlredacted.gg
    events:
        restart: unless-stopped
        networks:
            - urlredacted.gg
    nginx:
        container_name: bitwarden
        restart: unless-stopped
        networks:
            - urlredacted.gg

networks:
    urlredacted.gg:
        external:
            name: urlredacted.gg

@ollien, The zero file length might indicate that the file either never being read from or is failing just before updating this zero length to a real one. I'm inclined to think there's something going wrong during saving the send, prior to writing the attachment to disk, since you don't even have the send directory created. An error during write would actually delete the files, but should leave the send parent directory intact.

When uploading the Send, do you have any warnings or errors returned when creating the send in developer utilities?

There are actually some CSP issues on page load that I didn't notice previously (before I was only looking at download, not upload :sweat_smile: )

Content Security Policy: The page’s settings blocked the loading of a resource at eval (“default-src”). core.js:5268
Content Security Policy: The page’s settings blocked the loading of a resource at eval (“default-src”). 9 core.js:5268
Content Security Policy: The page’s settings blocked the loading of a resource at eval (“default-src”). 2 core.js:5268
GETwss://my_instance/notifications/hub?access_token=redacted
[HTTP/1.1 400 Bad Request 133ms]

Content Security Policy: The page’s settings blocked the loading of a resource at eval (“default-src”). 3 core.js:5268
Content Security Policy: The page’s settings blocked the loading of a resource at eval (“default-src”). 3 core.js:5268
Firefox can’t establish a connection to the server at wss://my_instance/notifications/hub?access_token=redacted
[2021-03-19T20:50:31.930Z] Error: Failed to start the connection: Error: There was an error with the transport. Utils.js:198

Not sure if those access tokens actually contain any information, so I'm removing them just in case. If there's anything in there that might be helpful, let me know.

That said, as for when I actually send the file, I get some of these

Content Security Policy: The page’s settings blocked the loading of a resource at eval (“default-src”). 12 core.js:5268
Content Security Policy: The page’s settings blocked the loading of a resource at eval (“default-src”). 4 core.js:5268

That core.js:5268 is here

function createNamedArrayType(name) {
    // This should never be called in prod mode, so let's verify that is the case.
    if (ngDevMode) {
        try {
            // We need to do it this way so that TypeScript does not down-level the below code.
            var FunctionConstructor = createNamedArrayType.constructor;
            // this guy is the one that the stack trace points to.
            return (new FunctionConstructor('Array', "return class " + name + " extends Array{}"))(Array);
        }
        catch (e) {
            // If it does not work just give up and fall back to regular Array.
            return Array;
        }
    }

Is this something on my end that I would need to tweak in my nginx instance? Not sure if this is a CSP issue caused by something on my end or not.

I did a bit of digging this afternoon. It looks like if I add unsafe-eval to the CSP (which isn't ideal), those CSP errors seem to go away. I don't know enough about Angular to understand exactly why that devmode path is activated, but I wonder if something is up there...

Also, I was able to make those websocket errors go away by tweaking my reverse proxy (I have a reverse proxy in front of the standard Bitwarden nginx server) settings to include your Upgrade headers. After doing that, the above errors go away (and there are no CSP errors in my console on upload), but the problem still persists.

I also tried creating the send directory by hand, to no avail.

A couple of follow-up questions: should uploading a file produce anything in the log of the bitwarden-attachments container? There's no request I'm seeing, there. Only calls to /alive (which I assume is some kind of heartbeat. I'm also not seeing any kind of indication that a request is failing either. This is what I see in docker-compose logs. Should there be? I'm able to hit the attachments container from the nginx container, so I'm not sure there's an issue there, either.

bitwarden-api    | info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
bitwarden-api    |       Request starting HTTP/1.0 POST http://my_instance/sends/file multipart/form-data; boundary=---------------------------8417346213975013032420093732 344116
bitwarden-api    | info: Microsoft.AspNetCore.Cors.Infrastructure.CorsService[4]
bitwarden-api    |       CORS policy execution successful.
bitwarden-api    | info: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[2]
bitwarden-api    |       Successfully validated the token.
bitwarden-api    | info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[1]
bitwarden-api    |       Authorization was successful.
bitwarden-api    | info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
bitwarden-api    |       Executing endpoint 'Bit.Api.Controllers.SendsController.PostFile (Api)'
bitwarden-api    | info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[3]
bitwarden-api    |       Route matched with {action = "PostFile", controller = "Sends"}. Executing controller action with signature System.Threading.Tasks.Task`1[Bit.Core.Models.Api.SendResponseModel] PostFile() on controller Bit.Api.Controllers.SendsController (Api).
bitwarden-identity | info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
bitwarden-identity |       Request starting HTTP/1.1 POST http://identity:5000/connect/token application/x-www-form-urlencoded 132
bitwarden-identity | info: IdentityServer4.Hosting.IdentityServerMiddleware[0]
bitwarden-identity |       Invoking IdentityServer endpoint: IdentityServer4.Endpoints.TokenEndpoint for /connect/token
bitwarden-identity | warn: Bit.Core.IdentityServer.ResourceOwnerPasswordValidator[12482444]
bitwarden-identity |       Failed login attempt. ::ffff:172.30.0.9
bitwarden-identity | info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
bitwarden-identity |       Request starting HTTP/1.1 POST http://identity:5000/connect/token application/x-www-form-urlencoded 132
bitwarden-identity | info: IdentityServer4.Hosting.IdentityServerMiddleware[0]
bitwarden-identity |       Invoking IdentityServer endpoint: IdentityServer4.Endpoints.TokenEndpoint for /connect/token
bitwarden-identity | warn: Bit.Core.IdentityServer.ResourceOwnerPasswordValidator[12482444]
bitwarden-identity |       Failed login attempt. ::ffff:172.30.0.9
bitwarden-api    | info: Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor[1]
bitwarden-api    |       Executing ObjectResult, writing value of type 'Bit.Core.Models.Api.SendResponseModel'.
bitwarden-api    | info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2]
bitwarden-api    |       Executed action Bit.Api.Controllers.SendsController.PostFile (Api) in 32.431ms
bitwarden-api    | info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
bitwarden-api    |       Executed endpoint 'Bit.Api.Controllers.SendsController.PostFile (Api)'
bitwarden-api    | info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
bitwarden-api    |       Request finished in 49.9258ms 200 application/json; charset=utf-8

Everything same here. Any updates?

Hi @ollien, We're cleaning up our repositories in preparation for a major reorganization. Issues from last year will be marked as stale and closed after two weeks. If you still need help, comment to let us know and we'll look into it. Thanks!

This issue is still present

bitwarden / server

Send file has filesize of 0 bytes, likely due to failed authentication #1206