uploaded file is crashed

ri8ika commented 2 years ago

In mac machine, you can compare this issue by getting info of the file ("Get Info"). Under "More Info", there's no any value compared to original file. Additionally in "Preview", you can not see the preview (We can understand it is crashed). If the file is of text type, then the preview section will show the information in json format like metadata etc.

Now, let's analyze what happens on both mac and linux machines:

Mac machine: When serving the file for the first time it loads the actual content. (Correct behavior) After restarting the server and serving the file it loads the actual content. (Correct behavior)

Linux machine: When serving the file for the first time it loads the actual content. (Correct behavior) After restarting the server and serving the file it loads json content with the metadata etc information. (Wrong behavior)

Even though I have mentioned Correct behavior, in fact is not correct behavior. Because file content is uploaded / modified with some kind of json data (metadata, etc.) than actual content which is not correct - is obviously a bug.

This looks like the issue is similar to: https://github.com/fsouza/fake-gcs-server/issues/296

But I have created this issue so that you guys can debug and solve the issue. Hope this issue helps fixing the bug!

fsouza commented 2 years ago

Yeah this is related to us not persisting metadata properly I think. I believe the work @gaul is doing in #683 should fix this. My plan is to review that PR before the end of this week. It should be good to go though, as @gaul definitely know what he's doing. It's just a matter of me finding the time (let me know if you can help test it!)

ri8ika commented 2 years ago

Definitely. I have checked that as well and continuously watch the changes. If something is wrong (ie. result is not as expected) then I'll point out and let you know here.

By the way, I'm also debugging on this, if I could do something then I'd be glad to contribute.

ri8ika commented 2 years ago

Hi @fsouza

I have tested #683 PR and it's working great! 🙇 However, I have listed out some issues.

Currently, I have tested in Mac machine only:

Issue 1:

Run server with docker, and upload file.
user.metadata: operation not supported (xattr.Set)

Issue 2:

When serving video (tested only with .mp4 format),
it's being downloaded instead of loading

Issue 3:

Step 1: run server with go command
Step 2: uploaded file
Step 3: stop server
Step 4: run server with docker
uploaded file is there but it starts with empty,
that is to say uploaded file is not available while running with docker

Issue 4:

When serving pdf file, and svg image: (Other formats are working fine)
It's not loading actual content, it's showing some encoded characters.
However, restarting the server, it's showing actual content, no issue.

Issue 5:

Not able to play audio file when serving in browser, tested only in latest chrome browser.
However, it's being played locally.
Tested only with .mp3

I will let you know if I find any issue later.

gaul commented 2 years ago

Issue 1: Run server with docker, and upload file. user.metadata: operation not supported (xattr.Set)

Docker has a variety of configurations so please be precise about what you are doing. Which path do you configure your -filesystem-root to point to? Is it inside the Docker container or does it point to a path on the macOS host? Does the path exist in /tmp which does not support xattr? If so you must you a different path and perhaps fake-gcs-server could warn more eagerly about this.

Issue 2: When serving video (tested only with .mp4 format), it's being downloaded instead of loading

Did you create these files with the current version of fake-gcs-server or were they externally created and exposed by fake-gcs-server? If the latter, you want to look at https://github.com/fsouza/fake-gcs-server/blob/0d523fe7c3677a92dc0bab97ee97777a91588d58/internal/backend/fs.go#L205 which can do something more intelligent when the xattr are missing or blank. Specifically you want to guess the Content-Type from the file extension, e.g., .pdf becomes application/pdf. There are a few libraries that can do this.

I believe that issues 3-5 are similar to 2 so let's work through that first. This is a new feature so it may take some more debugging to understand. If you are comfortable programming in Go please submit any PRs to help improve fake-gcs-server!

ri8ika commented 2 years ago

Oops, sorry. It's my mistake. I din't set content type. After checking it again:

Issue 1: Data mounted on docker:

services:
  storage:
    volumes:
          - /Users/ri8ika/Documents/data:/Users/ri8ika/Documents/data

And yes, I was using wrong data path in ENTRYPOINT by mistake. Now, it shows error while starting the server. Not sure about /tmp, it's created by running server by golang automatically, right? Can you please help me fix on this?

Issue 2: (Has issue) I have set content type to "video/mp4" Not able to play video.

Issue 3: (Issue persist) Maybe fixing issue 1 may help this?

Issue 4: (No issue) Ah, application/pdf worked fine, I was using octet stream - my mistake. Same mistake did with svg file - din't define content type.

Issue 5: (Issue persist) Content type is set to "audio/mpeg" Also tried with "audio/mp3"

I am fresher in Golang. Will contribute anytime if I can do something. Thanks.

gaul commented 2 years ago

I believe that issue 1 relates to the lack of macOS Docker filesystem sharing (osxfs): https://docker-docs.netlify.app/docker-for-mac/osxfs/#extended-attributes. You can try the referenced PR that communicates errors more completely.

Your experience suggests that the Windows fallback logic might be useful for macOS and I guess Linux tmpfs as well.

gaul commented 2 years ago

You might try running fake-gcs-server -data to copy from the macOS bind mount into a directory inside the Docker image as a workaround.

ri8ika commented 2 years ago

Didn't work:

server started at http://[::]:4443

Linux tmpfs as well.

rootDir does not support user extended attributes:
Consider using a different filesystem type, e.g., not tmpfs on Linux

I think we need to create a different issue for solving Issue 3, it's about persisting data between normal server start and start with docker. And work fixing issue 2 and 5 here. I am currently trying to figure out why video and audio is not being played in browser.

gaul commented 2 years ago

@ri8ika Could you file an issue for oxsfs xattr support at https://github.com/docker/for-mac/issues and link to it here? I lack the correct setup and this requires inputting some live diagnostic metadata or else they will just close the issue.

ri8ika commented 2 years ago

I am not pretty sure about diagnosing. I would be glad if you could guide me on this? Can we talk somewhere on Slack?

I tried self diagnose tool, but failed: https://docs.docker.com/desktop/mac/troubleshoot/#self-diagnose-tool

Failed with: vm has not started: vm has not started

Should I install VM or what steps need to take? Ouch, they have some issue here: https://github.com/docker/for-mac/issues/6091

fsouza commented 2 years ago

Hmm interesting. In my use cases I'd usually not mount the volume from Docker (I also haven't used docker4mac in a while). If this is a common use case, perhaps we should generalize the Windows fallback, but no need to go for that now I think, we can watch for other issues / comments from other users.

ri8ika commented 2 years ago

Yup! You are right. We can ignore docker issue for sometime. But what about the issue of video/audio not playing in browser. I don't think this should be ignored. There could be simple solution for this like fixing header or so.

fsouza commented 2 years ago

@ri8ika oh yeah absolutely. We can fix it to return whatever GCS returns. Curious if GCS does it based on metadata or automatically.

fsouza / fake-gcs-server

uploaded file is crashed #684