S3 Storage - Githubissues

palitu commented 1 year ago

Hi, i am trying to work out how the service exposes the gifs.

Is it done through the local webserver (ie, the server grabs the data from S3, then forwards it),
is it done by the browser, where the server sends a reference to the S3 object, and the client browser gets the asset directly from the object store.

I am going to use B2 (backblaze) and there is a question whether the new bucket should be public or private. I will start with private, as i do not think that i will allow others to access it, but was curious all the same.

thanks.

pietvanzoen commented 1 year ago

Media hosting is managed by your S3 provider. In AWS that would be Cloudfront with a bucket behind it.

The Gifable app is only responsible for managing the library, uploading images to the bucket, then knowing enough about the S3 setup to generate a URL that gets saved to the DB.

The key to that is this config: https://github.com/pietvanzoen/gifable/blob/f23569f00f9bd45a4cf80ffc3621e7b3eccfdd66/.env.example#L7

I've not used B2 backblaze before. I'd love to hear what the setup is like when you get it working. Thinking I should setup some docs for config on various providers. I'm using Cloudflare R2 with app.gifable.club

jason-weiser commented 9 months ago

@palitu if you are running this using Backblaze, would you be open to sharing how you configured it using the .env? I can’t seem to get it working at all and wonder if there are little idiosyncrasies I’m missing when it comes to all the different things I need to input.

pietvanzoen commented 9 months ago

Hey @jason-weiser. I managed to get backblaze working for me with this config:

S3_ENDPOINT=s3.us-east-005.backblazeb2.com
S3_BUCKET=gifable-club
S3_USE_SSL=true # required I think
S3_ACCESS_KEY=<key_id>
S3_SECRET_KEY=<secret>
S3_STORAGE_BASE_URL=https://f005.backblazeb2.com/file/gifable-club/

I found the endpoint on the buckets page:

The base url I found by just browsing files and looking at the "Friendly URL" in the file details modal.

Let me know if this helps. If not can you share what errors or behaviour you're seeing?

Phibonacci commented 7 months ago

I am running Gifable with docker and I installed MinIO as an s3 on the same server. I can connect to MinIO successfully using the mc tool in command line but Gifable is giving me the "Invalid" error anytime I try to upload a gif.

I do not know how to investigate this further.

S3 variables of my .env look this way:

S3_ENDPOINT=s3.mydomain.net # as shown in `mc admin --json info mys3` in info>servers>endpoint
S3_BUCKET=gif # As created in MinIO web UI
S3_STORAGE_BASE_URL=https://s3.mydomain.net/gif/ # Typing that in my browser shows an XML to the right bucket 
S3_ACCESS_KEY=access_key
S3_SECRET_KEY=secret_key # generated through the MinIO web UI
S3_REGION=
S3_BASE_PATH=
S3_PORT=443
S3_USE_SSL=true # I configured my nginx proxy properly and I can connect to https://s3.mydomain.net with the mc tool

In my browser https://s3.mydomain.net/gif/ shows the following (since I'm not sending the keys):

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied.</Message>
<BucketName>gif</BucketName>
<Resource>/gif/</Resource>
<RequestId>17A6DBF9817C650F</RequestId>
<HostId>
da9024bbb4ad464b049177c95ea2ebf374d3b3fd1af9251148b658df7ca2e3e8
</HostId>
</Error>

The only logs I get when refreshing the page are:

prisma:query SELECT 1
prisma:query SELECT `main`.`User`.`id`, `main`.`User`.`username`, `main`.`User`.`isAdmin`, `main`.`User`.`theme` FROM `main`.`User` WHERE (`main`.`User`.`id` = ? AND 1=1) LIMIT ? OFFSET ?
GET /media/new?uploadType=file 200 - - 5.080 ms

Clicking "upload" does not generate new logs.

Edit: I got it! It does not like spaces in file names! Now I can figure out the next error as the upload has failed.

I realise it's not even s3 related, yet.

Error: User not found
    at action5 (/app/build/index.js:2403:11)
    at Object.callRouteActionRR (/app/node_modules/@remix-run/node/node_modules/@remix-run/server-runtime/dist/data.js:35:16)
    at callLoaderOrAction (/app/node_modules/@remix-run/node/node_modules/@remix-run/router/router.ts:3550:16)
    at submit (/app/node_modules/@remix-run/node/node_modules/@remix-run/router/router.ts:2817:16)
    at queryImpl (/app/node_modules/@remix-run/node/node_modules/@remix-run/router/router.ts:2752:22)
    at Object.queryRoute (/app/node_modules/@remix-run/node/node_modules/@remix-run/router/router.ts:2702:18)
    at handleDataRequestRR (/app/node_modules/@remix-run/node/node_modules/@remix-run/server-runtime/dist/server.js:63:20)
    at requestHandler (/app/node_modules/@remix-run/node/node_modules/@remix-run/server-runtime/dist/server.js:37:18)
    at /app/node_modules/@remix-run/express/dist/server.js:39:22
POST /media/new?uploadType=file&_data=routes%2Fmedia.new 500 - - 97.844 ms

Alright, so you cannot upload as an admin. I created an user. Also you cannot set DISABLE_SIGNUP to 0 you need to comment it. Now I have an s3 error.

InvalidEndpointError: Invalid endPoint : s3.mydomain.net # e.g. "s3.amazonaws.com"
    at new TypedClient (/app/node_modules/minio/dist/main/internal/client.ts:112:13)
    at new Client (/app/node_modules/minio/dist/main/minio.js:76:8)
    at new S3Storage (/app/build/index.js:779:24)
    at storage2 (/app/build/index.js:775:10)
    at storeBuffer (/app/build/index.js:999:13)
    at action5 (/app/build/index.js:2406:23)
    at Object.callRouteActionRR (/app/node_modules/@remix-run/node/node_modules/@remix-run/server-runtime/dist/data.js:35:16)
    at callLoaderOrAction (/app/node_modules/@remix-run/node/node_modules/@remix-run/router/router.ts:3550:16)
    at submit (/app/node_modules/@remix-run/node/node_modules/@remix-run/router/router.ts:2817:16)
    at queryImpl (/app/node_modules/@remix-run/node/node_modules/@remix-run/router/router.ts:2752:22)

I tested with play.min.io to see if I would get the same error and I did, "Invalid endPoint".

Wait... is that the file comment I see in the log? Removing all trailing comments... ok, now it's working, I can upload a gif without an error message.

Now I need to figure out why the actual file fails to download, only showing a rectangle with the name of the file.

The file is not visible because the bucket is not public. But then wouldn't it expose the bucket to upload from anyone directly interacting with it?

Yes it would, so I kept it private and added a custom rule to make downloads public: mc anonymous set download mys3/gif

It's working, problem solved.

pietvanzoen commented 7 months ago

Hey @Phibonacci . Sorry it took me a while to get to this. Sounds like you battled through! Glad it's working now.

If you have any suggestions for docs or improved error messaging please let me know!

Phibonacci commented 7 months ago

It's perfectly fine. I kept editing my message so you could use my struggles as a user feedback.

It would help to have more explicit messages on the web interface when a filename is invalid. Maybe allowing space in filenames is doable?

I would suggest not using trailing comments in the .env file example as they are not supported. Moving them on the previous line should be fine.

I found the MinIO documentation hard to follow. Maybe this ticket will help others. I also had to fight with my nginx reverse proxy, although MinIO has a good documentation on the topic.

On a completely different topic I was surprised Gifable does not support MP4 as it is the preferred format of most "gif" creators nowadays. I converted mines using this great article. Maybe a future feature allowing to upload multiple formats of the same gif to then be able to pick one depending on the targeted platform?

Great work nonetheless.

pietvanzoen / gifable

S3 Storage #81