parcelvoy / platform

Parcelvoy: Open source multi-channel marketing automation platform. Send data-driven emails, sms, push notifications and more!
https://parcelvoy.com
MIT License
258 stars 47 forks source link

Not able to configure S3 on EC2 Instance #483

Closed tanmay-predisai closed 3 months ago

tanmay-predisai commented 3 months ago

Hello - We are trying to configure S3 by setting the parameters in the .env file.

When we try and upload a file, we get a 400 bad request on https:///api/admin/projects/1/images with the response as "{"status":"error","error":"An error occurred with this request."}". We observe this in the browser network tab.

If i hit the same URL directly, https://pv.predis.ai/api/admin/projects/1/images, i get this kind of output - {"results":[{"id":1,"created_at":"2024-07-24T06:20:39.000Z","updated_at":"2024-07-24T06:20:40.000Z","project_id":1,"uuid":"33537321-1a57-4bff-af17-bf8ae6ecb18a","name":"screenshot_2024_07_24_115023","original_name":"Screenshot 2024-07-24 115023","extension":".png","alt":null,"file_size":141192,"URL":"/33537321-1a57-4bff-af17-bf8ae6ecb18a.png"}],"limit":25}

Is there a way I can see the logs on what's going wrong?

We have set the following params in our .env file STORAGE_DRIVER=s3 STORAGE_BASE_URL=https:///parcelvoy-assets/ STORAGE_S3_BUCKET=(our s3 bucket name) AWS_REGION=us-east-1 AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= STORAGE_S3_ENDPOINT=parcelvoy-assets STORAGE_S3_FORCE_PATH_STYLE=true

Thank you for your help!

pushchris commented 3 months ago

Assuming you are running this in Docker Compose you should be able to view the logs by running docker compose logs to see what is being output

tanmay-predisai commented 3 months ago

Thank you for your reply. This is the error i see in the docker compose logs when I try to upload a new file -

error: { api-1 | "input": "parcelvoy-assets", api-1 | "code": "ERR_INVALID_URL", }

Is there a problem in one of these configurations?

STORAGE_DRIVER=s3 STORAGE_BASE_URL=https://assets.predis.ai/parcelvoy-assets/ STORAGE_S3_BUCKET=user-uploaded-img AWS_REGION=us-east-1 AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= STORAGE_S3_ENDPOINT=parcelvoy-assets STORAGE_S3_FORCE_PATH_STYLE=true

Or would it be possible to add some more description on how to get these values - esp - STORAGE_S3_BUCKET, STORAGE_S3_ENDPOINT,STORAGE_S3_FORCE_PATH_STYLE please.

Thank you for your help.

here is the log output :

[1723090496892] ERROR (9 on 6fb2dfceaa3a): error api-1 | ctx: { api-1 | "request": { api-1 | "method": "POST", api-1 | "url": "/api/admin/projects/1/images", api-1 | "header": { api-1 | "host": "127.0.0.1:3001", api-1 | "connection": "close", api-1 | "content-length": "1204962", api-1 | "sec-ch-ua": "\"Not)A;Brand\";v=\"99\", \"Google Chrome\";v=\"127\", \"Chromium\";v=\"127\"", api-1 | "accept": "application/json, text/plain, /", api-1 | "sec-ch-ua-platform": "\"macOS\"", api-1 | "sec-ch-ua-mobile": "?0", api-1 | "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36", api-1 | "content-type": "multipart/form-data; boundary=----WebKitFormBoundaryvLZTTAKvkSulN4ZC", api-1 | "origin": "https://pv.predis.ai", api-1 | "sec-fetch-site": "same-origin", api-1 | "sec-fetch-mode": "cors", api-1 | "sec-fetch-dest": "empty", api-1 | "referer": "https://pv.predis.ai/projects/1/campaigns/11/editor?locale=en", api-1 | "accept-encoding": "gzip, deflate, br, zstd", api-1 | "accept-language": "en-GB,en-US;q=0.9,en;q=0.8", api-1 | "cookie": "_fbp=fb.1.1712838844905.406076715; crisp-client%2Fsession%2F94435076-ab60-4eec-8a60-50044deec72f=session_d8152aec-a117-4c0d-945d-cd5df2188775; _tt_enable_cookie=1; _ttp=vKlXB2mRMYkO8lcK2qgExgNLypN; _gcl_au=1.1.325747883.1720617126; _fbc=fb.1.1721824136118.IwZXh0bgNhZW0BMAABHXRtM8ELVnfFWFg7Zq8x5OH2TmT-RNIAxExzI4Et8fWOAbK4Y1cRZOKFdg_aem_reko4qsaQNXgBjJUVQZRsA; pred_ref=page-fam; pref_ref_location=use-cases/facebook-ad-maker/; pred_history=1721913454002|page-fam|/use-cases/facebook-ad-maker; cf_clearance=M_XQhgDMjCrsGOq6oo5W5bCGlvJw_ShVdpAHFm4Do34-1722860950-1.0.1.1-pp2yfCimyywKbnr0nXKPtBHFMaeOPkJuxk8pneNQA.6_TB37BEoPsijGG.WlEH7jl0Q4sNiZWOwqDSATethxFA; _ga=GA1.1.632535694.1712838844; _clck=76buwp%7C2%7Cfo4%7C0%7C1562; oauth={\"access_token\":\"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpasI6MSwib3JnYW5pemF0aW9uX2lkIjoxLCJyb2xlIjoib3duZXIiLCJleHAiOjE3MjMxMTQ3NDksImlhdCI6MTcyMzAyODM0OX0.uzgz-0bsB_TFb7kzY-YqDYgHOeirHIN7xotg_ko3yaE\",\"expires_at\":\"2024-08-08T10:59:09.916Z\"}; oauth.sig=zJh0uQ-TIQNo94nas1xYkuVMMzFc; _ga_PQJ6V7Z34C=GS1.1.1723090210.429.0.1723090210.60.0.165529106" api-1 | } api-1 | }, api-1 | "response": { api-1 | "status": 404, api-1 | "message": "Not Found", api-1 | "header": { api-1 | "vary": "Origin", api-1 | "access-control-allow-origin": "" api-1 | } api-1 | }, api-1 | "app": { api-1 | "subdomainOffset": 2, api-1 | "proxy": true, api-1 | "env": "production" api-1 | }, api-1 | "originalUrl": "/api/admin/projects/1/images", api-1 | "req": "", api-1 | "res": "", api-1 | "socket": "" api-1 | } api-1 | error: { api-1 | "input": "parcelvoy-assets", api-1 | "code": "ERR_INVALID_URL", api-1 | "headers": { api-1 | "Access-Control-Allow-Origin": "", api-1 | "vary": "Origin" api-1 | } api-1 | } api-1 | api-1 | TypeError [ERR_INVALID_URL]: Invalid URL api-1 | at new NodeError (node:internal/errors:405:5) api-1 | at new URL (node:internal/url:676:13) api-1 | at parseUrl (/usr/src/app/node_modules/@smithy/url-parser/dist-cjs/index.js:29:21) api-1 | at toEndpointV1 (/usr/src/app/node_modules/@smithy/middleware-endpoint/dist-cjs/index.js:112:41) api-1 | at Object.customEndpointProvider [as endpoint] (/usr/src/app/node_modules/@smithy/middleware-endpoint/dist-cjs/index.js:232:65) api-1 | at process.processTicksAndRejections (node:internal/process/task_queues:95:5) api-1 | at async Promise.all (index 1) api-1 | at async _Upload.uploadUsingPut (/usr/src/app/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:230:22) api-1 | at async _Upload.doConcurrentUpload (/usr/src/app/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:293:18) api-1 | at async Promise.all (index 0) api-1 | at async _Upload.__doMultipartUpload (/usr/src/app/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:379:5) api-1 | at async _Upload.done (/usr/src/app/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:203:12) api-1 | at async S3StorageProvider.upload (/usr/src/app/build/storage/S3StorageProvider.js:32:9) api-1 | at async Storage.upload (/usr/src/app/build/storage/Storage.js:44:9) api-1 | at async Storage.save (/usr/src/app/build/storage/Storage.js:32:9) api-1 | at async uploadImage (/usr/src/app/build/storage/ImageService.js:11:20)

leobarcellos commented 3 months ago

Use this variable like this: STORAGE_S3_ENDPOINT=https://s3.us-east-1.amazonaws.com

More info on https://docs.aws.amazon.com/general/latest/gr/s3.html

tanmay-predisai commented 3 months ago

@leobarcellos Thank you for your help. This works!

Another question- Would you know how can I specify the key (subfolder) name here? The structure we wanted to have here was /parcelvoy-assets/. As of now, it's working for /.

leobarcellos commented 3 months ago

Not sure, I use it like you said (example.com/)

Try removing the trailing slash: STORAGE_BASE_URL=https://assets.predis.ai/parcelvoy-assets

pushchris commented 3 months ago

@tanmay-predisai the solution above is correct, however you can also just remove STORAGE_S3_FORCE_PATH_STYLE from your config or set it to false and it should work as well. That setting disables AWS defaults for endpoint generation for if you want to use another S3-compatible service for uploads such as Digital Ocean Spaces. You can find the original implementation here: https://github.com/parcelvoy/platform/pull/338. It is not possible currently to upload to a given folder, you are welcome to contribute changes to make this possible however!

tanmay-predisai commented 3 months ago

@pushchris this works for us. I appreciate your help!

We plan to start using PV fully from next week. I will keep you posted on the progress :)