502 on image upload, but works fine with `ghost run`

ekerstein commented 4 years ago

I'm having a similar issue as #3827 and the post on the forum here.

I just updated to 2.31.1 and now whenever I upload an image I get a 502 error. I'm using a DO droplet.

I tried debugging with ghost run but then it works. However, when I go back to ghost start it gives me 502s again.

Technical details:

Ghost Version: 2.31.1
Node Version: 10.16.3

ekerstein commented 4 years ago

Update: I rolled back my ghost version to 2.30.2 and it started working again.

andreborud commented 4 years ago

As you got it working again by downgrading I'm not sure if nginx is the problem. On my droplet running CentOS I had to change the max size of any file being uploaded in the base config of nginx located in /etc/nginx/nginx.conf. I don't remember the default size 2 or 5mb.

http {
    ...
    client_max_body_size 50M; <- change this line
    ...
}

label-actions[bot] commented 4 years ago

Hey @ekerstein :wave: We ask that you please do not use GitHub for help or support 😄. We use GitHub solely for bug-tracking and on-going feature development.

Many questions can be answered by reviewing our documentation. If you can't find an answer then our forum is a great place to get community support, plus it helps create a central location for searching problems/solutions.

FYI: Many projects have their own support guidelines and GitHub will highlight them for you as it did here, or the project owners will use issue templates to point you in the right direction. Reading the guidelines or issue templates before opening issues can save you and project maintainers valuable time.

ekerstein commented 4 years ago

@andreborud What do you recommend changing the client_max_body_size to?

andreborud commented 4 years ago

Up to you, depends on how large files you plan to upload. I set it to 50 so I don't need to think about it too much.

ivRodriguezCA commented 4 years ago

Dunno if it was the same problem. But in my case, the issue was that some images were stored in /content/images/{year}/{month}/ but referenced at content/images/size/w{number}/{year}/{month}/ when a GET request to size/w{number}/... came in Ghost would crash and restart. I had to manually copy and move some of them around.

For example:

I had to copy content/images/2019/06/hero-image.jpg into content/images/size/w2000/2019/06/hero-image.jpg for it to be loaded on the post and avoid Ghost crashing.

Edit: Adding this comment because editing /etc/nginx/nginx.conf didn't help with my problem.

andreborud commented 4 years ago

@ekerstein Did you manage to get it working? I have run in to the same issue now. I haven't tried to upload an image for a while, tried last night and I get a 502 error.

andreborud commented 4 years ago

In ghost I get "Something went wrong :(" and when inspecting I see this error: POST https://andreborud.com/ghost/api/canary/admin/images/upload/ 502

But going in to the error logs of ghost there is no 502 errors logged, the only persistent error is the following:

{
    "name": "Log",
    "hostname": "host",
    "pid": 4586,
    "level": 50,
    "req": {
        "meta": {
            "requestId": "1e7a7421-62f8-4ad2-9a32-a24020798489",
            "userId": null
        },
        "url": "/users/me/?include=roles",
        "method": "GET",
        "originalUrl": "/ghost/api/canary/admin/users/me/?include=roles",
        "params": {},
        "headers": {
            "host": "andreborud.com",
            "x-real-ip": "192.145.124.124",
            "x-forwarded-for": "192.145.124.124",
            "x-forwarded-proto": "https",
            "connection": "close",
            "sec-fetch-mode": "cors",
            "dnt": "1",
            "app-pragma": "no-cache",
            "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36",
            "content-type": "application/json; charset=UTF-8",
            "accept": "application/json, text/javascript, */*; q=0.01",
            "x-requested-with": "XMLHttpRequest",
            "x-ghost-version": "2.31",
            "sec-fetch-site": "same-origin",
            "referer": "https://andreborud.com/ghost/",
            "accept-encoding": "gzip, deflate, br",
            "accept-language": "en-GB,en-US;q=0.9,en;q=0.8",
            "cookie": "**REDACTED**"
        },
        "query": {
            "include": "roles"
        }
    },
    "res": {
        "_headers": {
            "x-powered-by": "Express",
            "cache-control": "no-cache, private, no-store, must-revalidate, max-stale=0, post-check=0, pre-check=0",
            "content-type": "application/json; charset=utf-8",
            "content-length": "321",
            "etag": "W/\"141-MpHclrM9KZiRaDmHfy2VdplNIzo\"",
            "vary": "Accept-Encoding"
        },
        "statusCode": 403,
        "responseTime": "9ms"
    },
    "err": {
        "id": "a7a38f30-e8cb-11e9-91b9-f398e47b2304",
        "domain": "https://andreborud.com",
        "code": null,
        "name": "NoPermissionError",
        "statusCode": 403,
        "level": "normal",
        "message": "Authorization failed",
        "context": "\"Unable to determine the authenticated user or integration. Check that cookies are being passed through if using session authentication.\"",
        "stack": "NoPermissionError: Authorization failed\n    at new NoPermissionError (/home/user/andreborud.com/versions/2.31.1/node_modules/ghost-ignition/lib/errors/index.js:114:23)\n    at authorizeAdminApi (/home/user/andreborud.com/versions/2.31.1/core/server/services/auth/authorize.js:76:25)\n    at Layer.handle [as handle_request] (/home/user/andreborud.com/versions/2.31.1/node_modules/express/lib/router/layer.js:95:5)\n    at next (/home/user/andreborud.com/versions/2.31.1/node_modules/express/lib/router/route.js:137:13)\n    at /home/user/andreborud.com/versions/2.31.1/core/server/services/auth/session/middleware.js:119:20\n    at /home/user/andreborud.com/versions/2.31.1/node_modules/express-session/index.js:495:7\n    at Child.SessionModel.findOne.then (/home/user/andreborud.com/versions/2.31.1/core/server/services/auth/session/store.js:24:28)\n    at Child.tryCatcher (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/promise.js:517:31)\n    at Promise._settlePromise (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/promise.js:574:18)\n    at Promise._settlePromise0 (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/promise.js:619:10)\n    at Promise._settlePromises (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/promise.js:699:18)\n    at _drainQueueStep (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/async.js:138:12)\n    at _drainQueue (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/async.js:131:9)\n    at Async._drainQueues (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/async.js:147:5)\n    at Immediate.Async.drainQueues [as _onImmediate] (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/async.js:17:14)"
    },
    "msg": "Authorization failed",
    "time": "2019-10-07T06:28:20.007Z",
    "v": 0
}

The NGINX error logs reports this:

upstream prematurely closed connection while reading response header from upstream, client: 192.145.124.124, server: andreborud.com, request: "POST /ghost/api/canary/admin/images/upload/ HTTP/2.0", upstream: "http://127.0.0.1:2371/ghost/api/canary/admin/images/upload/", host: "andreborud.com", referrer: "https://andreborud.com/ghost/"

I have also tried to temporarily disable SELinux, no difference.

ivRodriguezCA commented 4 years ago

Actually no. Yesterday I was publishing an article and had to ghost stop and ghost run, upload the image, then CTL + C and ghost start. Then manually copy the image from content/images/2019/10 to content/images/size/w2000 for it to work.

andreborud commented 4 years ago

Stopping ghost and then start it with run and then start works for me as well.

ekerstein commented 4 years ago

@andreborud Honestly, I haven't tried against since rolling back to 2.30.2. So I just tested it again now:

Checked my nginx configuration and it was already at 50M. Also, the file I uploaded when I got a 502 error was only 1-2mb. So unless the config file is in another folder, I'm not sure that's the issue.
The nginx config also doesn't explain why rolling back to a previous version would make it work again.

I came across this issue related to the config server.port being 2369 while nginx was on 2368. I realized I had the same issue! So I did ghost config set server.port 2368, restarted, and updated to the latest version. My site loaded fine but when I went to try and access my posts it gave me a 500 error. I did ghost run and it gave me this:

InternalServerError: Cannot read property '0' of undefined
    at new GhostError (/var/www/ghost/versions/2.32.0/core/server/lib/common/errors.js:10:26)
    at _private.prepareError (/var/www/ghost/versions/2.32.0/core/server/web/shared/middlewares/error-handler.js:51:19)
    at Layer.handle_error (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/layer.js:71:5)
    at trim_prefix (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:315:13)
    at /var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:335:12)
    at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:275:10)
    at Layer.handle_error (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/layer.js:67:12)
    at trim_prefix (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:315:13)
    at /var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:335:12)
    at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:275:10)
    at /var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:635:15
    at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:260:14)
    at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/route.js:127:14)
    at apiImpl.then.then.catch (/var/www/ghost/versions/2.32.0/core/server/api/shared/http.js:97:17)
    at tryCatcher (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:725:18)
    at _drainQueueStep (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:93:12)

TypeError: Cannot read property '0' of undefined
    at forEach (/var/www/ghost/versions/2.32.0/node_modules/@tryghost/url-utils/lib/utils/mobiledoc-relative-to-absolute.js:22:43)
    at Array.forEach (<anonymous>)
    at Object.mobiledocRelativeToAbsolute (/var/www/ghost/versions/2.32.0/node_modules/@tryghost/url-utils/lib/utils/mobiledoc-relativ                        e-to-absolute.js:21:39)
    at UrlUtils.mobiledocRelativeToAbsolute (/var/www/ghost/versions/2.32.0/node_modules/@tryghost/url-utils/lib/index.js:387:22)
    at Object.forPost (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/utils/url.js:34:36)
    at Object.mapPost (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/utils/mapper.js:35:9)
    at frame.response.posts.models.data.map.model (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/post                        s.js:15:56)
    at Array.map (<anonymous>)
    at Object.all (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/posts.js:15:36)
    at serializeOptionsShared (/var/www/ghost/versions/2.32.0/core/server/api/shared/serializers/handle.js:107:58)
    at /var/www/ghost/versions/2.32.0/core/server/lib/promise/sequence.js:10:31
    at tryCatcher (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/util.js:16:23)
    at Object.gotValue (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/reduce.js:168:18)
    at Object.gotAccum (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/reduce.js:155:25)
    at Object.tryCatcher (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromiseCtx (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:641:10)
    at _drainQueueStep (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:97:12)
    at _drainQueue (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:15:14)

I rolled back to 2.30.2 again and everything is working. So now I'm not sure if the issue is related to the port 2368 that I changed or something else.

ekerstein commented 4 years ago

FYI - v.2.34.0 fixed my 500 error issue.

However, now my 502 error on image upload (not occuring during ghost run) is back :(

ErisDS commented 4 years ago

I have a feeling this is to do with sharp. I think maybe it doesn't always install properly, and when it doesn't, we're not falling back gracefully like we should.

In the case of a "recommended stack" setup, it should always install. @ekerstein can you look through your versions/xx/node_modules folders and see if sharp is in some or all of them?

We are looking into this, but it's slow work & very hard to track down because we haven't reproduced it yet.

Also can you run ghost doctor?

andreborud commented 4 years ago

Updated to 2.34.0 this morning and the problem persists.

And sharp is installed on this and previous versions.

ErisDS commented 4 years ago

What does ghost doctor say?

andreborud commented 4 years ago

Well I'm running ghost on CentOS so the doctor isn't to excited about that. You can see the output below.

$ ghost doctor
✔ Checking system Node.js version
✔ Checking logged in user
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
System checks failed with message: 'Linux version is not Ubuntu 16 or 18'
Some features of Ghost-CLI may not work without additional configuration.
For local installs we recommend using `ghost install local` instead.
? Continue anyway? Yes
System stack check skipped
ℹ Checking operating system compatibility [skipped]
✔ Checking for a MySQL installation
+ sudo systemctl is-active ghost_andreborud
? Sudo Password [hidden]
Instance is currently running
ℹ Validating config [skipped]
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking content folder ownership
✔ Checking memory availability

However, I have been running ghost for a long time on CentOS without this problem, it popped up recently, not entirely certain what version introduced the bug. And stopping ghost and running it with ghost run instead make the image upload work again.

Is there any other information I can provide for you guys to make things easier?

ekerstein commented 4 years ago

@ErisDS Checked some of my versions folders. The version I kept rolling back to that worked (2.30.2) has sharp version 0.23.0. The latest version of Ghost has 0.23.1. Not sure if that means anything.. but I'm happy to provide more details if I can help.

timskap commented 4 years ago

I've got the same problem and image uploading works only with "ghost run". When I try use "ghost start" i've got error 502.

andreborud commented 4 years ago

This problem still happens in Ghost 3.0.0

shanteshpatil commented 4 years ago

This problem still happens in Ghost 3.0.0

I was really hoping this would be fixed in such a major version update. I am facing the same problem

Ghost CLI Version is 1.12.0

Hosted on a Digital Ocean droplet

ekerstein commented 4 years ago

I tried going into the current folder and downgrading sharp to 0.23.0 just out of curiosity (since that was the version on an earlier working ghost install). It was messy because some of the dependencies got messed up, but I was able to get the server started again. No luck though, same issue persisted.

ekerstein commented 4 years ago

Maybe discovered something? I decided to just add logging extensively throughout the image processing files.

In lib/image/manipulator.js I added logging to everything. Then I uploaded a file to see the normal process vs broken process.

Based on the logging, this is my understanding of what happens when you upload a file. This is using ghost run and works fine.

makeSafe
makeSafe require('sharp')
unsafeProcess
unsafeResizeImage
unsafeProcess return fs.writeFile

So then I tried it in production with ghost start. The server restarted in the makeSafe function at the same point every time. Right here:

const makeSafe = fn => (...args) => {
    common.logging.info('log 1');
    try {
        common.logging.info('log 2');
        require('sharp');
    } 
    ...

Every time it fails, "log 1" is logged, and then it never makes it to "log 2". It restarts immediately after "log 1" like this:

[2019-10-26 17:56:14] INFO log 1
[2019-10-26 17:56:20] INFO Ghost is running in production...
[2019-10-26 17:56:20] INFO Your site is now available on https://example.com/
[2019-10-26 17:56:20] INFO Ctrl+C to shut down
[2019-10-26 17:56:20] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2019-10-26 17:56:20] WARN Tries: 0
[2019-10-26 17:56:20] WARN Retrying...
[2019-10-26 17:56:20] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2019-10-26 17:56:20] WARN Tries: 1
[2019-10-26 17:56:20] WARN Retrying...
[2019-10-26 17:56:20] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2019-10-26 17:56:20] WARN Tries: 2
[2019-10-26 17:56:20] WARN Retrying...
[2019-10-26 17:56:20] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2019-10-26 17:56:20] INFO Ghost boot 4.203s

I assume it was failing on require('sharp'). So I removed that line and it continued on to unsafeProcess and then unsafeResizeImage where it failed again on this line where it's loading sharp again.

const unsafeResizeImage = (originalBuffer, { width, height } = {}) => {
    const sharp = require('sharp');

Seeing as the issue is sharp, I tried reinstalling Ghost, downgrading sharp to 0.23.0, and 0.22.1, but no luck.

1c7 commented 4 years ago

Ghost 3.0

Unable to upload image. version 2.3 work fine. I just update Ghost to 3.0 yesterday and this happen.

I didn't post log because I am not sure which log I should look at. let me know which log I should copy&paste here~

ErisDS commented 4 years ago

@ekerstein Your debug info is very helpful but also baffling!

Is there anything in your journal or syslog at all?

Given that the second log doesn't even get output, that means the require is dying and causing the process to crash without an exception and before the log even gets sent to stdout - there's a try-catch block there on purpose, it's meant to catch any problems with requiring sharp, but clearly it isn't able to.

This points at a problem with the install of sharp's c-libraries.

Could you possibly change directory into /versions/[latest] and try 1) deleting the sharp directory, and then running npm install sharp just to see if there's any output?

I'm sorry to make you jump, but I've not been able to reproduce this issue. It's happening to many people so it's definitely an issue, but without reproduction we don't have a hope in hell at fixing it.

It's also almost certainly a problem with the sharp install process, but we don't even yet understand it enough to report it.

To anyone else - please don't just comment here saying "me too". We know it's a problem.

If you want to help, share your full system information (os, node version, npm version, node install method, ghost-cli version or ghost install method) and if possible ghost & system logs.

ErisDS commented 4 years ago

Oh P.S. @ekerstein can you also share what OS and version are you running on & what version of node and npm you are on? Is it still node 10.16?

ErisDS commented 4 years ago

FWIW I keep trying to reproduce this and I can't! I do however have a potentially useful debug process - it's tricky, but here we go:

Step 1. Modify ghost-cli to be super verbose:

which ghost will tell you roughly where ghost-cli is installed

E.g. on a DO droplet it should be /usr/local/bin/ghost.

We want to edit the file lib/tasks/yarn-install.js. The path to the actual files will be up 2 levels from which ghost, in lib/node_modules/ghost-cli.

So, if your which ghost gives you /usr/local/bin/ghost then you need to edit the file /usr/local/lib/node_modules/ghost-cli/lib/tasks/yarn-install.js.

Change line 65 from:

const observable = yarn(['install', '--no-emoji', '--no-progress'], {

to:

const observable = yarn(['install', '--no-emoji', '--no-progress', '--verbose'], {

Step 2. Rerun update

First figure out what version of Ghost you are currently on with ghost version. Then update to that exact version again, with the force and verbose flags:

e.g. ghost version -> 2.37.0

Then run ghost update v2.37.0 --force --verbose.

This is going to give TONNES of output from yarn, but thankfully the interesting bit is at the end.

You should see some output like:

node-pre-gyp info install unpacking node-v64-linux-x64/node_sqlite3.node
node-pre-gyp info tarball done parsing tarball
[sqlite3] Success: "/var/www/ghost/versions/2.37.0/node_modules/sqlite3/lib/binding/node-v64-linux-x64/node_sqlite3.node" is installed via remote
node-pre-gyp info ok
[16:06:22] → verbose 27.014 info sharp Using cached /home/ghost-mgr/.npm/_libvips/libvips-8.8.1-linux-x64.tar.gz
[16:06:23] → Done in 26.92s.
[16:06:23] Installing dependencies [completed]

See that line in the middle with verbose 27.014 info sharp? That's the bit we're looking for - some output from sharp.

Please follow these steps if you can and share any sharp-related output at the end of the yarn output.

Additional info: The output from sharp comes from this file, in my test case, this line which says it's reusing cached sharp.

I did these steps, then I deleted the path that it mentioned:

rm /home/ghost-mgr/.npm/_libvips/libvips-8.8.1-linux-x64.tar.gz

Re-ran the force upgrade, and this time I got a different line:

verbose 28.056 info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.8.1/libvips-8.8.1-linux-x64.tar.gz

Image uploads continue to work fine.

andreborud commented 4 years ago

@ErisDS I've tried what you suggested. But I started with the last part, I had 4 versions of the libvips library. Deleted all and made a forced re-install. Got exactly the same output as you.

Then I thought maybe its a problem with the libvips library. So I re-downloaded v8.7.4 manually with wget. Then I modified ghostFolder/current/node_modules/sharp/package.json to use library v8.7.4 instead of v8.8.1 and now it works again.

I'm guessing version 8.8.1 has a problem with CentOS 7.x since its working for you.

What OS is @everyoneElse using?

timskap commented 4 years ago

@andreborud I'm using Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-66-generic x86_64) I Installed this version from Digital Ocean Ghost-CLI version: 1.12.0 Node: v10.16.3

andreborud commented 4 years ago

So its probably not the OS then. I was using Node v10.16.3 until today. But just updating to Node v12.13.0 didn't solve it. But I am on DigitalOcean as well.

timskap commented 4 years ago

@andreborud Maybe DigitalOcean make some trouble?

andreborud commented 4 years ago

I don't see how. But could you try to downgrade to libvips and see if it works for you as well?

timskap commented 4 years ago

@andreborud I don't tried to downgrade but found intresting issue with VipsJpeg 8.8.1: https://github.com/libvips/libvips/issues/1372

mergederg commented 4 years ago

Don't mean to just add another +1 here, but I've got two blogs running affected by this. Both are on a (somewhat dated, but kept up to date via regular apt-get upgrades) ghost DO droplet.

Ghost-CLI version: 1.12.0
Ghost version: 3.0.0 (at /var/www/ghost)
Linux personal-blog 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux`

I've run through these instructions from above and oddly enough, the 502s persist despite getting nearly identical output. Initially, it had been using the cached version from ghost-mgr's npm cache, then after deleting it, it downloaded a new one from https://github.com/lovell/sharp-libvips/releases/download/v8.8.1/libvips-8.8.1-linux-x64.tar.gz.

Happy to try and share the results of any other ideas anyone has. :)

Edit: Downgrading libvips to 8.7.4 in sharp's package.json seemed to do the trick, as here. Guess I'll just keep doing that with every release for a while. 🤷‍♂

johnski001 commented 4 years ago

@ErisDS I've tried what you suggested. But I started with the last part, I had 4 versions of the libvips library. Deleted all and made a forced re-install. Got exactly the same output as you.

Then I thought maybe its a problem with the libvips library. So I re-downloaded v8.7.4 manually with wget. Then I modified ghostFolder/current/node_modules/sharp/package.json to use library v8.7.4 instead of v8.8.1 and now it works again.

I'm guessing version 8.8.1 has a problem with CentOS 7.x since its working for you.

What OS is @everyoneElse using?

I fixed this problem as described by @andreborud and it works now with ghost 3.0.2 on DIgital Ocean. Droplet runs on the ubuntu Ghost package from their market.

timskap commented 4 years ago

@ErisDS I've tried what you suggested. But I started with the last part, I had 4 versions of the libvips library. Deleted all and made a forced re-install. Got exactly the same output as you. Then I thought maybe its a problem with the libvips library. So I re-downloaded v8.7.4 manually with wget. Then I modified ghostFolder/current/node_modules/sharp/package.json to use library v8.7.4 instead of v8.8.1 and now it works again. I'm guessing version 8.8.1 has a problem with CentOS 7.x since its working for you. What OS is @everyoneElse using?

I fixed this problem as described by @andreborud and it works now with ghost 3.0.2 on DIgital Ocean. Droplet runs on the ubuntu Ghost package from their market.

Can you explain step by step how you downgrade package?

johnski001 commented 4 years ago

Download the library https://github.com/lovell/sharp-libvips/releases/download/v8.7.4/libvips-8.7.4-linux-x64.tar.gz
Place the tarball in the /home/ghost-mgr/.npm/_libvips/ folder (you can download it in here through wget)
Edit ghostFolder/current/node_modules/sharp/package.json file, change the current version to 8.7.4
Restart blog, I don't know if it's necessary but I did it

lovell commented 4 years ago

Hi, I'm the sharp maintainer, is this problem only occurring with Digital Ocean VMs / "droplets"? Please can someone provide full details of which type/size/location this occurs with, including the output of cat /proc/cpuinfo.

If anyone is able to use gdb node to get a backtrace of the crash that would be amazing. Here's an example of what to type (run this from the directory that contains node_modules):

$ gdb node
GNU gdb (Ubuntu 8.2.91.20190405-0ubuntu3) 8.2.91.20190405-git
Copyright (C) 2019 Free Software Foundation, Inc.
...

(gdb) run -e "require('sharp')"

Starting program: /usr/bin/node -e "require('sharp')"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
...
[Detaching after fork from child process 15193]
[Inferior 1 (process 15183) exited normally]

In this example there is no crash, but if there was you can type backtrace to get full details.

mergederg commented 4 years ago

I'll answer what I can.

Most of us seem to be running off the Ghost DO droplet (it's sort of a premade Ubuntu 18.04 image with mySQL, Nginx, et cetera), but seems like at least one CentOS box has the problem.

Here's my /proc/cpuinfo. That instance has 2GB of memory.

Will have a go at bumping libvips back to 8.8.1 and attaching gdb to node.

colroyce commented 4 years ago

@lovell Hi! I’m also having this issue on DO Droplet (Ubuntu 18.04.3 LTS / 1 GB memory). And here’s my /proc/cpuinfo. Downgrading libvips library worked for me and solved the problem.

lovell commented 4 years ago

I created a droplet using https://marketplace.digitalocean.com/apps/ghost and all the latest versions seemed to be installed without error. I did notice a mixed use of root and ghost-mgr users, which I think is based on the way the Digital Ocean bootstrap script works (are these open source?).

In addition to a backtrace of the crash, please can someone provide a complete listing of the sharp directory from a machine on which the crash is occurring.

$ ls -alR /var/www/ghost/versions/3.0.2/node_modules/sharp

andreborud commented 4 years ago

@lovell thanks for trying to solve this problem. The output from cpuinfo, gdb node and ls -alR ended up being quite a lot. Instead of adding a long list here I made a .txt file you can view here: https://www.dropbox.com/s/g45li4af40i4jsf/sharp.txt?dl=0.

I'm on a 2 CPUs, 4GB RAM droplet running Centos 7.6.

Let me know if I can provide anything else.

lovell commented 4 years ago

Thank you, the output from gdb suggests that requiring sharp didn't crash. Does running the following commands crash or produce a list of versions?

$ cd /var/www/ghost/versions/3.0.2
$ node -e "console.log(require('sharp').versions)"

andreborud commented 4 years ago

Nope it does not:

$ node -e "console.log(require('sharp').versions)"
{
  cairo: '1.17.2',
  croco: '0.6.13',
  exif: '0.6.21',
  expat: '2.2.7',
  ffi: '3.2.1',
  fontconfig: '2.13.91',
  freetype: '2.10.1',
  fribidi: '1.0.5',
  gdkpixbuf: '2.36.12',
  gettext: '0.20.1',
  gif: '5.1.4',
  glib: '2.56.4',
  gsf: '1.14.46',
  harfbuzz: '2.5.3',
  jpeg: '2.0.2',
  lcms: '2.9',
  orc: '0.4.28',
  pango: '1.42.4',
  pixman: '0.38.4',
  png: '1.6.37',
  svg: '2.45.5',
  tiff: '4.0.10',
  vips: '8.8.1',
  webp: '1.0.3',
  xml: '2.9.9',
  zlib: '1.2.11'
}

lovell commented 4 years ago

Thanks, this suggests it's not sharp alone that's causing the problem but perhaps a conflict with another native module or shared library.

To help narrow down native module conflicts, what do the following return when run in the /var/www/ghost/current directory (when it is in a state that crashes)?

$ npm ls nan
$ npm ls -g nan

To help narrow down shared library conflicts, please can someone add the LD_DEBUG=files environment variable and provide the full log of ghost start until the crash.

vweevers commented 4 years ago

@lovell If it helps, here's a list of native addons in the dependencies of ghost master: iltorb@2.4.4, sharp@0.23.1, sqlite3@4.1.0, dtrace-provider@0.8.8. I used about-native-modules as a source of known native addons (includes more than nan dependents) (but in this case all 4 depend on nan@2.14.0). Testing these addons against each other (injecting one addon into the process of another, running unit tests) didn't reveal any obvious conflicts (on Ubuntu 18.04, excluding dtrace-provider).

As another lead, for CentOS specifically, your prebuild might be built against a glibc that's too new? In leveldown we solved this by compiling the prebuilds on CentOS (instead of Travis' default Ubuntu).

vweevers commented 4 years ago

@lovell Another lead is that some versions of sharp can't be loaded side by side into the same process. Probably doesn't apply to this ghost issue (because there should only be one version of sharp) but who knows.

	`0.23.2`	`0.23.1`	`0.23.0`	`0.22.1`
`0.23.2`	-	OK	OK	ERR
`0.23.1`	OK	-	OK	ERR
`0.23.0`	OK	OK	-	ERR
`0.22.1`	ERR	ERR	ERR	-

andreborud commented 4 years ago

@lovell do you need any more info from us?

lovell commented 4 years ago

@andreborud Are you able to provide any of the details requested in https://github.com/TryGhost/Ghost/issues/11129#issuecomment-549082570 ?

andreborud commented 4 years ago

No problem

$ npm ls nan
ghost@3.0.2 /home/architect/andreborud.com/versions/3.0.2
├─┬ ghost-ignition@3.1.0
│ └─┬ bunyan@1.8.12
│   └─┬ dtrace-provider@0.8.8
│     └── nan@2.14.0  deduped
├─┬ sharp@0.23.1
│ └── nan@2.14.0 
└─┬ sqlite3@4.1.0
  └── nan@2.14.0  deduped

and

$ npm ls -g nan
/usr/local/lib
└── (empty)

andreborud commented 4 years ago

Not sure how to do this part:

To help narrow down shared library conflicts, please can someone add the LD_DEBUG=files environment variable and provide the full log of ghost start until the crash.

TryGhost / Ghost