Closed ekerstein closed 4 years ago
Update: I rolled back my ghost version to 2.30.2 and it started working again.
As you got it working again by downgrading I'm not sure if nginx is the problem. On my droplet running CentOS I had to change the max size of any file being uploaded in the base config of nginx located in /etc/nginx/nginx.conf
. I don't remember the default size 2 or 5mb.
http {
...
client_max_body_size 50M; <- change this line
...
}
Hey @ekerstein :wave: We ask that you please do not use GitHub for help or support π. We use GitHub solely for bug-tracking and on-going feature development.
Many questions can be answered by reviewing our documentation. If you can't find an answer then our forum is a great place to get community support, plus it helps create a central location for searching problems/solutions.
FYI: Many projects have their own support guidelines and GitHub will highlight them for you as it did here, or the project owners will use issue templates to point you in the right direction. Reading the guidelines or issue templates before opening issues can save you and project maintainers valuable time.
@andreborud What do you recommend changing the client_max_body_size to?
Up to you, depends on how large files you plan to upload. I set it to 50 so I don't need to think about it too much.
Dunno if it was the same problem. But in my case, the issue was that some images were stored in /content/images/{year}/{month}/
but referenced at content/images/size/w{number}/{year}/{month}/
when a GET request to size/w{number}/...
came in Ghost would crash and restart. I had to manually copy and move some of them around.
For example:
I had to copy content/images/2019/06/hero-image.jpg
into content/images/size/w2000/2019/06/hero-image.jpg
for it to be loaded on the post and avoid Ghost crashing.
Edit: Adding this comment because editing /etc/nginx/nginx.conf
didn't help with my problem.
@ekerstein Did you manage to get it working? I have run in to the same issue now. I haven't tried to upload an image for a while, tried last night and I get a 502 error.
In ghost I get "Something went wrong :(" and when inspecting I see this error: POST https://andreborud.com/ghost/api/canary/admin/images/upload/ 502
But going in to the error logs of ghost there is no 502 errors logged, the only persistent error is the following:
{
"name": "Log",
"hostname": "host",
"pid": 4586,
"level": 50,
"req": {
"meta": {
"requestId": "1e7a7421-62f8-4ad2-9a32-a24020798489",
"userId": null
},
"url": "/users/me/?include=roles",
"method": "GET",
"originalUrl": "/ghost/api/canary/admin/users/me/?include=roles",
"params": {},
"headers": {
"host": "andreborud.com",
"x-real-ip": "192.145.124.124",
"x-forwarded-for": "192.145.124.124",
"x-forwarded-proto": "https",
"connection": "close",
"sec-fetch-mode": "cors",
"dnt": "1",
"app-pragma": "no-cache",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36",
"content-type": "application/json; charset=UTF-8",
"accept": "application/json, text/javascript, */*; q=0.01",
"x-requested-with": "XMLHttpRequest",
"x-ghost-version": "2.31",
"sec-fetch-site": "same-origin",
"referer": "https://andreborud.com/ghost/",
"accept-encoding": "gzip, deflate, br",
"accept-language": "en-GB,en-US;q=0.9,en;q=0.8",
"cookie": "**REDACTED**"
},
"query": {
"include": "roles"
}
},
"res": {
"_headers": {
"x-powered-by": "Express",
"cache-control": "no-cache, private, no-store, must-revalidate, max-stale=0, post-check=0, pre-check=0",
"content-type": "application/json; charset=utf-8",
"content-length": "321",
"etag": "W/\"141-MpHclrM9KZiRaDmHfy2VdplNIzo\"",
"vary": "Accept-Encoding"
},
"statusCode": 403,
"responseTime": "9ms"
},
"err": {
"id": "a7a38f30-e8cb-11e9-91b9-f398e47b2304",
"domain": "https://andreborud.com",
"code": null,
"name": "NoPermissionError",
"statusCode": 403,
"level": "normal",
"message": "Authorization failed",
"context": "\"Unable to determine the authenticated user or integration. Check that cookies are being passed through if using session authentication.\"",
"stack": "NoPermissionError: Authorization failed\n at new NoPermissionError (/home/user/andreborud.com/versions/2.31.1/node_modules/ghost-ignition/lib/errors/index.js:114:23)\n at authorizeAdminApi (/home/user/andreborud.com/versions/2.31.1/core/server/services/auth/authorize.js:76:25)\n at Layer.handle [as handle_request] (/home/user/andreborud.com/versions/2.31.1/node_modules/express/lib/router/layer.js:95:5)\n at next (/home/user/andreborud.com/versions/2.31.1/node_modules/express/lib/router/route.js:137:13)\n at /home/user/andreborud.com/versions/2.31.1/core/server/services/auth/session/middleware.js:119:20\n at /home/user/andreborud.com/versions/2.31.1/node_modules/express-session/index.js:495:7\n at Child.SessionModel.findOne.then (/home/user/andreborud.com/versions/2.31.1/core/server/services/auth/session/store.js:24:28)\n at Child.tryCatcher (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/util.js:16:23)\n at Promise._settlePromiseFromHandler (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/promise.js:517:31)\n at Promise._settlePromise (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/promise.js:574:18)\n at Promise._settlePromise0 (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/promise.js:619:10)\n at Promise._settlePromises (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/promise.js:699:18)\n at _drainQueueStep (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/async.js:138:12)\n at _drainQueue (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/async.js:131:9)\n at Async._drainQueues (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/async.js:147:5)\n at Immediate.Async.drainQueues [as _onImmediate] (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/async.js:17:14)"
},
"msg": "Authorization failed",
"time": "2019-10-07T06:28:20.007Z",
"v": 0
}
The NGINX error logs reports this:
upstream prematurely closed connection while reading response header from upstream, client: 192.145.124.124, server: andreborud.com, request: "POST /ghost/api/canary/admin/images/upload/ HTTP/2.0", upstream: "http://127.0.0.1:2371/ghost/api/canary/admin/images/upload/", host: "andreborud.com", referrer: "https://andreborud.com/ghost/"
I have also tried to temporarily disable SELinux, no difference.
Actually no. Yesterday I was publishing an article and had to ghost stop
and ghost run
, upload the image, then CTL + C
and ghost start
. Then manually copy the image from content/images/2019/10
to content/images/size/w2000
for it to work.
Stopping ghost and then start it with run and then start works for me as well.
@andreborud Honestly, I haven't tried against since rolling back to 2.30.2. So I just tested it again now:
The nginx config also doesn't explain why rolling back to a previous version would make it work again.
I came across this issue related to the config server.port being 2369 while nginx was on 2368. I realized I had the same issue! So I did ghost config set server.port 2368
, restarted, and updated to the latest version. My site loaded fine but when I went to try and access my posts it gave me a 500 error. I did ghost run
and it gave me this:
InternalServerError: Cannot read property '0' of undefined
at new GhostError (/var/www/ghost/versions/2.32.0/core/server/lib/common/errors.js:10:26)
at _private.prepareError (/var/www/ghost/versions/2.32.0/core/server/web/shared/middlewares/error-handler.js:51:19)
at Layer.handle_error (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/layer.js:71:5)
at trim_prefix (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:315:13)
at /var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:284:7
at Function.process_params (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:335:12)
at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:275:10)
at Layer.handle_error (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/layer.js:67:12)
at trim_prefix (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:315:13)
at /var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:284:7
at Function.process_params (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:335:12)
at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:275:10)
at /var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:635:15
at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:260:14)
at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/route.js:127:14)
at apiImpl.then.then.catch (/var/www/ghost/versions/2.32.0/core/server/api/shared/http.js:97:17)
at tryCatcher (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:547:31)
at Promise._settlePromise (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:604:18)
at Promise._settlePromise0 (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:649:10)
at Promise._settlePromises (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:725:18)
at _drainQueueStep (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:93:12)
TypeError: Cannot read property '0' of undefined
at forEach (/var/www/ghost/versions/2.32.0/node_modules/@tryghost/url-utils/lib/utils/mobiledoc-relative-to-absolute.js:22:43)
at Array.forEach (<anonymous>)
at Object.mobiledocRelativeToAbsolute (/var/www/ghost/versions/2.32.0/node_modules/@tryghost/url-utils/lib/utils/mobiledoc-relativ e-to-absolute.js:21:39)
at UrlUtils.mobiledocRelativeToAbsolute (/var/www/ghost/versions/2.32.0/node_modules/@tryghost/url-utils/lib/index.js:387:22)
at Object.forPost (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/utils/url.js:34:36)
at Object.mapPost (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/utils/mapper.js:35:9)
at frame.response.posts.models.data.map.model (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/post s.js:15:56)
at Array.map (<anonymous>)
at Object.all (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/posts.js:15:36)
at serializeOptionsShared (/var/www/ghost/versions/2.32.0/core/server/api/shared/serializers/handle.js:107:58)
at /var/www/ghost/versions/2.32.0/core/server/lib/promise/sequence.js:10:31
at tryCatcher (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/util.js:16:23)
at Object.gotValue (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/reduce.js:168:18)
at Object.gotAccum (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/reduce.js:155:25)
at Object.tryCatcher (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:547:31)
at Promise._settlePromise (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:604:18)
at Promise._settlePromiseCtx (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:641:10)
at _drainQueueStep (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:97:12)
at _drainQueue (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:86:9)
at Async._drainQueues (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:102:5)
at Immediate.Async.drainQueues [as _onImmediate] (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:15:14)
I rolled back to 2.30.2 again and everything is working. So now I'm not sure if the issue is related to the port 2368 that I changed or something else.
FYI - v.2.34.0 fixed my 500 error issue.
However, now my 502 error on image upload (not occuring during ghost run
) is back :(
I have a feeling this is to do with sharp. I think maybe it doesn't always install properly, and when it doesn't, we're not falling back gracefully like we should.
In the case of a "recommended stack" setup, it should always install. @ekerstein can you look through your versions/xx/node_modules folders and see if sharp is in some or all of them?
We are looking into this, but it's slow work & very hard to track down because we haven't reproduced it yet.
Also can you run ghost doctor
?
Updated to 2.34.0 this morning and the problem persists.
And sharp is installed on this and previous versions.
What does ghost doctor
say?
Well I'm running ghost on CentOS so the doctor isn't to excited about that. You can see the output below.
$ ghost doctor
β Checking system Node.js version
β Checking logged in user
β Ensuring user is not logged in as ghost user
β Checking if logged in user is directory owner
β Checking current folder permissions
System checks failed with message: 'Linux version is not Ubuntu 16 or 18'
Some features of Ghost-CLI may not work without additional configuration.
For local installs we recommend using `ghost install local` instead.
? Continue anyway? Yes
System stack check skipped
βΉ Checking operating system compatibility [skipped]
β Checking for a MySQL installation
+ sudo systemctl is-active ghost_andreborud
? Sudo Password [hidden]
Instance is currently running
βΉ Validating config [skipped]
β Checking folder permissions
β Checking file permissions
β Checking content folder ownership
β Checking memory availability
However, I have been running ghost for a long time on CentOS without this problem, it popped up recently, not entirely certain what version introduced the bug. And stopping ghost and running it with ghost run instead make the image upload work again.
Is there any other information I can provide for you guys to make things easier?
@ErisDS Checked some of my versions folders. The version I kept rolling back to that worked (2.30.2) has sharp version 0.23.0. The latest version of Ghost has 0.23.1. Not sure if that means anything.. but I'm happy to provide more details if I can help.
I've got the same problem and image uploading works only with "ghost run". When I try use "ghost start" i've got error 502.
This problem still happens in Ghost 3.0.0
This problem still happens in Ghost 3.0.0
I was really hoping this would be fixed in such a major version update. I am facing the same problem
Ghost CLI Version is 1.12.0
Hosted on a Digital Ocean droplet
I tried going into the current
folder and downgrading sharp to 0.23.0 just out of curiosity (since that was the version on an earlier working ghost install). It was messy because some of the dependencies got messed up, but I was able to get the server started again. No luck though, same issue persisted.
Maybe discovered something? I decided to just add logging extensively throughout the image processing files.
In lib/image/manipulator.js I added logging to everything. Then I uploaded a file to see the normal process vs broken process.
Based on the logging, this is my understanding of what happens when you upload a file. This is using ghost run
and works fine.
So then I tried it in production with ghost start
. The server restarted in the makeSafe function at the same point every time. Right here:
const makeSafe = fn => (...args) => {
common.logging.info('log 1');
try {
common.logging.info('log 2');
require('sharp');
}
...
Every time it fails, "log 1" is logged, and then it never makes it to "log 2". It restarts immediately after "log 1" like this:
[2019-10-26 17:56:14] INFO log 1
[2019-10-26 17:56:20] INFO Ghost is running in production...
[2019-10-26 17:56:20] INFO Your site is now available on https://example.com/
[2019-10-26 17:56:20] INFO Ctrl+C to shut down
[2019-10-26 17:56:20] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2019-10-26 17:56:20] WARN Tries: 0
[2019-10-26 17:56:20] WARN Retrying...
[2019-10-26 17:56:20] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2019-10-26 17:56:20] WARN Tries: 1
[2019-10-26 17:56:20] WARN Retrying...
[2019-10-26 17:56:20] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2019-10-26 17:56:20] WARN Tries: 2
[2019-10-26 17:56:20] WARN Retrying...
[2019-10-26 17:56:20] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2019-10-26 17:56:20] INFO Ghost boot 4.203s
I assume it was failing on require('sharp')
. So I removed that line and it continued on to unsafeProcess
and then unsafeResizeImage
where it failed again on this line where it's loading sharp again.
const unsafeResizeImage = (originalBuffer, { width, height } = {}) => {
const sharp = require('sharp');
Seeing as the issue is sharp, I tried reinstalling Ghost, downgrading sharp to 0.23.0, and 0.22.1, but no luck.
Unable to upload image. version 2.3 work fine. I just update Ghost to 3.0 yesterday and this happen.
I didn't post log because I am not sure which log I should look at. let me know which log I should copy&paste here~
@ekerstein Your debug info is very helpful but also baffling!
Is there anything in your journal or syslog at all?
Given that the second log doesn't even get output, that means the require is dying and causing the process to crash without an exception and before the log even gets sent to stdout - there's a try-catch block there on purpose, it's meant to catch any problems with requiring sharp, but clearly it isn't able to.
This points at a problem with the install of sharp's c-libraries.
Could you possibly change directory into /versions/[latest]
and try 1) deleting the sharp directory, and then running npm install sharp
just to see if there's any output?
I'm sorry to make you jump, but I've not been able to reproduce this issue. It's happening to many people so it's definitely an issue, but without reproduction we don't have a hope in hell at fixing it.
It's also almost certainly a problem with the sharp install process, but we don't even yet understand it enough to report it.
To anyone else - please don't just comment here saying "me too". We know it's a problem.
If you want to help, share your full system information (os, node version, npm version, node install method, ghost-cli version or ghost install method) and if possible ghost & system logs.
Oh P.S. @ekerstein can you also share what OS and version are you running on & what version of node and npm you are on? Is it still node 10.16?
FWIW I keep trying to reproduce this and I can't! I do however have a potentially useful debug process - it's tricky, but here we go:
which ghost
will tell you roughly where ghost-cli is installed
E.g. on a DO droplet it should be /usr/local/bin/ghost
.
We want to edit the file lib/tasks/yarn-install.js
. The path to the actual files will be up 2 levels from which ghost
, in lib/node_modules/ghost-cli
.
So, if your which ghost
gives you /usr/local/bin/ghost
then you need to edit the file /usr/local/lib/node_modules/ghost-cli/lib/tasks/yarn-install.js
.
Change line 65 from:
const observable = yarn(['install', '--no-emoji', '--no-progress'], {
to:
const observable = yarn(['install', '--no-emoji', '--no-progress', '--verbose'], {
First figure out what version of Ghost you are currently on with ghost version
. Then update to that exact version again, with the force and verbose flags:
e.g. ghost version
-> 2.37.0
Then run ghost update v2.37.0 --force --verbose
.
This is going to give TONNES of output from yarn, but thankfully the interesting bit is at the end.
You should see some output like:
node-pre-gyp info install unpacking node-v64-linux-x64/node_sqlite3.node
node-pre-gyp info tarball done parsing tarball
[sqlite3] Success: "/var/www/ghost/versions/2.37.0/node_modules/sqlite3/lib/binding/node-v64-linux-x64/node_sqlite3.node" is installed via remote
node-pre-gyp info ok
[16:06:22] β verbose 27.014 info sharp Using cached /home/ghost-mgr/.npm/_libvips/libvips-8.8.1-linux-x64.tar.gz
[16:06:23] β Done in 26.92s.
[16:06:23] Installing dependencies [completed]
See that line in the middle with verbose 27.014 info sharp
? That's the bit we're looking for - some output from sharp.
Please follow these steps if you can and share any sharp-related output at the end of the yarn output.
Additional info: The output from sharp comes from this file, in my test case, this line which says it's reusing cached sharp.
I did these steps, then I deleted the path that it mentioned:
rm /home/ghost-mgr/.npm/_libvips/libvips-8.8.1-linux-x64.tar.gz
Re-ran the force upgrade, and this time I got a different line:
verbose 28.056 info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.8.1/libvips-8.8.1-linux-x64.tar.gz
Image uploads continue to work fine.
@ErisDS I've tried what you suggested. But I started with the last part, I had 4 versions of the libvips library. Deleted all and made a forced re-install. Got exactly the same output as you.
Then I thought maybe its a problem with the libvips library. So I re-downloaded v8.7.4
manually with wget. Then I modified ghostFolder/current/node_modules/sharp/package.json
to use library v8.7.4
instead of v8.8.1
and now it works again.
I'm guessing version 8.8.1 has a problem with CentOS 7.x since its working for you.
What OS is @everyoneElse using?
@andreborud I'm using Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-66-generic x86_64) I Installed this version from Digital Ocean Ghost-CLI version: 1.12.0 Node: v10.16.3
So its probably not the OS then. I was using Node v10.16.3 until today. But just updating to Node v12.13.0 didn't solve it. But I am on DigitalOcean as well.
@andreborud Maybe DigitalOcean make some trouble?
I don't see how. But could you try to downgrade to libvips and see if it works for you as well?
@andreborud I don't tried to downgrade but found intresting issue with VipsJpeg 8.8.1: https://github.com/libvips/libvips/issues/1372
Don't mean to just add another +1 here, but I've got two blogs running affected by this. Both are on a (somewhat dated, but kept up to date via regular apt-get upgrades) ghost DO droplet.
Ghost-CLI version: 1.12.0
Ghost version: 3.0.0 (at /var/www/ghost)
Linux personal-blog 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux`
I've run through these instructions from above and oddly enough, the 502s persist despite getting nearly identical output. Initially, it had been using the cached version from ghost-mgr's npm cache, then after deleting it, it downloaded a new one from https://github.com/lovell/sharp-libvips/releases/download/v8.8.1/libvips-8.8.1-linux-x64.tar.gz
.
Happy to try and share the results of any other ideas anyone has. :)
Edit: Downgrading libvips to 8.7.4
in sharp's package.json seemed to do the trick, as here. Guess I'll just keep doing that with every release for a while. π€·ββ
@ErisDS I've tried what you suggested. But I started with the last part, I had 4 versions of the libvips library. Deleted all and made a forced re-install. Got exactly the same output as you.
Then I thought maybe its a problem with the libvips library. So I re-downloaded
v8.7.4
manually with wget. Then I modifiedghostFolder/current/node_modules/sharp/package.json
to use libraryv8.7.4
instead ofv8.8.1
and now it works again.I'm guessing version 8.8.1 has a problem with CentOS 7.x since its working for you.
What OS is @everyoneElse using?
I fixed this problem as described by @andreborud and it works now with ghost 3.0.2 on DIgital Ocean. Droplet runs on the ubuntu Ghost package from their market.
@ErisDS I've tried what you suggested. But I started with the last part, I had 4 versions of the libvips library. Deleted all and made a forced re-install. Got exactly the same output as you. Then I thought maybe its a problem with the libvips library. So I re-downloaded
v8.7.4
manually with wget. Then I modifiedghostFolder/current/node_modules/sharp/package.json
to use libraryv8.7.4
instead ofv8.8.1
and now it works again. I'm guessing version 8.8.1 has a problem with CentOS 7.x since its working for you. What OS is @everyoneElse using?I fixed this problem as described by @andreborud and it works now with ghost 3.0.2 on DIgital Ocean. Droplet runs on the ubuntu Ghost package from their market.
Can you explain step by step how you downgrade package?
Download the library https://github.com/lovell/sharp-libvips/releases/download/v8.7.4/libvips-8.7.4-linux-x64.tar.gz
Place the tarball in the /home/ghost-mgr/.npm/_libvips/ folder (you can download it in here through wget)
Edit ghostFolder/current/node_modules/sharp/package.json file, change the current version to 8.7.4
Restart blog, I don't know if it's necessary but I did it
Hi, I'm the sharp maintainer, is this problem only occurring with Digital Ocean VMs / "droplets"? Please can someone provide full details of which type/size/location this occurs with, including the output of cat /proc/cpuinfo
.
If anyone is able to use gdb node
to get a backtrace of the crash that would be amazing. Here's an example of what to type (run this from the directory that contains node_modules
):
$ gdb node
GNU gdb (Ubuntu 8.2.91.20190405-0ubuntu3) 8.2.91.20190405-git
Copyright (C) 2019 Free Software Foundation, Inc.
...
(gdb) run -e "require('sharp')"
Starting program: /usr/bin/node -e "require('sharp')"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
...
[Detaching after fork from child process 15193]
[Inferior 1 (process 15183) exited normally]
In this example there is no crash, but if there was you can type backtrace
to get full details.
I'll answer what I can.
Most of us seem to be running off the Ghost DO droplet (it's sort of a premade Ubuntu 18.04 image with mySQL, Nginx, et cetera), but seems like at least one CentOS box has the problem.
Here's my /proc/cpuinfo. That instance has 2GB of memory.
Will have a go at bumping libvips back to 8.8.1 and attaching gdb to node.
@lovell Hi! Iβm also having this issue on DO Droplet (Ubuntu 18.04.3 LTS / 1 GB memory). And hereβs my /proc/cpuinfo. Downgrading libvips library worked for me and solved the problem.
I created a droplet using https://marketplace.digitalocean.com/apps/ghost and all the latest versions seemed to be installed without error. I did notice a mixed use of root
and ghost-mgr
users, which I think is based on the way the Digital Ocean bootstrap script works (are these open source?).
In addition to a backtrace of the crash, please can someone provide a complete listing of the sharp directory from a machine on which the crash is occurring.
$ ls -alR /var/www/ghost/versions/3.0.2/node_modules/sharp
@lovell thanks for trying to solve this problem. The output from cpuinfo, gdb node and ls -alR ended up being quite a lot. Instead of adding a long list here I made a .txt file you can view here: https://www.dropbox.com/s/g45li4af40i4jsf/sharp.txt?dl=0.
I'm on a 2 CPUs, 4GB RAM droplet running Centos 7.6.
Let me know if I can provide anything else.
Thank you, the output from gdb
suggests that requiring sharp didn't crash. Does running the following commands crash or produce a list of versions?
$ cd /var/www/ghost/versions/3.0.2
$ node -e "console.log(require('sharp').versions)"
Nope it does not:
$ node -e "console.log(require('sharp').versions)"
{
cairo: '1.17.2',
croco: '0.6.13',
exif: '0.6.21',
expat: '2.2.7',
ffi: '3.2.1',
fontconfig: '2.13.91',
freetype: '2.10.1',
fribidi: '1.0.5',
gdkpixbuf: '2.36.12',
gettext: '0.20.1',
gif: '5.1.4',
glib: '2.56.4',
gsf: '1.14.46',
harfbuzz: '2.5.3',
jpeg: '2.0.2',
lcms: '2.9',
orc: '0.4.28',
pango: '1.42.4',
pixman: '0.38.4',
png: '1.6.37',
svg: '2.45.5',
tiff: '4.0.10',
vips: '8.8.1',
webp: '1.0.3',
xml: '2.9.9',
zlib: '1.2.11'
}
Thanks, this suggests it's not sharp alone that's causing the problem but perhaps a conflict with another native module or shared library.
To help narrow down native module conflicts, what do the following return when run in the /var/www/ghost/current
directory (when it is in a state that crashes)?
$ npm ls nan
$ npm ls -g nan
To help narrow down shared library conflicts, please can someone add the LD_DEBUG=files
environment variable and provide the full log of ghost start
until the crash.
@lovell If it helps, here's a list of native addons in the dependencies of ghost
master: iltorb@2.4.4
, sharp@0.23.1
, sqlite3@4.1.0
, dtrace-provider@0.8.8
. I used about-native-modules
as a source of known native addons (includes more than nan
dependents) (but in this case all 4 depend on nan@2.14.0
). Testing these addons against each other (injecting one addon into the process of another, running unit tests) didn't reveal any obvious conflicts (on Ubuntu 18.04, excluding dtrace-provider
).
As another lead, for CentOS specifically, your prebuild might be built against a glibc that's too new? In leveldown
we solved this by compiling the prebuilds on CentOS (instead of Travis' default Ubuntu).
@lovell Another lead is that some versions of sharp
can't be loaded side by side into the same process. Probably doesn't apply to this ghost
issue (because there should only be one version of sharp
) but who knows.
0.23.2 |
0.23.1 |
0.23.0 |
0.22.1 |
|
---|---|---|---|---|
0.23.2 |
- | OK | OK | ERR |
0.23.1 |
OK | - | OK | ERR |
0.23.0 |
OK | OK | - | ERR |
0.22.1 |
ERR | ERR | ERR | - |
@lovell do you need any more info from us?
@andreborud Are you able to provide any of the details requested in https://github.com/TryGhost/Ghost/issues/11129#issuecomment-549082570 ?
No problem
$ npm ls nan
ghost@3.0.2 /home/architect/andreborud.com/versions/3.0.2
βββ¬ ghost-ignition@3.1.0
β βββ¬ bunyan@1.8.12
β βββ¬ dtrace-provider@0.8.8
β βββ nan@2.14.0 deduped
βββ¬ sharp@0.23.1
β βββ nan@2.14.0
βββ¬ sqlite3@4.1.0
βββ nan@2.14.0 deduped
and
$ npm ls -g nan
/usr/local/lib
βββ (empty)
Not sure how to do this part:
To help narrow down shared library conflicts, please can someone add the LD_DEBUG=files environment variable and provide the full log of ghost start until the crash.
I'm having a similar issue as #3827 and the post on the forum here.
I just updated to 2.31.1 and now whenever I upload an image I get a 502 error. I'm using a DO droplet.
I tried debugging with
ghost run
but then it works. However, when I go back toghost start
it gives me 502s again.Technical details: