PHP scrapers that use composer are all failed - Githubissues

openaustralia / morph

Take the hassle out of web scraping

https://morph.io

GNU Affero General Public License v3.0

461 stars 74 forks source link

PHP scrapers that use composer are all failed #1204

Open LoveMyData opened 5 years ago

LoveMyData commented 5 years ago

It was working until some upgrade performed in the backend in Dec.

Error message

Injecting configuration and compiling...
 [1G       [1G-----> PHP app detected
 [1G-----> Bootstrapping...
 [1G       
 [1G       !     ERROR: Failed to download minimal PHP for bootstrapping.
 [1G       Please try again, or contact support if this problem persists.
 [1G

And maybe relating to this with heroku

Examples of scrapers that use composer and not working: https://morph.io/planningalerts-scrapers/townsville https://morph.io/planningalerts-scrapers/tweed-shire https://morph.io/planningalerts-scrapers/albury

Example of scraper doesn't use composer and working: https://morph.io/planningalerts-scrapers/hurstville

jamezpolley commented 5 years ago

I've been able to reproduce this in my dev environment.

jamezpolley commented 5 years ago

Our buildstep image is based on the herokuish https://devcenter.heroku.com/articles/cedar-14-stack stack and herokuish v0.3.33, which uses v126 of the PHP buildpack, which looks for php-min-7.0.25.tar.gz in https://lang-php.s3.amazonaws.com/, but it's not there.
cedar-14 is deprecated, and reaches EOL in April 2019
- The latest version of herokuish we can use is v0.3.36, which is the last version to support cedar-14. It uses buildpack v132 which looks for 7.3.1 which is also not present.

So at minimum we're going to have to jump to heroku-16 which has quite a different set of packages. I don't know how this will affect things and I don't really have a good way to test this before we roll it out everywhere. I can easily test things on my dev instance, but I'm not going to be able to test everything..

mlandauer commented 5 years ago

@jamezpolley I'm pretty sure it's possible to change the buildpacks without upgrading to heroku-16.

See for example https://github.com/openaustralia/buildstep/blob/master/Dockerfile#L11. That install a new buildpack that isn't there by default. I strongly suspect you can use the same or similar command to upgrade an existing buildpack.

mlandauer commented 5 years ago

However, having said that it will obviously be necessary soon to upgrade to heroku-16 if cedar-14 is near its end-of-life. Maybe it can be done in stages to minimise potential disruption and catch small errors soon?

gobnat commented 4 years ago

Is there a fix to this issue or do no php scrapers work at this time?

jamezpolley commented 4 years ago

Is there a fix to this issue or do no php scrapers work at this time?

It's not well documented, but we've added support for herokuish:18 as a build platform. To use it, add a platform file to your repo like this: https://github.com/planningalerts-scrapers/bankstown/blob/master/platform

gobnat commented 4 years ago

Is there a fix to this issue or do no php scrapers work at this time?

It's not well documented, but we've added support for herokuish:18 as a build platform. To use it, add a platform file to your repo like this: https://github.com/planningalerts-scrapers/bankstown/blob/master/platform

I've tried that fix and it does not resolve the issue. It does change the the error slightly to this (I've left in all the "tofu" characters morph seems to be adding. I'm not sure why they are appearing):

[0m [91m-----> [0m[91mBootstrapping... [0m[91m[1;31m ! ERROR: [0m [91m-----> [0m[91mBootstrapping... [0m[91m[1;31m ! ERROR: [0m[91mFailed to download minimal PHP for bootstrapping! ! ! This is most likely a temporary internal error. If the problem [0m [91m-----> [0m[91mBootstrapping... [0m[91m[1;31m ! ERROR: [0m[91mFailed to download minimal PHP for bootstrapping! ! ! This is most likely a temporary internal error. If the problem [0m[91m ! persists, make sure that you are not running a custom or forked ! version of the Heroku PHP buildpack which may need updating.