search

mendix / docker-mendix-buildpack

Build and Run Mendix in Docker
https://www.mendix.com
Apache License 2.0
79 stars 110 forks source link

Issues with deploying new buildpack to Google Cloud #91

Open macrel opened 4 years ago

macrel commented 4 years ago

Hi,

We have succesfully been using the docker buildpack for about a year now. However, with the latest update it's not working anymore and we cannot deploy to the Google cloud. We had to revert to an older buildpack.

First we tried the latest release here on github, but it's throwing errors when running docker build... , specifically on the compilation file some import was causing troubles. So I checked the current master version and the compilation file had been changed resolving that error. Seems like this latest release is not even stable..?

Anyway, using the master version I was able to do a build so I pushed the image to Google Cloud container registry and restarted our server. It downloaded the newer image, but it couldn't start Mendix. Please see the following errors I retreived from the log.

Please advise how to fix, we would like to update to the latest buildpack but it's broken.

This is our build command: docker build ./ --build-arg BUILD_PATH=/build/ -t latestimage --no-cache

The /build/ folder contains our Mendix mpr and some other folders (userlib, javasource, theme). In the Dockerfile we changed the 'expose nginx port' from 8080 to 443. This has always worked great, but now gives a bunch of errors.

Log:

`A 2020-05-06T19:16:26.143119498Z ERROR: Unable to fixup permissions of directory 'model' with mode 700: [Errno 1] Operation not permitted: '/opt/mendix/build/model', Ignoring.

A 2020-05-06T19:16:26.143246971Z ERROR: Unable to fixup permissions of directory 'web' with mode 755: [Errno 1] Operation not permitted: '/opt/mendix/build/web', Ignoring.

A 2020-05-06T19:16:26.143277288Z ERROR: Unable to fixup permissions of directory 'data' with mode 700: [Errno 1] Operation not permitted: '/opt/mendix/build/data', Ignoring.

A 2020-05-06T19:15:50.774237532Z INFO: Trying to start the MxRuntime...

A 2020-05-06T19:15:55.150378539Z No permission to open socket or another server is already running on localhost:445: Permission denied (Bind failed)

A 2020-05-06T19:15:55.300207479Z ERROR: Java subprocess terminated with errorcode 2

A 2020-05-06T19:15:55.302118234Z ERROR: Starting the JVM process did not succeed...

A 2020-05-06T19:15:55.307230088Z ERROR: Starting app container failed: Traceback (most recent call last):

A 2020-05-06T19:15:55.307260862Z File "/opt/mendix/buildpack/buildpack/start.py", line 301, in

A 2020-05-06T19:15:55.307275359Z runtime.run(m2ee)

A 2020-05-06T19:15:55.307287413Z File "/opt/mendix/buildpack/buildpack/runtime.py", line 458, in run

A 2020-05-06T19:15:55.307294897Z complete_start_procedure_safe_to_use_for_restart(m2ee)

A 2020-05-06T19:15:55.307301007Z File "/opt/mendix/buildpack/buildpack/runtime.py", line 376, in complete_start_procedure_safe_to_use_for_restart

A 2020-05-06T19:15:55.307307216Z start_app(m2ee)

A 2020-05-06T19:15:55.307312833Z File "/opt/mendix/buildpack/buildpack/runtime.py", line 385, in start_app

A 2020-05-06T19:15:55.307318923Z if not m2ee.send_runtime_config():

A 2020-05-06T19:15:55.307325421Z File "/opt/mendix/buildpack/lib/m2ee/core.py", line 269, in send_runtime_config

A 2020-05-06T19:15:55.307334399Z m2eeresponse = self.client.update_configuration(config)

A 2020-05-06T19:15:55.307340735Z File "/opt/mendix/buildpack/lib/m2ee/client.py", line 136, in update_configuration

A 2020-05-06T19:15:55.307346784Z return self.request("update_configuration", params)

A 2020-05-06T19:15:55.307353152Z File "/opt/mendix/buildpack/lib/m2ee/client.py", line 50, in request

A 2020-05-06T19:15:55.307363706Z headers=self._headers)

A 2020-05-06T19:15:55.307375934Z File "/opt/mendix/buildpack/lib/python3.6/site-packages/httplib2/init.py", line 1322, in request

A 2020-05-06T19:15:55.307384828Z (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)

A 2020-05-06T19:15:55.307391016Z File "/opt/mendix/buildpack/lib/python3.6/site-packages/httplib2/init.py", line 1072, in _request

A 2020-05-06T19:15:55.307397187Z (response, content) = self._conn_request(conn, request_uri, method, body, headers)

A 2020-05-06T19:15:55.307403777Z File "/opt/mendix/buildpack/lib/python3.6/site-packages/httplib2/init.py", line 995, in _conn_request

A 2020-05-06T19:15:55.307410065Z conn.connect()

A 2020-05-06T19:15:55.307416152Z File "/usr/lib/python3.6/http/client.py", line 948, in connect

A 2020-05-06T19:15:55.307422178Z (self.host,self.port), self.timeout, self.source_address)

A 2020-05-06T19:15:55.307427791Z File "/usr/lib/python3.6/socket.py", line 724, in create_connection

A 2020-05-06T19:15:55.307434019Z raise err

A 2020-05-06T19:15:55.307439757Z File "/usr/lib/python3.6/socket.py", line 713, in create_connection

A 2020-05-06T19:15:55.307446076Z sock.connect(sa)

A 2020-05-06T19:15:55.307475370Z ConnectionRefusedError: [Errno 111] Connection refused

A 2020-05-06T19:15:55.307483770Z

A 2020-05-06T19:15:55.312344863Z INFO: stopping app... `

cstevandy commented 4 years ago

Which docker mendix buildpack and cf-buildpack version did you use?

Can you check your port 445? A 2020-05-06T19:15:55.150378539Z No permission to open socket or another server is already running on localhost:445: Permission denied (Bind failed)

macrel commented 4 years ago

I downloaded the latest docker buildpack (master) and it's using CF buildpack master as well.

Since it's on our live server I cannot check right now what's on port 445, but I can check later on a test server.

However, I think it's not something running on port 445. I think it's a pemissions issue: "No permission to open socket OR ..."

Looking at the rest of the errors it's more likely a permission issue, since I also got errors like:

"`A 2020-05-06T19:16:26.143119498Z ERROR: Unable to fixup permissions of directory 'model' with mode 700: [Errno 1] Operation not permitted: '/opt/mendix/build/model', Ignoring."

I see a lot of changes in the Dockerfile regarding permissions, USER, chmod, etc. So my guess is it has something to do with that.

cstevandy commented 4 years ago

I tried to build just now, it was fine in my machine. I was thinking about the port because you mentioned you exposed 443 but the logs mentioned about port 445. Which OS do you use in google cloud?

macrel commented 4 years ago

Ok.. I use Linux Compute Engine VM with image "cos-stable-72-11316-136-0"

You do not see any errors like mine about unable to fixing permissions?

cstevandy commented 4 years ago

I didn't see any problem when I built and deployed to pivotal

macrel commented 4 years ago

Ok, thanks for your help so far. I will try with a clean deployment on a new server in Google Cloud tonight and let you know.

macrel commented 4 years ago

Hi,

I tested it with a completely blank, clean app on Google cloud and I have the same issue. Here's what I did:

  1. Downloaded Docker Buildpack master
  2. Changed expose 8080 to expose 443 in the Dockerfile
  3. Created a folder 'Build' within the extracted docker buildpack directory
  4. Placed a clean empty Mendix project in the build directory (javascource, project.mpr, userlib, widgets)
  5. Then I run "docker build ./ --build-arg BUILD_PATH=/build/ -t dockertest --no-cache"
  6. I pushed the build image to Google cloud
  7. Spun up a new VM server in Google Cloud (VM). Selected 'Load containerized image' and added the image I just pushed to Google Cloud
  8. Start the VM server
  9. Same errors

So obviously something has changed. It has been working with exactly these steps for over 2 years for me.

You can easily reproduce it with above steps and deploy to Google Cloud. Google Cloud offers 500 $ free credits so you can do plenty of free testing.

Please advise!

I will for now revert back to an older version of the buildpack but of course we need to get this working so we stay up-to-date.

Thanks in advance!

macrel commented 4 years ago

Ps. when I leave the port to 8080 I do not have this error. I then still get a bunch of permissions errors (see below). Not sure if I need to be worried about those?

Anyway, we changed the expose port to 443 to match nginx configuration since nginx.conf has listen 443 ssl. I might be able to chage this to listen 8080 ssl, not sure. Any advice on this?

A 2020-05-10T08:37:12.153921028Z ERROR: Unable to fixup permissions of directory 'model' with mode 700: [Errno 1] Operation not permitted: '/opt/mendix/build/model', Ignoring.

A 2020-05-10T08:37:12.154715679Z ERROR: Unable to fixup permissions of directory 'web' with mode 755: [Errno 1] Operation not permitted: '/opt/mendix/build/web', Ignoring.

A 2020-05-10T08:37:12.155358355Z ERROR: Unable to fixup permissions of directory 'data' with mode 700: [Errno 1] Operation not permitted: '/opt/mendix/build/data', Ignoring.

[EDIT] changing nginx to listen 8080 ssl is not an option, because that would mean the user has to go to https://ourdomain.com:8080 instead of https://ourdomain.com (since default port for https in browser is 443). Maybe I can proxy mendix 8080 to nginx 443. Still strange the log is complaining about port 445

[EDIT] Oh, I see why port 445 is used. In cloudfoundry buildpack it does the nginx port +2 = admin port

macrel commented 4 years ago

Hi,

I have done some more testing.

With the latest Docker buildpack it is impossible to deploy on the Google Cloud. I tried everything. In my opinion it has something todo with the permissions.

I just verified the following:

  1. Downloaded Docker Buildpack of 3 weeks ago
  2. Changed docker buildpack 'Dockerfile' to https://github.com/mendix/cf-mendix-buildpack/archive/v4.4.1.tar.gz instead of master.tar.gz
  3. Left admin and nginx port to 8080
  4. Deployed to Google Cloud
  5. Everything is working fine and I can see the Mendix app when I go to the server's IP

Then I tried this:

  1. Downloaded latest docker buildpack (master)
  2. Not changed a single thing, just added the project to /build/ folder and build the image
  3. Deployed to Google Cloud
  4. Permission errors on startup and not working at all. The log does say Mendix runtime is started but I cannot view the Mendix app

Please help!

cstevandy commented 4 years ago

Hi macrel,

I see in GCP you need root access in order to access privileged port (below 1024): https://stackoverflow.com/questions/53606351/gcp-compute-engine-cannot-listen-on-port-80

The older docker-mendix-buildpack was using root user (v1.x.x), did you use (v.1.x.x) when it was working?

macrel commented 4 years ago

Hi,

Thanks for your help. I have just run a test and I get it working with this setup:

So probably the issue here is not the latest CF buidpack update but the upgrade from 1.x to 2.x of the Docker Buildpack.

Any ideas how I can fix this so I can run the latest docker buildpack on GCP? I would love to stay up-to-date with the buildpack.

macrel commented 4 years ago

Btw, the default buildpack now uses port 8080 right, that's above 1024 so I don't see why that is also giving problems? I haven't even changed it to 443 yet in my tests.

cstevandy commented 4 years ago

Hi macrel,

For now the latest version you need to use the combination of Docker buildpack v2.4.0 and CF buildpack v4.4.2. If you really want to use the latest CF buildpack, then you need to use docker buildpack master as for now.

To fix your issue, either change your port to > 1024 (it's a good practice not to use a privileged port for your application) or you allow the app to access your privileged port I don't know by heart how to do that but there are some answers in internet such as https://stackoverflow.com/questions/413807/is-there-a-way-for-non-root-processes-to-bind-to-privileged-ports-on-linux

About your 8080 port failed, probably caused by the GCP firewall. You need to expose the port to the internet, I don't have many experiences in GCP user so I don't know how to expose the port you can check their documentation. You can at least test your app inside your VM (via SSH): curl http://localhost:8080

macrel commented 4 years ago

Hi,

Thanks for this. This got me a little further and I managed to get something running on port 1024. However, now I need to go to http://my,ip.addres:1024 in order to see the app.

Do you have any idea on how to make it so that I can browse to https://my.ip.address and it will show the app? The browser will map this url to port :443 because of the HTTPS so somehow I must map that port to 1024.

Previously since I had configured the dockerfile to use port 443 this would automatically map correctly to the Mendix runtime.

zlogic commented 4 years ago

Hi @macrel,

Could you provide some details on how you're deploying the app into Google Cloud:

If the Docker container is exposed directly to the internet, and there's no possibility to configure port forwarding or set up a reverse proxy, the only option to use port 80 (or 443) is to run the Mendix Docker image as root:

docker run -p 443:443 -u 0 -e PORT=443 ...

Alternatively, the USER 1001 can be removed from the Dockerfile, so that the image is built to run as root, as it was done in older versions of the Docker Buildpack.

macrel commented 4 years ago

Hi zlogic,

Thanks for your response.

Right now I am pushing the docker build image to Google Container Registry. In the virtual machine I have selected 'containerized image' and I refer to the latest image. Then I have selected 'https' which allows port 443 on the container. Also I overwrite the nginx.conf file to include our SSL certificate and force SSL. Since I changed the docker buildpack port to 443 this is working without further use of proxies or load balancers.

So, just removing the USER 1001 from the dockerfile would allow me to use the latest buildpack as well? Are there any downsides of doing this?

Otherwise, I think configuring a load balancer in Google Cloud in front of the virtual machine might be a way to go around this, however I don't know exactly how to set this up. Is there any documentation on this or do you have tips? I checked the Google documentation but it's quite general and not specifically targeting containerized VMs.

zlogic commented 4 years ago

Hi @macrel,

Removing USER 1001 makes the container run as the root user by default and allows to use ports <1024. But this is discouraged, as any vulnerability might give an attacker complete control over the container (and possibly other processes on the system) https://engineering.bitnami.com/articles/why-non-root-containers-are-important-for-security.html

Indeed, it's best to use a load balancer in Google Cloud. I cannot provide specific instructions, but sounds like an External HTTPS load balancer might work (https://cloud.google.com/load-balancing/docs/choosing-load-balancer#summary-of-google-cloud-load-balancers)

wedomendixapps commented 3 months ago

Hi @macrel .

Have you been able to solve this issue?

What was the root cause? I'm having exact same issue with AWS.

Best regards