Unable to start app with staticfile_buildpack v1.5.3

[cdromas]

What version of Cloud Foundry and CF CLI are you using? (i.e. What is the output of running cf curl /v2/info && cf version? { "name": "Pivotal Application Service", "build": "2.6.14-build.14", "support": "https://support.pivotal.io", "version": 0, "description": "https://docs.pivotal.io/pivotalcf/2-6/pcf-release-notes/runtime-rn.html", "authorization_endpoint": "https://login.sys.stage-eu.kamereon.io", "token_endpoint": "https://uaa.sys.stage-eu.kamereon.io", "min_cli_version": "6.23.0", "min_recommended_cli_version": "6.23.0", "app_ssh_endpoint": "ssh.sys.stage-eu.kamereon.io:2222", "app_ssh_host_key_fingerprint": "b6:92:ed:2e:31:06:3b:a1:5a:69:41:3e:07:09:3a:f3", "app_ssh_oauth_client": "ssh-proxy", "doppler_logging_endpoint": "wss://doppler.sys.stage-eu.kamereon.io:443", "api_version": "2.135.0", "osbapi_version": "2.14", "routing_endpoint": "https://api.sys.stage-eu.kamereon.io/routing", "user": "6a75719f-cf11-47e2-aea7-e0b590dd4bbe" } cf version 6.42.0+0cba12168.2019-01-10

What version of the buildpack you are using? v1.5.3

If you were attempting to accomplish a task, what was it you were attempting to do? Unable to start the application

What did you expect to happen? Start the application

What was the actual behavior? Since last update of our PCF configurations, we are unable to deploy some of our projects with staticfile buildpack v1.5.3.

This failure seams to be introduced by the creation of an incorrect nginx.conf file as we can see in logs bellow:

2020-01-24T11:39:54.893+01:00 [CELL/0] [OUT] Starting health monitoring of container
2020-01-24T11:39:55.875+01:00 [APP/PROC/WEB/0] [ERR] + /home/vcap/app/start_logging.sh
2020-01-24T11:39:55.878+01:00 [APP/PROC/WEB/0] [ERR] + nginx -p /home/vcap/app/nginx -c /home/vcap/app/nginx/conf/nginx.conf
2020-01-24T11:39:57.045+01:00 [APP/PROC/WEB/0] [ERR] Killed
2020-01-24T11:40:56.642+01:00 [HEALTH/0] [ERR] Failed to make TCP connection to port 8080: connection refused (out of memory)
2020-01-24T11:40:56.642+01:00 [CELL/0] [ERR] Timed out after 1m0s: health check never passed.

Running the task "nginx -p /home/vcap/app/nginx -c /home/vcap/app/nginx/conf/nginx.conf " results to:

2020-01-24T11:45:57.973+01:00 [CELL/0] [OUT] Cell e38d7f2b-006d-4a7c-a6b7-ecf199cc24b8 creating container for instance eab085a3-1d60-4237-a2db-83d3665cc65d
2020-01-24T11:45:59.393+01:00 [CELL/0] [OUT] Cell e38d7f2b-006d-4a7c-a6b7-ecf199cc24b8 successfully created container for instance eab085a3-1d60-4237-a2db-83d3665cc65d
2020-01-24T11:46:02.752+01:00 [APP/TASK/238ec363/0] [ERR] 2020/01/24 11:46:02 [emerg] 15#0: invalid number of arguments in "listen" directive in /home/vcap/app/nginx/conf/nginx.conf:33
2020-01-24T11:46:02.901+01:00 [APP/TASK/238ec363/0] [OUT] Exit status 1

Running the command "cat /home/vcap/app/nginx/conf/nginx.conf" results to:

worker_processes 1;
daemon off;
error_log /home/vcap/app/nginx/logs/error.log;
events { worker_connections 1024; }
http {
  charset utf-8;
  log_format cloudfoundry '$http_x_forwarded_for - $http_referer - [$time_local] "$request" $status $body_bytes_sent';
  access_log /home/vcap/app/nginx/logs/access.log cloudfoundry;
  default_type application/octet-stream;
  include mime.types;
  sendfile on;
  gzip on;
  gzip_disable "msie6";
  gzip_comp_level 6;
  gzip_min_length 1100;
  gzip_buffers 16 8k;
  gzip_proxied any;
  gunzip on;
  gzip_static always;
  gzip_types text/plain text/css text/js text/xml text/javascript application/javascript application/x-javascript application/json application/xml application/xml+rss;
  gzip_vary on;
  tcp_nopush on;
  keepalive_timeout 30;
  port_in_redirect off; # Ensure that redirects don't include the internal container PORT -
  server_tokens off;

  server {
    listen ;
    server_name localhost;
    root /home/vcap/app/public;
    set $updated_host $host;
    if ($http_x_forwarded_host != "") {
      set $updated_host $http_x_forwarded_host;
    }
    if ($http_x_forwarded_proto != "https") {
      return 301 https://$updated_host$request_uri;
    }

    location / {

      if (!-e $request_filename) {
        rewrite ^(.*)$ / break;
      }

      index index.html index.htm Default.htm;
      add_header Strict-Transport-Security "max-age=31536000";
      include includes/*.conf;
    }

    location ~ /\. {
      deny all;
      return 404;
    }
  }
}

We can see in this last log that the listen line is incorrect (no port specified).

Please confirm where necessary:

• [X] I have included a log output
• [X] My log includes an error message
• [ ] I have included steps for reproduction

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/170892559

The labels on this github issue will be updated when the story is started.

[ryanmoran]

@cdromas The listen directive should be populated with a port value from the $PORT environment variable. What is the value of the $PORT environment variable inside your application container?

[cdromas]

@ryanmoran

We don't modify anything considering this $PORT environment variable inside our manifest.yml files.

From our Pivotal Apps Manager I don't see any environment variable named PORT inside "User Provided Env Vars".

Our applications can be correctly deployed on our servers if we modify the manifests like it:

---
applications:
  - instances: 1
    memory: 64M
    buildpack: https://github.com/cloudfoundry/staticfile-buildpack#v1.5.1
    stack: cflinuxfs3
    env:
      FORCE_HTTPS: true
      TZ: Europe/Paris

But doing this, we can had security problems in future...

[ryanmoran]

@cdromas can you cf ssh <appname> into your app container and report what you see in the environment for the $PORT variable?

[ukabu]

this has nothing to do with the port, we have the same issue and it is related to memory usage. It was working great before with 64M, now more memory is required, 128M seems to be the minimum now.

[cdromas]

Seems to do the trick. Thanks! 👍

[ukabu]

Even if we have a workaround, I do think that doubling the memory requirement is a bug. If this is expected, an explanation as to why would be nice :-)

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/171654551

The labels on this github issue will be updated when the story is started.

[dmikusa]

Normally, if you don't have enough memory the OOM killer will stop your app and you'll get a message about the container crashing because it went over the memory limit. That doesn't seem to be happening here.

2020-01-24T11:39:54.893+01:00 [CELL/0] [OUT] Starting health monitoring of container
2020-01-24T11:39:55.875+01:00 [APP/PROC/WEB/0] [ERR] + /home/vcap/app/start_logging.sh
2020-01-24T11:39:55.878+01:00 [APP/PROC/WEB/0] [ERR] + nginx -p /home/vcap/app/nginx -c /home/vcap/app/nginx/conf/nginx.conf
2020-01-24T11:39:57.045+01:00 [APP/PROC/WEB/0] [ERR] Killed
2020-01-24T11:40:56.642+01:00 [HEALTH/0] [ERR] Failed to make TCP connection to port 8080: connection refused (out of memory)
2020-01-24T11:40:56.642+01:00 [CELL/0] [ERR] Timed out after 1m0s: health check never passed.

From what you've posted, you can see that something is getting killed on the 4th line. It almost seems like the health check process is being killed, cause you then get a message from the health check process saying "out of memory" & the health check fails.

I did a quick test and wasn't able to reproduce this:

$ cf push
Pushing from manifest to org dmikusa / space development as dmikusa@pivotal.io...
Using manifest file /Users/dmikusa/Code/Samples/smallest-app/manifest.yml
Getting app info...
Creating app with these attributes...
+ name:         smallest-app
  path:         /Users/dmikusa/Code/Samples/smallest-app
  buildpacks:
+   staticfile_buildpack
+ memory:       64M
  routes:
+   smallest-app.cfapps.io

Creating app smallest-app...
Mapping routes...
Comparing local files to remote cache...
Packaging files to upload...
Uploading files...
 318 B / 318 B [==============================================================================================================================================================] 100.00% 1s

Waiting for API to complete processing files...

Staging app and tracing logs...
   Cell 46298d9d-6991-4c73-a8db-0d7e8d50214d creating container for instance 53e51375-95b1-45e4-b344-aae880a71c5a
   Cell 46298d9d-6991-4c73-a8db-0d7e8d50214d successfully created container for instance 53e51375-95b1-45e4-b344-aae880a71c5a
   Downloading app package...
   Downloaded app package (318B)
   -----> Staticfile Buildpack version 1.5.4
   -----> Installing nginx
          Using nginx version 1.17.8
   -----> Installing nginx 1.17.8
          Copy [/tmp/buildpacks/646f0b5af7563a84395aed4d4c669237/dependencies/a8ab3426bf38edc3510ebfbebccdc51b/nginx-1.17.8-linux-x64-cflinuxfs3-6eade3ab.tgz]
   -----> Root folder /tmp/app
   -----> Copying project files into public
   -----> Configuring nginx
   Exit status 0
   Uploading droplet, build artifacts cache...
   Uploading droplet...
   Uploading build artifacts cache...
   Uploaded build artifacts cache (217B)
   Uploaded droplet (2M)
   Uploading complete
   Cell 46298d9d-6991-4c73-a8db-0d7e8d50214d stopping instance 53e51375-95b1-45e4-b344-aae880a71c5a
   Cell 46298d9d-6991-4c73-a8db-0d7e8d50214d destroying container for instance 53e51375-95b1-45e4-b344-aae880a71c5a
   Cell 46298d9d-6991-4c73-a8db-0d7e8d50214d successfully destroyed container for instance 53e51375-95b1-45e4-b344-aae880a71c5a

Waiting for app to start...

name:              smallest-app
requested state:   started
routes:            smallest-app.cfapps.io
last uploaded:     Thu 05 Mar 14:24:26 EST 2020
stack:             cflinuxfs3
buildpacks:        staticfile

type:            web
instances:       1/1
memory usage:    64M
start command:   $HOME/boot.sh
     state     since                  cpu    memory     disk      details
#0   running   2020-03-05T19:24:39Z   0.0%   0 of 64M   0 of 1G

and after sending some requests, memory usage seems to stabilize around 8-9M, well under 64M.

$ cf app smallest-app
Showing health and status for app smallest-app in org dmikusa / space development as dmikusa@pivotal.io...

name:              smallest-app
requested state:   started
routes:            smallest-app.cfapps.io
last uploaded:     Thu 05 Mar 14:24:26 EST 2020
stack:             cflinuxfs3
buildpacks:        staticfile

type:           web
instances:      1/1
memory usage:   64M
     state     since                  cpu    memory        disk         details
#0   running   2020-03-05T19:24:38Z   0.2%   8.5M of 64M   5.3M of 1G

What are the specific steps that you can use to reproduce this? Does it fail every time for you with a 64M memory limit? Also, are you including a .profile script with your app to do anything before the app starts?

Lastly, you mentioned Since last update of our PCF configurations. What specifically do you mean by this? Did your operations team upgrade PCF itself? If so, what version were you running previous and what version are you running now (looks like 2.6.14 now)?

[ryanmoran]

Closing due to inactivity. Feel free to open another issue with more details.

[ritubasant44]

staticfile_buildpack version 1.5.29 giving issue due to not getting support. How to upgrade version