[BUG] 502 Bad Gateway when fronting with a Load Balancer on K8's

sOblivionsCall commented 4 years ago

Describe the bug Nginx is giving a 502 when running on my K8's deployment.

To Reproduce Steps to reproduce the behavior: Create a K8's Deployment with the following:

cat speedtest-deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: speedtest-deployment
  labels:
    app: speedtest
spec:
  replicas: 1
  selector:
    matchLabels:
      app: speedtest
  template:
    metadata:
      labels:
        app: speedtest
    spec:
      volumes:
      - name: host-volume
        persistentVolumeClaim:
          claimName: speedtest-pvc
      containers:
      - name: speedtest
        image: henrywhitaker3/speedtest-tracker
        env:
        - name: OOKLA_EULA_GDPR
          value: "true"
        ports:
        - containerPort: 80
        volumeMounts:
        - name: host-volume
          mountPath: /config
          #      dnsConfig:
          #        nameservers:
          #          - 10.1.1.1
          #        options:
          #          - name: ndots
          #            value: "1"

cat speedtest-svc.yml

apiVersion: v1
kind: Service
metadata:
  name: speedtest-service
spec:
  type: LoadBalancer
  selector:
    app: speedtest
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

cat speedtest-pvc.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: speedtest-pvc
spec:
  storageClassName: nfs
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

kubectl describe pod speedtest-deployment-7799f4f96b-9sdkh
Name:         speedtest-deployment-7799f4f96b-9sdkh
Namespace:    default
Priority:     0
Node:         ryzen-hypervisor/10.1.1.20
Start Time:   Fri, 17 Jul 2020 13:39:01 -0500
Labels:       app=speedtest
              pod-template-hash=7799f4f96b
Annotations:  cni.projectcalico.org/podIP: 10.42.1.36/32
              cni.projectcalico.org/podIPs: 10.42.1.36/32
Status:       Running
IP:           10.42.1.36
IPs:
  IP:           10.42.1.36
Controlled By:  ReplicaSet/speedtest-deployment-7799f4f96b
Containers:
  speedtest:
    Container ID:   docker://23945a11bcb2ea5909a9cbe1e8851fb6fc5650b7c90cfbd2da2f7366c457959e
    Image:          henrywhitaker3/speedtest-tracker
    Image ID:       docker-pullable://henrywhitaker3/speedtest-tracker@sha256:43538792b05238a06199da8aa157dff5daea0d4862feb6ca2c8178664b109f16
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 17 Jul 2020 13:39:15 -0500
    Ready:          True
    Restart Count:  0
    Environment:
      OOKLA_EULA_GDPR:  true
    Mounts:
      /config from host-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xrc2x (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  host-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  speedtest-pvc
    ReadOnly:   false
  default-token-xrc2x:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-xrc2x
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                       Message
  ----    ------     ----  ----                       -------
  Normal  Scheduled  53m   default-scheduler          Successfully assigned default/speedtest-deployment-7799f4f96b-9sdkh to ryzen-hypervisor
  Normal  Pulling    53m   kubelet, ryzen-hypervisor  Pulling image "henrywhitaker3/speedtest-tracker"
  Normal  Pulled     53m   kubelet, ryzen-hypervisor  Successfully pulled image "henrywhitaker3/speedtest-tracker"
  Normal  Created    52m   kubelet, ryzen-hypervisor  Created container speedtest
  Normal  Started    52m   kubelet, ryzen-hypervisor  Started container speedtest

kubectl describe svc speedtest-service
Name:                     speedtest-service
Namespace:                default
Labels:                   <none>
Annotations:              field.cattle.io/publicEndpoints:
                            [{"addresses":["10.1.1.35"],"port":80,"protocol":"TCP","serviceName":"default:speedtest-service","allNodes":false}]
Selector:                 app=speedtest
Type:                     LoadBalancer
IP:                       10.43.161.75
LoadBalancer Ingress:     10.1.1.35
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  30978/TCP
Endpoints:                10.42.1.36:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason        Age                 From             Message
  ----    ------        ----                ----             -------
  Normal  nodeAssigned  18m (x43 over 15h)  metallb-speaker  announcing from node "ryzen-hypervisor"

Pod Logs:

kubectl logs speedtest-deployment-768967465b-td9xm
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01-envfile: executing... 
[cont-init.d] 01-envfile: exited 0.
[cont-init.d] 10-adduser: executing... 
usermod: no changes

-------------------------------------
          _         ()
         | |  ___   _    __
         | | / __| | |  /  \ 
         | | \__ \ | | | () |
         |_| |___/ |_|  \__/

Brought to you by linuxserver.io
-------------------------------------

To support LSIO projects visit:
https://www.linuxserver.io/donate/
-------------------------------------
GID/UID
-------------------------------------

User uid:    911
User gid:    911
-------------------------------------

[cont-init.d] 10-adduser: exited 0.
[cont-init.d] 20-config: executing... 
[cont-init.d] 20-config: exited 0.
[cont-init.d] 30-keygen: executing... 
using keys found in /config/keys
[cont-init.d] 30-keygen: exited 0.
[cont-init.d] 40-config: executing... 
Starting 2019/12/30, GeoIP2 databases require personal license key to download. Please manually download/update the GeoIP2 db and save as /config/geoip2db/GeoLite2-City.mmdb
[cont-init.d] 40-config: exited 0.
[cont-init.d] 50-speedtest: executing... 
Ookla GDPR and EULA accepted. Downloading Speedtest CLI.
wget: bad address 'bintray.com'
tar: can't open 'speedtest.tgz': No such file or directory
cp: cannot stat 'speedtest': No such file or directory
s6-applyuidgid: fatal: unable to exec /site/app/Bin/speedtest: No such file or directory
Copying latest site files to config
Database file exists
Env file exists
Running database migrations
Nothing to migrate.
App key exists
JWT secret exists
Slack webhook is unset
Telegram chat id and bot token unset
Base path is unset
[cont-init.d] 50-speedtest: exited 0.
[cont-init.d] 99-custom-files: executing... 
[custom-init] no custom files found exiting...
[cont-init.d] 99-custom-files: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
nginx: [alert] detected a LuaJIT version which is not OpenResty's; many optimizations will be disabled and performance will be compromised (see https://github.com/openresty/luajit2 for OpenResty's LuaJIT or, even better, consider using the OpenResty releases from https://openresty.org/en/download.html)

curl 10.1.1.35
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>

Expected behavior Hit load balancer ip address and have the react front end presented

Screenshots If applicable, add screenshots to help explain your problem.

Context

OS: Ubuntu 18.04 Docker 19.3.11
Speedtest-Tracker Version: henrywhitaker3/speedtest-tracker:latest

Additional context Rancher K8's

Am happy to supply spec files and documentation to implement this on K8's via PR if you can help figured out the 502

sOblivionsCall commented 4 years ago

this snippet from logs may be the issue:

Ookla GDPR and EULA accepted. Downloading Speedtest CLI. wget: bad address 'bintray.com' tar: can't open 'speedtest.tgz': No such file or directory cp: cannot stat 'speedtest': No such file or directory s6-applyuidgid: fatal: unable to exec /site/app/Bin/speedtest: No such file or directory

sOblivionsCall commented 4 years ago

301 may be breaking wget?

curl bintray.com
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

sOblivionsCall commented 4 years ago

can manually wget so not i'm starting to think it's DNS

wget https://bintray.com/ookla/download/download_file?file_path=ookla-speedtest-1.0.0-x86_64-linux.tgz -O speedtest.tgz
--2020-07-17 15:12:17--  https://bintray.com/ookla/download/download_file?file_path=ookla-speedtest-1.0.0-x86_64-linux.tgz
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving bintray.com (bintray.com)... 75.126.208.206
Connecting to bintray.com (bintray.com)|75.126.208.206|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://dl.bintray.com/ookla/download/ookla-speedtest-1.0.0-x86_64-linux.tgz?expiry=1595016768008&signature=arLNIc0goid19%2F9oWDdRdqcwaLgx4sN8BCXLG%2Fvh5F%2FUCjvP7pw3ORDmt3L7Lv%2BKH7pXFG9y8qoiQFPWtmOVYA%3D%3D [following]
--2020-07-17 15:12:18--  https://dl.bintray.com/ookla/download/ookla-speedtest-1.0.0-x86_64-linux.tgz?expiry=1595016768008&signature=arLNIc0goid19%2F9oWDdRdqcwaLgx4sN8BCXLG%2Fvh5F%2FUCjvP7pw3ORDmt3L7Lv%2BKH7pXFG9y8qoiQFPWtmOVYA%3D%3D
Resolving dl.bintray.com (dl.bintray.com)... 52.41.115.189, 52.88.32.158, 54.148.239.199, ...
Connecting to dl.bintray.com (dl.bintray.com)|52.41.115.189|:443... connected.
HTTP request sent, awaiting response... 302 
Location: https://akamai.bintray.com/5f/5fe2028f0d4427e4f4231d9f9cf70e6691bb890a70636d75232fe4d970633168?__gda__=exp=1595017458~hmac=b057f05d0d221e37f46c1e0fad95c79f9e9f80d3460be94c6d9f7c6201f4f712&response-content-disposition=attachment%3Bfilename%3D%22ookla-speedtest-1.0.0-x86_64-linux.tgz%22&response-content-type=application%2Fgzip&requestInfo=U2FsdGVkX19zEpjO73VsVxAM8EMTMaVxEYioTUWCZCVywVBHGT5q-hCn_z1Yzs_fjO8Q388eLUlJXaZDBKZIaFR-Kt2UiqslPB1ibOOqME5_0LOrg6nLoCkxnSZsNIRdDhribtKsFYBfgXSpvWJDEA&response-X-Checksum-Sha1=41ca19b8bea7614c27370453be3c6ef7ea7fa76a&response-X-Checksum-Sha2=5fe2028f0d4427e4f4231d9f9cf70e6691bb890a70636d75232fe4d970633168 [following]
--2020-07-17 15:12:18--  https://akamai.bintray.com/5f/5fe2028f0d4427e4f4231d9f9cf70e6691bb890a70636d75232fe4d970633168?__gda__=exp=1595017458~hmac=b057f05d0d221e37f46c1e0fad95c79f9e9f80d3460be94c6d9f7c6201f4f712&response-content-disposition=attachment%3Bfilename%3D%22ookla-speedtest-1.0.0-x86_64-linux.tgz%22&response-content-type=application%2Fgzip&requestInfo=U2FsdGVkX19zEpjO73VsVxAM8EMTMaVxEYioTUWCZCVywVBHGT5q-hCn_z1Yzs_fjO8Q388eLUlJXaZDBKZIaFR-Kt2UiqslPB1ibOOqME5_0LOrg6nLoCkxnSZsNIRdDhribtKsFYBfgXSpvWJDEA&response-X-Checksum-Sha1=41ca19b8bea7614c27370453be3c6ef7ea7fa76a&response-X-Checksum-Sha2=5fe2028f0d4427e4f4231d9f9cf70e6691bb890a70636d75232fe4d970633168
Resolving akamai.bintray.com (akamai.bintray.com)... 23.50.129.116
Connecting to akamai.bintray.com (akamai.bintray.com)|23.50.129.116|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 930614 (909K) [application/gzip]
Saving to: ‘speedtest.tgz’

speedtest.tgz                                         100%[=======================================================================================================================>] 908.80K  --.-KB/s    in 0.09s   

2020-07-17 15:12:18 (10.4 MB/s) - ‘speedtest.tgz’ saved [930614/930614]

henrywhitaker3 commented 4 years ago

I won't be able to help until tomorrow evening, maybe the day after. Do you get the same 301 when curling https://bintray.com. Can you try running wget https://bintray.com/ookla/download/download_file?file_path=ookla-speedtest-1.0.0-x86_64-linux.tgz and see if you get the same error?

I have 0 experience with k8 so not sure how much I can help there, but if you can maybe try running it as a standalone container and see if that works, then we can see if it's an issue with k8 or your system in general.

Edit: ignore the part about running the wget command then, just seen your latest comment

Is there a way to set a Dns resolver in k8? Might solve the issue

sOblivionsCall commented 4 years ago

working through it still, starting to think it's an ndots issue with K8's i've seen this same behavior with Organizr

edit: confirmed ndots issue

Ookla GDPR and EULA accepted. Downloading Speedtest CLI.
Connecting to bintray.com (75.126.118.184:443)
Connecting to dl.bintray.com (54.148.239.199:443)
Connecting to akamai.bintray.com (23.50.129.116:443)
saving to 'speedtest.tgz'
speedtest.tgz        100% |********************************|  908k  0:00:00 ETA
'speedtest.tgz' saved
==============================================================================

You may only use this Speedtest software and information generated
from it for personal, non-commercial use, through a command line
interface on a personal computer. Your use of this software is subject
to the End User License Agreement, Terms of Use and Privacy Policy at
these URLs:

    https://www.speedtest.net/about/eula
    https://www.speedtest.net/about/terms
    https://www.speedtest.net/about/privacy

==============================================================================

sOblivionsCall commented 4 years ago

interesting, still getting a 502 even when the speedtest downloads. I can manually go and exec the speedtest and it appears as if it isn't accepting the eula:

kubectl exec -it speedtest-deployment-59b5c84ddf-chhlj /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
root@speedtest-deployment-59b5c84ddf-chhlj:/# cd /site/app/Bin/
root@speedtest-deployment-59b5c84ddf-chhlj:/site/app/Bin# ./speedtest
==============================================================================

You may only use this Speedtest software and information generated
from it for personal, non-commercial use, through a command line
interface on a personal computer. Your use of this software is subject
to the End User License Agreement, Terms of Use and Privacy Policy at
these URLs:

    https://www.speedtest.net/about/eula
    https://www.speedtest.net/about/terms
    https://www.speedtest.net/about/privacy

==============================================================================

Do you accept the license? [type YES to accept]: YES
License acceptance recorded. Continuing.

   Speedtest by Ookla

Edit: redacted personal info

henrywhitaker3 commented 4 years ago

Yeah you have to run HOME=/config && ./speedtest as the files marking your EULA agreement is stored there.

Seems weird your 502 error, is there anything in the nginx logs? You shouldn't get a 503 even if the speedtest binary is missing.

sOblivionsCall commented 4 years ago

went ahead and did a sanity check, using pure docker everything does work. We can confirm it's something with K8's

docker run -it -p 8765:80 -e OOKLA_EULA_GDPR=true henrywhitaker3/speedtest-tracker /bin/bash
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01-envfile: executing... 
[cont-init.d] 01-envfile: exited 0.
[cont-init.d] 10-adduser: executing... 
usermod: no changes

-------------------------------------
          _         ()
         | |  ___   _    __
         | | / __| | |  /  \ 
         | | \__ \ | | | () |
         |_| |___/ |_|  \__/

Brought to you by linuxserver.io
-------------------------------------

To support LSIO projects visit:
https://www.linuxserver.io/donate/
-------------------------------------
GID/UID
-------------------------------------

User uid:    911
User gid:    911
-------------------------------------

[cont-init.d] 10-adduser: exited 0.
[cont-init.d] 20-config: executing... 
[cont-init.d] 20-config: exited 0.
[cont-init.d] 30-keygen: executing... 
generating self-signed keys in /config/keys, you can replace these with your own keys if required
Generating a RSA private key
..............................+++++
......................+++++
writing new private key to '/config/keys/cert.key'
-----
[cont-init.d] 30-keygen: exited 0.
[cont-init.d] 40-config: executing... 
Starting 2019/12/30, GeoIP2 databases require personal license key to download. Please manually download/update the GeoIP2 db and save as /config/geoip2db/GeoLite2-City.mmdb
[cont-init.d] 40-config: exited 0.
[cont-init.d] 50-speedtest: executing... 
Ookla GDPR and EULA accepted. Downloading Speedtest CLI.
Connecting to bintray.com (75.126.208.206:443)
Connecting to dl.bintray.com (52.41.115.189:443)
Connecting to akamai.bintray.com (23.50.129.116:443)
saving to 'speedtest.tgz'
speedtest.tgz        100% |**********************************************************************************************************************************************************************|  908k  0:00:00 ETA
'speedtest.tgz' saved
==============================================================================

You may only use this Speedtest software and information generated
from it for personal, non-commercial use, through a command line
interface on a personal computer. Your use of this software is subject
to the End User License Agreement, Terms of Use and Privacy Policy at
these URLs:

    https://www.speedtest.net/about/eula
    https://www.speedtest.net/about/terms
    https://www.speedtest.net/about/privacy

==============================================================================

License acceptance recorded. Continuing.

Copying latest site files to config
Database file not found! Creating empty database
Env file not found! Creating .env file
Running database migrations
Migration table created successfully.
Migrating: 2014_10_12_000000_create_users_table
Migrated:  2014_10_12_000000_create_users_table (0.14 seconds)
Migrating: 2019_08_19_000000_create_failed_jobs_table
Migrated:  2019_08_19_000000_create_failed_jobs_table (0.01 seconds)
Migrating: 2020_04_03_104234_create_email_verifications_table
Migrated:  2020_04_03_104234_create_email_verifications_table (0.01 seconds)
Migrating: 2020_04_03_105113_create_active_sessions_table
Migrated:  2020_04_03_105113_create_active_sessions_table (0.01 seconds)
Migrating: 2020_04_08_120837_create_speedtests_table
Migrated:  2020_04_08_120837_create_speedtests_table (0.01 seconds)
Migrating: 2020_04_08_125647_create_jobs_table
Migrated:  2020_04_08_125647_create_jobs_table (0.03 seconds)
Migrating: 2020_05_18_211812_create_settings_table
Migrated:  2020_05_18_211812_create_settings_table (0.06 seconds)
Migrating: 2020_06_20_164502_update_speedtests_table
Migrated:  2020_06_20_164502_update_speedtests_table (0.05 seconds)
Migrating: 2020_06_21_171849_add_notifications_settings
Migrated:  2020_06_21_171849_add_notifications_settings (0.04 seconds)
Generating app key
Application key set successfully.
Generating JWT secret
jwt-auth secret [cYnXpTg6vaXS4vY9W4jUqMMI7tf2Omj7mMCqxUQbAsaS390ewKWKy4SUfmRyooP9] set successfully.
Slack webhook is unset
Telegram chat id and bot token unset
Base path is unset
[cont-init.d] 50-speedtest: exited 0.
[cont-init.d] 99-custom-files: executing... 
[custom-init] no custom files found exiting...
[cont-init.d] 99-custom-files: exited 0.
[cont-init.d] done.
[services.d] starting services
crond[410]: crond (busybox 1.31.1) started, log level 5
[services.d] done.
root@f8dce0c2ba7d:/# [17-Jul-2020 20:55:20] NOTICE: fpm is running, pid 408
[17-Jul-2020 20:55:20] NOTICE: ready to handle connections
nginx: [alert] detected a LuaJIT version which is not OpenResty's; many optimizations will be disabled and performance will be compromised (see https://github.com/openresty/luajit2 for OpenResty's LuaJIT or, even better, consider using the OpenResty releases from https://openresty.org/en/download.html)
crond[410]: USER root pid 451 cmd php /config/www/artisan schedule:run >> /config/log/speedtest/cron.log
crond[410]: USER root pid 458 cmd php /config/www/artisan schedule:run >> /config/log/speedtest/cron.log

I have to be missing something we need in my deployment spec file

sOblivionsCall commented 4 years ago

nothing to be found in the nginx logs, i don't think the app is starting even thought stdout says it is

root@speedtest-deployment-59b5c84ddf-chhlj:/var/log/nginx# cat error.log 
2020/07/17 20:26:11 [alert] 387#387: detected a LuaJIT version which is not OpenResty's; many optimizations will be disabled and performance will be compromised (see https://github.com/openresty/luajit2 for OpenResty's LuaJIT or, even better, consider using the OpenResty releases from https://openresty.org/en/download.html)

sOblivionsCall commented 4 years ago

i have found the difference in logs but am not sure what is causing it:

Docker:

[services.d] starting services
crond[410]: crond (busybox 1.31.1) started, log level 5
[services.d] done.
root@f8dce0c2ba7d:/# [17-Jul-2020 20:55:20] NOTICE: fpm is running, pid 408
[17-Jul-2020 20:55:20] NOTICE: ready to handle connections
nginx: [alert] detected a LuaJIT version which is not OpenResty's; many optimizations will be disabled and performance will be compromised (see https://github.com/openresty/luajit2 for OpenResty's LuaJIT or, even better, consider using the OpenResty releases from https://openresty.org/en/download.html)

K8's

[cont-init.d] 50-speedtest: exited 0.
[cont-init.d] 99-custom-files: executing... 
[custom-init] no custom files found exiting...
[cont-init.d] 99-custom-files: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
nginx: [alert] detected a LuaJIT version which is not OpenResty's; many optimizations will be disabled and performance will be compromised (see https://github.com/openresty/luajit2 for OpenResty's LuaJIT or, even better, consider using the OpenResty releases from https://openresty.org/en/download.html)

the docker implementation says that app is ready to serve traffic but the K8's implementation doesn't

sOblivionsCall commented 4 years ago

going further down the rabbit hole, it looks like there is no laravel.log so i'm 99% sure fpm isn't starting at all

sOblivionsCall commented 4 years ago

ok so have done all the troubleshooting i can think of at this point.

root@speedtest-deployment-646d7bf48-hb6gj:/config/log/nginx# ls
access.log  error.log
root@speedtest-deployment-646d7bf48-hb6gj:/config/log/nginx# cat access.log error.log

root@speedtest-deployment-646d7bf48-hb6gj:/config/log# cd php/
root@speedtest-deployment-646d7bf48-hb6gj:/config/log/php# ls

root@speedtest-deployment-646d7bf48-hb6gj:/config/log# cd speedtest/
root@speedtest-deployment-646d7bf48-hb6gj:/config/log/speedtest# ls
cron.log  queue.log
root@speedtest-deployment-646d7bf48-hb6gj:/config/log/speedtest# cat cron.log queue.log 
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
Running scheduled command: App\Jobs\SpeedtestJob
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
Running scheduled command: App\Jobs\SpeedtestJob
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
Running scheduled command: App\Jobs\SpeedtestJob
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
No scheduled commands are ready to run.
[2020-07-18 01:00:07][1] Processing: App\Jobs\SpeedtestJob
[2020-07-18 01:00:36][1] Processed:  App\Jobs\SpeedtestJob
[2020-07-18 02:00:04][2] Processing: App\Jobs\SpeedtestJob
[2020-07-18 02:00:30][2] Processed:  App\Jobs\SpeedtestJob
[2020-07-18 03:00:02][3] Processing: App\Jobs\SpeedtestJob
[2020-07-18 03:00:34][3] Processed:  App\Jobs\SpeedtestJob

root@speedtest-deployment-646d7bf48-hb6gj:/config/log/speedtest# ps -ef | grep fpm
root       405     1  0 00:38 ?        00:00:14 s6-supervise php-fpm
root     32066 31317  0 03:26 pts/1    00:00:00 grep fpm

At first i thought it may be a file ownership issue with the NFS mount providing storage. I removed the PVC entirely and let it run as an ephemeral volume but that doesn't seem to be the issue
The ndots issue with speedtest downloading has been resolved but isn't related to the 502
FPM seems to be started and running but isn't generating logs, i even set the environment variable for DEBUG but still no joy.

My only assumption at this point is FPM is either hung or crashing but i have no idea how to find out how/why. Any ideas on next steps?

henrywhitaker3 commented 4 years ago

So will be a day or so until I have a computer in front of my to try figure it out, but this sounds like an issue with the base image for the container rather than anything I have changed about it. It might be worth creating an issue here or asking a question in their discord server.

sOblivionsCall commented 4 years ago

Issue has been resolved, it wasn't an issue with your code or the base container. FPM was throwing a core dump error on startup it just wasn't showing up in any logs. Specifically the container needs huge pages to be enabled:

https://github.com/kubernetes/kubernetes/issues/71233

        resources:
          limits:
            hugepages-2Mi: 100Mi
            memory: 100Mi

I can make a PR with my K8's spec files and implementation documentation if you are interested in having that as a deployment option for your users.

henrywhitaker3 commented 4 years ago

Good to hear it's fixed.

Yeah that would be great, I haven't been asked by anyone else about k8, but it can't hurt to have some documentation for it.

sOblivionsCall commented 4 years ago

not sure how you have your access setup, also i'm not very familiar with collaboration on public git repos, but it looks like i can't push due to an access issue:

git push --set-upstream origin feature/k8s_support
ERROR: Permission to henrywhitaker3/Speedtest-Tracker.git denied to sOblivionsCall.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

henrywhitaker3 commented 4 years ago

Ah you need to fork the repo, make changes to it then PR from your repo to mine

sOblivionsCall commented 4 years ago

will do , thanks

henrywhitaker3 / Speedtest-Tracker

[BUG] 502 Bad Gateway when fronting with a Load Balancer on K8's #183