Closed nohant closed 1 month ago
withinboredom
commented
1 month ago I believe there is an issue with alpine (musl) + openssl atm (see #676) which affects Octane. If this is the same issue, please close this one. It appears the workaround is to use Debian based images instead.
nohant
commented
1 month ago we have the same problem with the debian image @withinboredom. i forgot to write that down in the error note. Same error, same configuration, only 139 in the pod.
withinboredom
commented
1 month ago Ok, 139 usually means it was killed by the kernel (sigsev, over memory limits, etc) so I'd check that the memory limits are sane or not-applicable.
nohant
commented
1 month ago we removed any request/limit from the pod, i can put here the manifest if in need, its no problem:
apiVersion: apps/v1
kind: Deployment
metadata:
name: rimborsi
spec:
selector:
matchLabels:
app: rimborsi
template:
metadata:
labels:
app: rimborsi
spec:
serviceAccountName: rimborsi
containers:
- name: rimborsi
image: rimborsi
imagePullPolicy: Always
command:
- ./entrypoint.sh
- serve_backend
livenessProbe:
httpGet:
path: /metrics-g3lka4TZ2H4G5gT
port: 8081
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
envFrom:
- configMapRef:
name: rimborsi-env
- secretRef:
name: rimborsi-secret-env
volumeMounts:
- mountPath: /data/caddy
name: caddy-data
volumes:
- name: caddy-data
emptyDir: {}
securityContext:
runAsNonRoot: true
runAsUser: 65534 # nobodyi can try again to test debian instead of alpine for some reason, baybe third time a charm xD
withinboredom
commented
1 month ago Also check that you have enough memory on the host. There should be something in /var/log on the host that tells why the kernel killed it too (/var/log/syslog on my machine)
nohant
commented
1 month ago This is the max i can get from the kubelet about this pod:
generic.go:334] "Generic (PLEG): container finished" podID="b27a9ed5-0288-4759-9639-46328341de13" containerID="c7d58d16ea2aa709b00c235358f75a4b386926349e8719b8305489409480a0cd" exitCode=139
thats why we are so confused, we cant get anything from anywhere. we are also removing libraries and components to test if is something wrong in our php extensions
nohant
commented
1 month ago restarted with the debian version, this is the last log i get:
php artisan octane:frankenphp --host 0.0.0.0 --port 8080 --admin-port 2019 --log-level debug
{"level":"info","ts":1717146237.7450576,"msg":"using provided configuration","config_file":"/composer/vendor/laravel/octane/src/Commands/stubs/Caddyfile","config_adapter":""}
{"level":"info","ts":1717146237.7472234,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1717146237.7474031,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x400040db00"}
{"level":"info","ts":1717146237.7477,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1717146237.7503412,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"}
{"level":"info","ts":1717146237.750461,"logger":"tls","msg":"finished cleaning storage units"}but it still dies to error 139.
withinboredom
commented
1 month ago This is the max i can get from the kubelet about this pod
You're going to need to look at kernel-level logs, which will be in the host and nothing to do with Kubernetes. That's the last bet to figure out what is happening.
Looking at your deployment though, I do see something that might be the problem. Try adding this to your container spec:
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_BIND_SERVICE
drop:
- ALLThe docker images have the cap built-in:
https://github.com/dunglas/frankenphp/blob/main/Dockerfile#L93
And need the cap at runtime, especially if running as non-root.
nohant
commented
1 month ago @withinboredom
at a certain point i had this configured following this but probably i lost that piece of configuration when i reverted lots of thing. im testing the BIND on kubernetes now. (waiting for tests and build time, debian is way slower).
im searching into the aws ec2 arm node where to find kernel logs in the meantime because i cant find them
withinboredom
commented
1 month ago debian is way slower
Interesting :) are you using a build cache? It should be insta-build if the code didn't change.
@dunglas just did a pretty epic talk about some of that: https://dunglas.dev/2024/05/containerization-tips-and-tricks-for-php-apps/ if you are using github actions.
nohant
commented
1 month ago Thanks @withinboredom , dmsg doesnt have anything about the memory problem, maybe im looking in the wrong position
-rw------- 1 root root 0 May 3 23:42 tallylog
-rw-r--r-- 1 root root 193 May 3 23:42 grubby_prune_debug
-rw------- 1 root utmp 0 May 3 23:42 btmp
-rw------- 1 root root 0 May 3 23:42 spooler
drwxr-xr-x 18 root root 254 May 14 04:15 ..
drwx------ 2 root root 23 May 14 04:15 audit
-rw------- 1 root root 2.4K May 14 04:15 grubby
-rw-r--r-- 1 root root 24K May 14 04:16 dmesg.old
-rw------- 1 root root 6.4K May 14 04:19 yum.log
-rw-r--r-- 1 root root 290K May 14 04:19 lastlog
drwxr-sr-x+ 4 root systemd-journal 86 May 29 00:00 journal
-rw-r--r-- 1 root root 27K May 29 00:00 dmesg
drwxr-xr-x 3 root root 17 May 29 00:00 amazon
-rw------- 1 root root 798 May 29 00:00 maillog
-rw------- 1 root root 8.6K May 29 00:00 secure
-rw-r--r-- 1 root root 6.3K May 29 00:00 user-data.log
-rw-r----- 1 root root 2.8K May 29 00:00 cloud-init-output.log
-rw-r--r-- 1 root root 96K May 29 00:00 cloud-init.log
drwxr-xr-x 9 root root 4.0K May 29 00:00 .
-rw-rw-r-- 1 root utmp 2.4K May 29 00:00 wtmp
drwxr-xr-x 2 root root 121 May 29 00:00 aws-routed-eni
drwxr-x--- 2 chrony chrony 276 May 31 03:32 chrony
-rw------- 1 root root 19K May 31 09:01 cron
-rw------- 1 root root 11K May 31 09:03 boot.log
drwxr-xr-x 58 root root 8.0K May 31 09:21 pods
drwxr-xr-x 2 root root 12K May 31 09:21 containers
-rw------- 1 root root 11M May 31 09:21 messagesabout the CI youre right, i forgot i had already run that, it took 40s xD still no working, same 139 error (alpine or debian is the same.) do you think i have to put [this] (https://frankenphp.dev/docs/docker/#running-with-no-capabilities) in the dockerfile too?
withinboredom
commented
1 month ago You could look in the journal file, but you probably need journalctl to parse it as it is binary-ish. It's still parsable by humans in a pinch, somewhat.
I also see that you're using the regular PHP to start up artisan... dunno if that has any affect. Maybe change your entrypoint:
frankenphp php-cli artisan octane:frankenphpIf I'm reading right though, it works locally but not in kubernetes? You also mentioned that the k8s machines are arm? I wonder if there is something architecturally specific going on there...
nohant
commented
1 month ago Locally works without problem (mac m1), but on kubernetes under arm nodes no. :/ i will try the frankenphp command for entrypoint and be back in a sec :)
withinboredom
commented
1 month ago You could also set up a deployment where the entrypoint is ["sleep", "infinity"] (and set the user to root so you can install tooling) then exec into the container and try running it manually. From there, you can use this:
https://mercure.rocks/docs/hub/debug
which also works for frankenphp. You can also use gdb to grab a stack trace. Sadly, nothing is standing out to me about your setup and I'm out of "is it plugged in" type of debugging. Time to get our hands dirty, so to speak.
nohant
commented
1 month ago [feat/data][~/Documents/workspace/infrastructure/staging]$ k exec -it rimborsi-76fdd6db88-qwjq4 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
nobody@rimborsi-76fdd6db88-qwjq4:/app$this was pretty easy, we can do that at runtime if we stop argo :D
Where would you suggest to start first?
withinboredom
commented
1 month ago I would start by running your entry point manually. Maybe you'll get more information. It's mostly standard debugging from this point, though the tools might be unfamiliar (if coming from a php-only background) or you may be a bit rusty if you haven't run gdb in awhile.
nohant
commented
1 month ago ok, finally something i can work with @withinboredom
[2024-05-31 12:08:56] stag.ERROR: The /tmp/bootstrap/cache directory must be present and writable. {"exception":"[object] (Exception(code: 0): The /tmp/bootstrap/cache directory must be present and writable. at /composer/vendor/laravel/framework/src/Illuminate/Foundation/PackageManifest.php:178)i should create this folder in the dockerfile since its missing, but having added that folder manually now, nothing change compared to the usual behaviour.
withinboredom
commented
1 month ago If you want, you can shoot me an email (landers dot robert at gmail) and maybe we can hop on a call? Might be faster. I have some free time this afternoon (around 4:30 pm CEDT or later).
nohant
commented
1 month ago ive enabled an emptydir for tmp since it was missing at the moment. i will sure send you an email, but this afternoon i cant :( thanks for real <3.
nohant
commented
1 month ago nobody@rimborsi-55d86f9b7-j9vql:/app$ env | grep -i tmp
APP_SERVICES_CACHE=/tmp/bootstrap/cache/services.php
APP_ROUTES_CACHE=/tmp/bootstrap/cache/routes-v7.php
HOME=/tmp
APP_CONFIG_CACHE=/tmp/bootstrap/cache/config.php
FILESYSTEM_TMP_DISK=s3_tmp
APP_PACKAGES_CACHE=/tmp/bootstrap/cache/packages.php
APP_STORAGE_PATH=/tmp/storage
APP_EVENTS_CACHE=/tmp/bootstrap/cache/events.phpwe are getting somewhere now since we do not have tmp, but the bootstrap is under app
nohant
commented
1 month ago [2024-05-31 12:36:17] stag.ERROR: The /app/bootstrap/cache directory must be present and writable. {"exception":"[object] (Exception(code: 0): The /app/bootstrap/cache directory must be present and writable. at /composer/vendor/laravel/framework/src/Illuminate/Foundation/PackageManifest.php:178)
yeah, even if its in the right folder we are not getting anywhere. :D
nohant
commented
1 month ago with a brutal 777 we are getting now:
[2024-05-31 13:04:00] stag.ERROR: Unable to write to process ID file. {"exception":"[object] (RuntimeException(code: 0): Unable to write to process ID file. at /composer/vendor/laravel/octane/src/FrankenPhp/ServerStateFile.php:49)
AlliBalliBaba
commented
1 month ago If you want to start Caddy as www-data user, you need to make/data and /config writable for www-data, since Caddy will generate files by default in /data/caddy and /config/caddy.
The reason you are seeing the last error is probably because there is no /app/storage/logs folder or it's not writable. Octane will try to write a file there when it boots.
nohant
commented
1 month ago thanks @AlliBalliBaba
RUN mkdir -p /tmp/storage/logs; chmod -R 0777 /tmp/storage
we added this configuration, and even after this we are receiving the error you see before.
we spent whole dau on that and we are still suck at that errore.
about caddy we have this step:
RUN \ setcap CAP_NET_BIND_SERVICE=+eip /usr/local/bin/frankenphp; \ chown -R ${USER}:${USER} /data/caddy && chown -R ${USER}:${USER} /config/caddy
but nothing changed, we are kinda burned xD
withinboredom
commented
1 month ago Have you looked at that php file to determine where it is trying to write the PID file?
nohant
commented
1 month ago i cant understand where he wants to work
<?php
namespace Laravel\Octane\FrankenPhp;
use RuntimeException;
class ServerStateFile
{
public function __construct(protected string $path)
{
}
/**
* Read the server state from the server state file.
*/
public function read(): array
{
$state = is_readable($this->path)
? json_decode(file_get_contents($this->path), true)
: [];
return [
'masterProcessId' => $state['masterProcessId'] ?? null,
'state' => $state['state'] ?? [],
];
}
/**
* Write the given process ID to the server state file.
*/
public function writeProcessId(int $masterProcessId): void
{
if (! is_writable($this->path) && ! is_writable(dirname($this->path))) {
throw new RuntimeException('Unable to write to process ID file.');
}
file_put_contents($this->path, json_encode(
array_merge($this->read(), ['masterProcessId' => $masterProcessId]),
JSON_PRETTY_PRINT
));
}
/**
* Write the given state array to the server state file.
*/
public function writeState(array $newState): void
{
if (! is_writable($this->path) && ! is_writable(dirname($this->path))) {
throw new RuntimeException('Unable to write to process ID file.');
}
file_put_contents($this->path, json_encode(
array_merge($this->read(), ['state' => $newState]),
JSON_PRETTY_PRINT
));
}
/**
* Delete the process ID file.
*/
public function delete(): bool
{
if (is_writable($this->path)) {
return unlink($this->path);
}
return false;
}
/**
* Get the path to the process ID file.
*/
public function path(): string
{
return $this->path;
}
}do you have any idea?
withinboredom
commented
1 month ago If you're in the container, just slap a vardump or logging statement or throw an exception in the file to dump the path.
nohant
commented
1 month ago i think we have made some actual progress:
+ frankenphp php-cli artisan octane:frankenphp --host 0.0.0.0 --port 8080 --admin-port 2019 --log-level debug
{"level":"info","ts":1717403392.9643722,"msg":"using provided configuration","config_file":"/composer/vendor/laravel/octane/src/Commands/stubs/Caddyfile","config_adapter":""}
{"level":"info","ts":1717403392.9664795,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1717403392.9666653,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x400050cf00"}thats now the only logs we get before the 139. nothing on the machine, nothing anywhere, just this logs and the crash :D. its the same on the debian image and in the alpine.
nohant
commented
1 month ago Starting program: /usr/local/bin/frankenphp php-cli artisan octane:frankenphp --host 0.0.0.0 --port 80 --admin-port 2019
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[New Thread 0xffff4ae6dde0 (LWP 311)]
[New Thread 0xffff4265dde0 (LWP 312)]
[New Thread 0xffff4a61dde0 (LWP 313)]
[New Thread 0xffff49e0dde0 (LWP 314)]
[New Thread 0xffff495fdde0 (LWP 315)]
[New Thread 0xffff48c0dde0 (LWP 316)]
[New Thread 0xffff43ffede0 (LWP 317)]
[New Thread 0xffff437eede0 (LWP 318)]
[Attaching after Thread 0xffff437eede0 (LWP 318) vfork to child process 319]
[New inferior 2 (process 319)]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[Detaching vfork parent process 309 after child exec]
[Inferior 1 (process 309) detached]
process 319 is executing new program: /usr/bin/dash
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[Attaching after Thread 0xffff88568fa0 (LWP 319) vfork to child process 320]
[New inferior 3 (process 320)]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[Detaching vfork parent process 319 after child exec]
[Inferior 2 (process 319) detached]
process 320 is executing new program: /usr/bin/stty
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[Inferior 3 (process 320) exited normally]
(gdb)
INFO Server running…
Local: http://0.0.0.0:80
Press Ctrl+C to stop the server
WARN server is listening only on the HTTP port, so no automatic HTTPS will be applied to this serverrunning the sofware with gdb dont cause a crash apparently (on debian) but will not move anything nor give an error of sort. If i launch manually the command outside of gdb or like entrypoiht for the pod, the 139 error is there waiting for us.
nohant
commented
1 month ago we made some progress, the pod is now working, the problem was from this opcache part.
now we are fighting caddy/webserver because instead of going into app/public it goes www and for obvious reasons the app dont work, but this config in caddy gets ignored,it seem
{
frankenphp
admin off
auto_https off
order php_server before file_server
servers {
metrics
}
log {
format json
}
}
:8080 {
log
header {
-server
-X-Powered-By
}
root * /app/public
encode zstd br gzip
file_server
}
:8081 {
metrics /metrics
}@withinboredom do you have any idea why its ignoring this even if its in the right position?
COPY docker/configs/Caddyfile.production /etc/caddy/Caddyfile
withinboredom
commented
1 month ago Does your entry point tell it to load the caddy file? -c /etc/caddy/Caddyfile
nohant
commented
1 month ago we edited the entrypoint and now it has the correct entryfile, yes, but nonetheless the default answer from the url is /www, even if i do a port forward i get localhost/www.
withinboredom
commented
1 month ago ah, that's weird but that smells like an app configuration issue. You could always stick a header('I: am here' in your index.php to see if it is loading the PHP or not. If it isn't, then it is a caddy issue, if it is, then it is a config issue.
nohant
commented
1 month ago if i manually put localhost/index.php in the browser it works correctly, thats why i tought it was a caddy problem that i cant undestand or something going under the hood. :/
withinboredom
commented
1 month ago I guess I don't understand the problem :)
Are you being redirected to http://localhost/www when going to http://localhost and you're trying to figure out why?
nohant
commented
1 month ago I guess I don't understand the problem :)
Are you being redirected to
http://localhost/wwwwhen going tohttp://localhostand you're trying to figure out why?
exactly.
we dont have anywhere www in the code, and our caddyfile, as you can see, dont use www anywhere xD
withinboredom
commented
1 month ago What is the status code? Could it be cached?
nohant
commented
1 month ago At the moment there is no cache enabled, since we delete the pod and we recreate that, the first call will go directly to this www and we are searching at why.
thats our superpowered now entrypoint for this container: frankenphp php-cli artisan octane:frankenphp --host 0.0.0.0 --port 8080 --admin-port 2019 --log-level debug --caddyfile /etc/caddy/Caddyfile
and this is our caddyfile that is configured following the documentation from franken:
{
frankenphp
admin off
auto_https off
order php_server before file_server
servers {
metrics
}
log {
format json
}
}
:8080 {
log
header {
-server
-X-Powered-By
}
root * /app/public
encode zstd br gzip
php_server
file_server
}
:8081 {
metrics /metrics
}I meant, is the redirect cached in your browser? The browser caches permanent redirects forever (308).
nohant
commented
1 month ago I meant, is the redirect cached in your browser? The browser caches permanent redirects forever (
308).
oh sorry i misunderstood that, the answer is no, the app is not correctly starting because the nginx cant find the app and goes 503. in order to "see" something i need to go manually to port-forward.
also, our colleagues when try to port forward they get the same error.
withinboredom
commented
1 month ago Yeah, I don't see anything that stands out in your caddyfile. Could nginx be the one doing the redirect?
withinboredom
commented
1 month ago The only thing I can think of is that maybe you have an index.htm that is being loaded instead of index.php
nohant
commented
1 month ago The only thing I can think of is that maybe you have an
index.htmthat is being loaded instead ofindex.php
in the port-forward there is no nginx, that is the problem.
its only the app and the caddyfile, if we go to the localhost:8080 (that is the service and/or the pod) we meet that www.
im gonna try again something difference and a cleanup, because i can see that the octane caddyfile is more complex and require this file: /composer/vendor/laravel/octane/src/Commands/stubs/frankenphp-worker.php
i tought too for a moment about the index.htm or something and i doublechecked xD
withinboredom
commented
1 month ago You might need something like this in your Caddyfile:
@notIndex {
not {
path /franken-worker.php
}
}
rewrite @notIndex /franken-worker.phpbut I can't remember laravel specifics off the top of my head.
nohant
commented
1 month ago You might need something like this in your Caddyfile:
@notIndex { not { path /franken-worker.php } } rewrite @notIndex /franken-worker.phpbut I can't remember laravel specifics off the top of my head.
ok, this i don't honestly understand, why i would need this if its not mentioned in the franken documentation? its because of octane? but, looking at the octane doc for frankenphp it doenst state anywhere to use this component/piece.
withinboredom
commented
1 month ago You probably don't need it. I've needed it when testing various dev cases ... but if it isn't documented, then you likely don't need it :) Sorry to confuse you.
nohant
commented
1 month ago this is the octane caddyfile:
{
{$CADDY_GLOBAL_OPTIONS}
admin localhost:{$CADDY_SERVER_ADMIN_PORT}
frankenphp {
worker "{$APP_PUBLIC_PATH}/frankenphp-worker.php" {$CADDY_SERVER_WORKER_COUNT}
}
}
{$CADDY_SERVER_SERVER_NAME} {
log {
level {$CADDY_SERVER_LOG_LEVEL}
# Redact the authorization query parameter that can be set by Mercure...
format filter {
wrap {$CADDY_SERVER_LOGGER}
fields {
uri query {
replace authorization REDACTED
}
}
}
}
route {
root * "{$APP_PUBLIC_PATH}"
encode zstd br gzip
# Mercure configuration is injected here...
{$CADDY_SERVER_EXTRA_DIRECTIVES}
php_server {
index frankenphp-worker.php
# Required for the public/storage/ directory...
resolve_root_symlink
}
}
}its correct too technically, if i put
CADDY_SERVER_SERVER_NAME as :8080
and `$APP_PUBLIC_PATH' as /app/public
so now i have a real doubt:
how is this caddyfile working correctly witht his: frankenphp php-cli artisan octane:frankenphp --host 0.0.0.0 --port 80 --admin-port 2019 --max-requests=1 --log-level debug locally, but our caddyfile dont?
withinboredom
commented
1 month ago Just an "is it plugged in" kind of question: but did you verify the caddyfile in the container is the correct one?
nohant
commented
1 month ago if i dont say what caddyfile to use, it has to use the octane one from the frankenphp default configuration, or this is what i understand at the moment.
What happened?
Hello, we tried to switch from a normal
laravel/fpmconfiguration to afrankenphp/octaneone. the fist step was configuring our local environment to work and that was almost flawless:and our entrypoint configuration is pretty simple to be honest:
and all this config works correctly, locally i can test the software and everything works fine (i love this.). But oh boy, when we tried to configure our staging environment (same image as production, you will read that in the dockerfile) everything went crazy, from caddy to octane to php?
we use a rootless environment and use www-data as our main user, the caddyfile is pretty simple too:
and to complete everything, the entrypoint for production, this is also pretty standard/straightforward:
i dont like to have to force
--admin-portbut i cant make the command work without that, and the log-level debug was needed in order to try to understand something of what was happening to be honest.we cant make this work on kubernetes with a barebone deployment, but we don't understand what we are doing wrong, since the only error we get is a 139 (we can see that from the
kubectl describe pod)by the look of this there is something wrong we are doing that im missing? Thank you for your time.
Build Type
Docker (Alpine)
Worker Mode
No
Operating System
GNU/Linux
CPU Architecture
aarch64
PHP configuration
PHPINFO from the local enviroment (same dockerfile.)
Relevant log output
No response