Game PV Storage CephFS issues

[Dracozny]

Due to the ReadWriteMany flag I need to use cephfs in order to deploy but it seems that with all 3 flags being called at once cephfs wont provision.

  game:
    accessModes:
      - ReadOnlyMany
      - ReadWriteMany
      - ReadWriteOnce
    size: 30Gi
    annotations: {}

Describing the PVC I get:

failed to provision volume with StorageClass "cephfs": rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source

I've only ever seen an app call out one mode and not three different ones so if I am misunderstanding things please let me know.

[DrPsychick]

You can configure the access modes with Values.persistence.game.accessModes. If I'm not mistaken the ReadWriteMany is not supported in all cases in Ceph, a quick search revealed this: https://github.com/ceph/ceph-csi/issues/1038

[Dracozny]

The error looks like it was about the ROX. cephfs seemed happy to provision for the cluster since it only had RWX. After reading this link toward the bottom it seems like a strange conflict for read Only on an empty volume.

➜  ark-cluster kubectl get pvc -n ark
NAME                                   STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
arkcluster-ark-cluster-aberration      Bound     pvc-fb678370-2d22-4952-bb02-70b50fe42a89   2Gi        RWO            ceph-xfs       108m
arkcluster-ark-cluster-cluster         Bound     pvc-c8a68066-a9b5-4256-9b4b-7b66e34a49ba   200Mi      RWX            cephfs         108m
arkcluster-ark-cluster-crystalisles    Bound     pvc-a940e882-36ad-4316-a392-d5d3c6a2e73d   2Gi        RWO            ceph-xfs       108m
arkcluster-ark-cluster-extinction      Bound     pvc-4ad0d191-bf72-4313-b597-0533c8a269a0   2Gi        RWO            ceph-xfs       108m
arkcluster-ark-cluster-fjordur         Bound     pvc-15725f8c-b16e-4064-be72-540cca2fc5a9   2Gi        RWO            ceph-xfs       107m
arkcluster-ark-cluster-game            Pending                                                                        cephfs         108m
arkcluster-ark-cluster-lostisland      Bound     pvc-9d64c1a5-e022-41c4-9882-90fd901188a9   2Gi        RWO            ceph-xfs       107m
arkcluster-ark-cluster-ragnarok        Bound     pvc-cf67af6f-d5c0-44e5-a9f8-9547e904ca23   2Gi        RWO            ceph-xfs       107m
arkcluster-ark-cluster-scorchedearth   Bound     pvc-30f3f942-d65a-4ed4-bcc6-c893c3e29a40   2Gi        RWO            ceph-xfs       107m
arkcluster-ark-cluster-theisland       Bound     pvc-32beedab-63b9-4f8f-ad2c-58b7e6ca5a5b   2Gi        RWO            ceph-xfs       107m
arkcluster-ark-cluster-valguero        Bound     pvc-32bf601c-3121-43a8-a760-c5c83f61bb9e   2Gi        RWO            ceph-xfs       107m

Logs:

I0115 05:51:15.302159       1 controller.go:1337] provision "ark/arkcluster-ark-cluster-game" class "cephfs": started
I0115 05:51:15.302397       1 controller.go:1337] provision "ark/arkcluster-ark-cluster-cluster" class "cephfs": started
I0115 05:51:15.302498       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"ark", Name:"arkcluster-ark-cluster-game", UID:"f075ed11-6288-4521-b005-44a838ba9985", APIVersion:"v1", ResourceVersion:"262580", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "ark/arkcluster-ark-cluster-game"
I0115 05:51:15.302606       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"ark", Name:"arkcluster-ark-cluster-cluster", UID:"c318dd41-d7a8-48bb-a3c2-50b94bde4e89", APIVersion:"v1", ResourceVersion:"262581", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "ark/arkcluster-ark-cluster-cluster"
W0115 05:51:15.699876       1 controller.go:934] Retrying syncing claim "f075ed11-6288-4521-b005-44a838ba9985", failure 0
E0115 05:51:15.699920       1 controller.go:957] error syncing claim "f075ed11-6288-4521-b005-44a838ba9985": failed to provision volume with StorageClass "cephfs": rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source
I0115 05:51:15.699968       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"ark", Name:"arkcluster-ark-cluster-game", UID:"f075ed11-6288-4521-b005-44a838ba9985", APIVersion:"v1", ResourceVersion:"262580", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "cephfs": rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source
I0115 05:51:16.200906       1 controller.go:1337] provision "ark/arkcluster-ark-cluster-game" class "cephfs": started
I0115 05:51:16.201137       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"ark", Name:"arkcluster-ark-cluster-game", UID:"f075ed11-6288-4521-b005-44a838ba9985", APIVersion:"v1", ResourceVersion:"262580", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "ark/arkcluster-ark-cluster-game"
W0115 05:51:16.349065       1 controller.go:934] Retrying syncing claim "f075ed11-6288-4521-b005-44a838ba9985", failure 1
E0115 05:51:16.349110       1 controller.go:957] error syncing claim "f075ed11-6288-4521-b005-44a838ba9985": failed to provision volume with StorageClass "cephfs": rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source
I0115 05:51:16.349123       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"ark", Name:"arkcluster-ark-cluster-game", UID:"f075ed11-6288-4521-b005-44a838ba9985", APIVersion:"v1", ResourceVersion:"262580", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "cephfs": rpc error: code = InvalidArgument desc = readOnly accessMode is supported only with content source
I0115 05:51:17.226012       1 controller.go:1442] provision "ark/arkcluster-ark-cluster-cluster" class "cephfs": volume "pvc-c318dd41-d7a8-48bb-a3c2-50b94bde4e89" provisioned

I'm wondering if this is an issue with cephfs pool implementation with Jaas... Or is this actually an order of operations issue as suggested by t_ume

[Dracozny]

I tested removing ROX and it provisioned. So now I'm back to the issue with chown permissions which might be more pool related. I'll have to research that, so far I'm only seeing suggestions of no_root_squash or norootexperimental as suggestions but that's for NFS typically. hopefully the lack of ROX doesn't bite me later.

[DrPsychick]

So the idea behind allowing ROX is to have game servers mount the game files read-only. Only one instance needs update permissions to update the game server files. But ofc there are other ways to do this and giving all game servers write permissions should not be a problem either, as long as not multiple instances try to update the server version at the same time.

[Dracozny]

my permission issue is really frustrating at this point I've tried changing the securityContext to get around it but it seems like no group (sudo, root, etc..) can even bypass the permissions. I created a separate pod to test the environment and the filesystem is mounted as root:root. Since Kubectl exec drops you in as root I can chmod any folder I want in that pod. though that was mounted in the default namespace so I might wanna look into that.

securityContext:
# need to adjust image...
   runAsUser: 1000
   runAsGroup: 27

pod log

Ensuring correct permissions...
changed ownership of '/home/steam/.sudo_as_admin_successful' from steam:sudo to steam:steam
Shared server files in /arkserver...
chown: changing ownership of '/arkserver/ShooterGame/Saved': Operation not permitted

[Dracozny]

/var/lib/www/html is mounted by the PVC and consumed by the replication controller under the namespace test. so It works but not from what I have tried doing in ark. ark of course won't let me run as root as it's not part of the sudoers group. not that running as root was adviseable anyways but I had to try it just to see if it would bypass the problem.

➜  ~ kubectl exec -it arkserver-vnbs6 -n test -- bash
root@arkserver-vnbs6:/# ls -la /var/lib/
apt/     dpkg/    misc/    pam/     systemd/ ucf/     www/
root@arkserver-vnbs6:/# ls -la /var/lib/www/
total 0
drwxr-xr-x 1 root root 8 Jan 16 00:22 .
drwxr-xr-x 1 root root 6 Jan 16 00:22 ..
drwxr-xr-x 2 root root 6 Jan 16 00:22 html
root@arkserver-vnbs6:/# chown -R www-data:www-data /var/lib/www/html
root@arkserver-vnbs6:/# ls -la /var/lib/www/
total 0
drwxr-xr-x 1 root     root     8 Jan 16 00:22 .
drwxr-xr-x 1 root     root     6 Jan 16 00:22 ..
drwxr-xr-x 2 www-data www-data 6 Jan 16 00:22 html

[Dracozny]

I cheated a bit. I tried to launch the first server (extinction) knowing it would fail but it completes a few things in the pvc. I'm not sure what as of yet but if I tried to mount the game pvc in another pod first, it complained about the volumemount.name so I skipped it and went for the extinction. I then could do a kubectl exec into the pod I created and chown -R 1000:1000 /arkserver , after that I deleted the pod and rescaled the extinction server and now it's updating. it did make some initial complaints but perhaps that is the process of checking for all the needed files.

So it's a bit of a hack but it works... Perhaps there should be a method in the config to switch to root for ceph users and then downgrade to the steam user. Granted sudo should do this but something isn't working there and it seems that's an issue for other helm apps as well when I do a search. Alternatively a terraforming script might be useful for establishing the claims without introducing root into the operation of starting the servers. Though if you go that route you could always auto scale each requested server if desired. I think that would be possible...

[Dracozny]

ok, it looks like it crashed on me... it also spit out the same error I saw earlier.

Ensuring correct permissions...
changed ownership of '/home/steam/.sudo_as_admin_successful' from steam:root to steam:steam
Shared server files in /arkserver...
Shared clusters files in /arkserver/ShooterGame/Saved/clusters...
Cleaning up any leftover arkmanager files...
Creating arkmanager.cfg from environment variables...
Creating crontab...
Starting cron service...
 * Starting periodic command scheduler cron
   ...done.
Loading crontab...
Save file validation is not enabled.
Backup on start is not enabled.
Running command 'start' for instance 'main'
[  WARN  ]      Your ARK server exec could not be found.
touch: cannot touch '/arkserver/.startAfterUpdate-main': Permission denied
Checking for updates before starting
Checking for update; PID: 46
sed: can't read /arkserver/steamapps/appmanifest_376030.acf: No such file or directory
The server is already stopped
Performing ARK updateExecuting /usr/games/steamcmd +@NoPromptForPassword 1 +force_install_dir /arkserver +login anonymous +app_update 376030 +quit
Redirecting stderr to '/home/steam/.steam/logs/stderr.txt'
[  0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation - version 1669935972
-- type 'quit' to exit --
Loading Steam API...OK
"@NoPromptForPassword" = "1"

Connecting anonymously to Steam Public...OK
Waiting for client config...OK

[Dracozny]

 Update state (0x61) downloading, progress: 37.09 (6981410354 / 18824280149)
 Update state (0x61) downloading, pRunning command 'broadcast' for instance 'main'
[  WARN  ]      Your ARK server exec could not be found.
Error connecting to server: Connection refused at -e line 33.

[Dracozny]

So this may be my own ignorance. I launched my pod.yaml file again to create a pod I can execute in with and the arkserver is owned by root:root. I'm not sure if it reverted in the middle of operations or if that was just from my pod mounting. If it's the former than I have an issue with ceph and likely juju as well. I noticed when I attempted to modify the rdb pools that they reverted back within an hour or so but usually anything still connected kept the settings I had placed on them.

[Dracozny]

ok while it was attempting to run I was able to exec into the pod for Extinction and sudo su. It let me chown the /arkserver directory so I'm not sure what is going on there.

[Dracozny]

so the error I am currently getting appears to be that the download is not completing. it's a bit random in places. earlier its at 37% as posted above, then it was at 27 and, just now it was at 62.5. I have a 1Gbps connection so this shouldn't be an issue normally. but perhaps the upstream connection is the cause. I guess I'll find out in the AM if it ever completes.

Update: it finished downloading but keeps dropping out during the verification step.

State:          Running
      Started:      Mon, 16 Jan 2023 06:54:52 -0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    111
      Started:      Mon, 16 Jan 2023 06:42:52 -0800
      Finished:     Mon, 16 Jan 2023 06:54:49 -0800
    Ready:          False
    Restart Count:  39

Seems like it terminates itself after a specified amount of time. Almost 12 min

Last State:     Terminated
      Reason:       Error
      Exit Code:    111
      Started:      Mon, 16 Jan 2023 07:42:52 -0800
      Finished:     Mon, 16 Jan 2023 07:54:49 -0800

[Dracozny]

I extended the initial Delay on the startup probe. But I'm wondering if there is a bigger issue at hand. this thing is so slow. I stumbled across this issue and by removing the probes they had improved performance. I know I'm not being throttled and I have tons of resources available. the node has 32 cpu threads available and it's barely using half of 1 at this point. Update: it ran longer (20 min) but still got killed. it died at 83.58% of the verification.
Update2: removed probes. still slow. running an iops test on another node to rule out storage issue

[DrPsychick]

which base OS are you using? IIRC steamcmd may not work reliably on newer distributions. But if that is the problem, I honestly cannot provide any help but to suggest trying an older distribution.

[Dracozny]

The Base OS is Jammy. I'm running with a stack of spinning disks so that could be part of the problem.

IOPS (Read/Write)               ceph-xfs   vs                cephfs                   Change
        Random:                946 / 223   vs              1,028 / 237    :       8.67% / 6.28%
    Sequential:              4,759 / 786   vs              4,778 / 694    :     0.40% / -11.70%

It's taken an hour to progress through the verification and currently @11% of the commit. I'm not sure what else to look at

[DrPsychick]

oh, ARK is heavy on the disk indeed, but I thought that would mostly affect the client, because it's so huge. I never tried to use distributed storage for any of my servers, to be honest, and none of my installations run on spinning disks. That combined with spinning disks might indeed be the main issue.

So, if you do not lack storage and want to distribute instances across multiple nodes, I guess the best approach would be to have a local disk with the game server files per nodes and bind that to the pods, somehow.

Save games and such should not be an issue with distributed storage as they are not big.

[Dracozny]

I'm switching them to SSD. I'll check back in after that process. I need to figure out if my clients are even connecting. my roommates all use the Epic version and were not able to connect. so it's ether the drives causing delays or something else is screwy on my end.

[Dracozny]

Ok, I can confirm SSD was the way to go there as far as loading and speed goes. I still had to jump in (kubectl exec -it) with a replication pod to chown the mount points to 1000:1000 in order to get the server to start though. I'm guessing this is something to do with how ceph was deployed. I'll probably have to pester the peeps at charhub.io.

The probes apparently are still too fast.

 Update state (0x81) verifying update, progress: 79.02 (14876188631 / 18825064310)
 Update state (0x81) verifying update, progress: 79.10 (14890389308 / 1882506431Running command 'broadcast' for instance 'main'
[  WARN  ]      Your ARK server exec could not be found.
Error connecting to server: Connection refused at -e line 33.

I'll have to try extending them or just disabling them.

[Dracozny]

ok. long time later. I wound up extending the startup probe delay to 1200 before it finally ran through without crashing. but still no joy...

2023-01-26 06:31:56: start
2023-01-26 06:31:56: Running /arkserver/ShooterGame/Binaries/Linux/ShooterGameServer Extinction\?GameModIds\?MaxPlayers=10\?RCONEnabled=True\?ServerAdminPassword=************\?AltSaveDirectoryName=SavedArks\?SessionName=Extinction\?QueryPort=27015\?Port=7777\?RCONPort=32330\?GameModIds\?ServerPassword\?listen -clusterid=arkcluster -log
2023-01-26 06:31:56: Server PID: 307
[S_API FAIL] SteamAPI_Init() failed; SteamAPI_IsSteamRunning() failed.
Setting breakpad minidump AppID = 346110

OK that's not actually an error from what I am reading. However it did give another error and restarted last night.

Error connecting to server: Connection refused at -e line 33.](error: stream error: stream ID 3; INTERNAL_ERROR; received from peer)

Maybe that was a roommate trying without telling me. Roommates are going to test it today so cross fingers

[Dracozny]

Closing: Primary issue is not caused by the chart and is likely upstream.

[DrPsychick]

FYI: @Dracozny we found out with @bhechinger that the securityContext fsGroup: 1000 needs to be set when using a ceph volume.

[bhechinger]

Just to be clear this probably isn't limited to Ceph but needs to be set for any block based storage.

[Dracozny]

Thanks for reaching out. The fsgroup set to 1000 is one thing I did try but found unsuccessful. I actually bailed on Ceph entirely and switched to longhorn which didn't have these startup issues at all.
Actually I went a step further and bailed on the entire stack and built from scratch using kubeadm. As nice as Juju seemed to be it's just has too many hidden gremlins. Ceph being one of the many bits I found suspect. But even calico was giving me issues when it's normally a pretty benign router.

I actually have extinction up and running now and am in the process of launching the others.
I would point out that the probes were still too short even with longhorn on 870 EVO SATA SSDs. I'm not a kubernetes guru but if there is a more statefull method of checking startup that would be a big step in the right direction. Supposedly Juju Charms are supposed to do that but that gets other gremlins vs what we have in helm charts.

[DrPsychick]

@Dracozny if you also want to join the party in slack, poke me on slack -at - drsick.net for an invite.