Cluster issue - everything works except character upload

[marcel-dempers]

Good day all,

I've been deep diving this for a number of days now and it's doing my head in.

I've setup a cluster with Island and SE with arkmanager. Tried to keep it as simple as possible to get minimalistic cluster working. I have a central arkmanager.cfg

arkstChannel=master                                                 # change it to a different branch to get non-stable versions
install_bindir="/usr/bin"
install_libexecdir="/usr/libexec/arkmanager"
install_datadir="/usr/share/arkmanager"

# config SteamCMD
steamcmdroot="/home/steam/steamcmd"                                 # path of your steamcmd instance
steamcmdexec="steamcmd.sh"                                          # name of steamcmd executable
steamcmd_user="steam"                                               # name of the system user who own steamcmd folder
steamcmd_appinfocache="/home/steam/Steam/appcache/appinfo.vdf"      # cache of the appinfo command

# config environment
arkserverexec="ShooterGame/Binaries/Linux/ShooterGameServer"        # name of ARK server executable
arkautorestartfile="ShooterGame/Saved/.autorestart"                 # path to autorestart file

# config Service
servicename="arkserv"                                               # Name of the service (don't change if you don't know what are you doing)
logdir="/ark/log/"                                                  # Logs path (default /var/log/arktools)

# steamdb specific
appid=376030                                                        # Linux server App ID
mod_appid=346110                                                    # App ID for mods

# Need to be true to work with UPDATEPONSTART (See #10)
arkAutoUpdateOnStart="false"                                        # set this to true if you want to always update before startup

# We don't use the dots because it doesn't show.
progressDisplayType=spinner

ark_PreventDownloadSurvivors=False
ark_PreventDownloadItems=False
ark_PreventDownloadDinos=False
ark_PreventUploadSurvivors=False
ark_PreventUploadDinos=False
ark_PreventUploadItems=False
ark_noTributeDownloads=False
arkopt_clusterid=${CLUSTER_ID}
arkflag_NoTransferFromFiltering=True
arkopt_ClusterDirOverride=/ark/clusters
#arkflag_exclusivejoin=true
ark_ServerPassword=${SERVERPASSWORD}
ark_ServerAdminPassword=${ADMINPASSWORD}
ark_MaxPlayers=${MAX_PLAYERS}
arkflag_log=""
mod_branch=Windows

msgWarnUpdateMinutes="This ARK server will shutdown for an update in %d minutes"
msgWarnUpdateSeconds="This ARK server will shutdown for an update in %d seconds"
msgWarnRestartMinutes="This ARK server will shutdown for a restart in %d minutes"
msgWarnRestartSeconds="This ARK server will shutdown for a restart in %d seconds"
msgWarnShutdownMinutes="This ARK server will shutdown in %d minutes"
msgWarnShutdownSeconds="This ARK server will shutdown in %d seconds"
arkwarnminutes="60"                                                 # number of minutes to warn players when using update --warn
arkBackupPreUpdate="false"                                          # set this to true if you want to perform a backup before updating
arkMaxBackupSizeMB="500"                                            # Set to automatically remove old backups when size exceeds this limit

I have my island instance config:

arkserverroot="/ark/island"                   
arkbackupdir="/ark/backup/island"               
arkStagingDir="/ark/staging/island"  
# Port config 
ark_Port=30778                               
ark_QueryPort=30015                            
ark_RCONEnabled="True"                      
ark_RCONPort="32330"
ark_AltSaveDirectoryName=ArkTheIslandSave
serverMap=TheIsland
ark_SessionName=ThatDevOpsArkIsland

I also have SE instance config:

arkserverroot="/ark/scorchedearth"                 
arkbackupdir="/ark/backup/scorchedearth"                
arkStagingDir="/ark/staging/scorchedearth"  
# Port config 
ark_Port=30780                                       
ark_QueryPort=30017                                   
ark_RCONEnabled="True"                                
ark_RCONPort="32331"
ark_AltSaveDirectoryName=ArkScorchedEarthSave
serverMap=ScorchedEarth_P
ark_SessionName=ThatDevOpsArkScorchedEarth

I use arkmanager start , both instances start up. I can connect to both individually I can view each arkmanager status @all it it looks good.

For testing I keep server pass blank and also keep exclusive join off for testing.

The problem: I can log into the Island, tame a creature, gather items, head over to an obelisk, transfer the creature and items, but cannot upload a character. The feature to upload character simply does not appear on the interface and there is so many reports of players giving mix of information. Some saying the button is no longer in game, others saying its moved. I do see a button Travel to another server, however, that button never lists any of my clustered servers no matter what I do.

In the past (from my memory) I've always uploaded character,item,dinos , logged into the other server and downloaded everything, so from my memory, that should work but I cannot see the button anymore.

Here are my arkserver.log too:

2021-01-19 04:03:00: [arkmanager-island] 2021-01-19 04:03:00: Running /ark/island/ShooterGame/Binaries/Linux/ShooterGameServer TheIsland\?Port=30777\?QueryPort=30015\?RCONEnabled=True\?RCONPort=32330\?AltSaveDirectoryName=ArkTheIslandSave\?MaxPlayers=70\?PreventDownloadDinos=False\?PreventDownloadItems=False\?PreventDownloadSurvivors=False\?PreventUploadDinos=False\?PreventUploadItems=False\?PreventUploadSurvivors=False\?ServerAdminPassword=test\?ServerPassword\?SessionName=ThatDevOpsArkIsland\?noTributeDownloads=False\?listen -NoTransferFromFiltering -log -ClusterDirOverride=/ark/clusters -clusterid=testcluster
2021-01-19 04:03:00: [arkmanager-island] 2021-01-19 04:03:00: Server PID: 73
2021-01-19 04:03:00: [arkmanager-se] 2021-01-19 04:03:00: start
2021-01-19 04:03:00: [arkmanager-se] 2021-01-19 04:03:00: Running /ark/scorchedearth/ShooterGame/Binaries/Linux/ShooterGameServer ScorchedEarth_P\?Port=30779\?QueryPort=30017\?RCONEnabled=True\?RCONPort=32331\?AltSaveDirectoryName=ArkScorchedEarthSave\?MaxPlayers=70\?PreventDownloadDinos=False\?PreventDownloadItems=False\?PreventDownloadSurvivors=False\?PreventUploadDinos=False\?PreventUploadItems=False\?PreventUploadSurvivors=False\?ServerAdminPassword=test\?ServerPassword\?SessionName=ThatDevOpsArkScorchedEarth\?noTributeDownloads=False\?listen -NoTransferFromFiltering -log -ClusterDirOverride=/ark/clusters -clusterid=testcluster
2021-01-19 04:03:00: [arkmanager-se] 2021-01-19 04:03:00: Server PID: 107
2021-01-19 04:03:00: [arkmanager-se] [S_API FAIL] SteamAPI_Init() failed; SteamAPI_IsSteamRunning() failed.
2021-01-19 04:03:00: [arkmanager-island] [S_API FAIL] SteamAPI_Init() failed; SteamAPI_IsSteamRunning() failed.
2021-01-19 04:03:00: [arkmanager-island] Setting breakpad minidump AppID = 346110
2021-01-19 04:03:00: [arkmanager-se] Setting breakpad minidump AppID = 346110
2021-01-19 04:04:05: [arkmanager-se] 2021-01-19 04:04:05: server is up
2021-01-19 04:04:21: [arkmanager-island] 2021-01-19 04:04:21: server is up

Above looks good to me, and i can see files in the /ark/clusters folder. Since Dino + item xfers work.

Any ideas why I cannot upload character data ?

Thanks a million in advance

[klightspeed]

I presume you're using the same $CLUSTER_ID for both servers? Also make sure you are using a globally unique cluster ID, and not a common one.

The Travel to another server option showing up suggests that the server detects it is in a cluster, and being able to transfer dinos and items suggests that the server is able to write to the cluster directory, but the lack of other servers suggests that the client can't communicate with both servers simultaneously.

Edit: https://github.com/arkmanager/ark-server-tools/issues/927#issuecomment-615118437 also suggests that the servers need to be able to communicate with each other over the network.

[marcel-dempers]

Hey @klightspeed ,

Thanks for your response and effort I am using the same ID on both yea. As above, I used "testcluster" which is a bit generic. I've changed it to a more globally unique one as suggested but no luck.

I've noticed the cluster directory is created, I can xfer both creatures and items and can see files generated in that folder when I do so. So it appears clustering is partially working. I can log into both Island and SE and access dinos and items successfully.

I've tried various things, start both instances, wait 15min, attempt to travel between server, restart, wait 15min and try again but no luck. Seems like the two servers cannot see each other in the "session on cluster" list. I've seen numerous forums about the game filters which i've played with quite a bit but no luck.

My ark manager is running locally so both servers and the game i test locally. Interesting observations is I cannot see my server on "unofficial" or "LAN". I cannot find my servers in the ark game, unless I go to Steam | Servers | Add server , add them with either localhost or public-ip:port | Add server to favourite In game I go to either "Favourites" or "My survivors" then I can see the servers and join either of them successfully. So I can join via local host or the public IP address with no issues.

I have the entire port range open on my router as well as windows firewall (TCP + UDP) From a network point of view, everything seems wide open and accessible so i feel like its a configuration or a game client issue to be honest.

Hope this info helps :)

[Romaq]

https://steamcommunity.com/app/346110/discussions/0/3114772813480349435/ is where I did the same thing, and where I spend time grousing about how WildCard totally botched Ark Server, particularly with regard to clustering. WAY too late in Ark 1 for this to ever see a fix, but I hope to hell for Ark 2 they change their thinking about the abstraction layer and make what we are trying to do more "Genesis 1", easier to navigate and less "OMG WTF LOL" the pain we are suffering.

That said, my link might help you solve some of what you are seeing. In my base, I'm using one user account "steam" with the instances in /home/steam/ARK/ and I have the actual data being saved to /home/steam/copper (my cluster name). I also have the following symlinks that have arkmanager put all the data where I want it:

In the default instance (Ragnarok is the "base" map we are using)... ~/ARK/ragnarok/ShooterGame/Saved/copper -> /home/steam/copper ~/ARK/ragnarok/ShooterGame/Saved/clusters/copper -> /home/steam/copper

Why are both of those links to ~/copper needed? I don't know, but that is where "copper cluster" data is written and read from, and this might be the fix to your trouble. Maybe. I hope. Have I mentioned how bass-ackwards WildCard was on their abstraction layer for the server, particularly in regards to clustering?

~/ARK/ragnarok/ShooterGame/Binaries/Linux/PlayerExclusiveJoinList.txt -> /home/steam/copper/PlayerExclusiveJoinList.txt So when I make backups of the cluster data, I also make a backup of my ACCESS list file who's name and location is so damned obtuse have I mentioned lately how WildCard botched the server, particularly with respect to clustering?

For all non-default instances, I have added the following: ~/ARK//ShooterGame/Content -> /home/steam/ARK/ragnarok/ShooterGame/Content

I was going to VERY quickly fill up my drive with a dozen copies of "Content", all intended to be identical. Now I just update the default Ragnarok with ALL the servers stopped, and the problem is solved. I also only have to backup ~/copper and I've saved a snapshot of the entire cluster, including the ACCESS file who's name escapes me and I can't find it when I needed it IN EVERY DAMNED INSTANCE.

If I'm missing anything or didn't consider problems that would arise from how I solved this issue, I'd really like to know what can be done to resolve the problem. Hopefully this helps @marcel-dempers and anyone else trying to do clustering with WildCard's horrid piece of ... work that's the Ark Server/ clustering.

Edit: to, too, two... whooo!