search

googleforgames / agones

Dedicated Game Server Hosting and Scaling for Multiplayer Games on Kubernetes
https://agones.dev
Apache License 2.0
6.09k stars 810 forks source link

Upgrade to Kubernetes 1.15 #1478

Closed roberthbailey closed 4 years ago

roberthbailey commented 4 years ago

Now that Amazon EKS Kubernetes Versions shows that Kubernetes 1.15 is now available on EKS, as per our impending policy change for which Kubernetes version to support, it's time to upgrade to Kubernetes 1.15.

List of items to do for upgrading to 1.15 (this is copied from the 1.14 issue and may need to be updated):

aLekSer commented 4 years ago

Added EKS to terraform list, since 1.15 is supported there.

roberthbailey commented 4 years ago

@heartrobotninja - do you want to do the client-go updates again this release cycle?

roberthbailey commented 4 years ago

I'll start working on the checklist this week.

/assign

heartrobotninja commented 4 years ago

Yeah, I will get started tomorrow.

On Tue, Apr 21, 2020 at 11:56 AM Robert Bailey notifications@github.com wrote:

I'll start working on the checklist this week.

/assign

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/googleforgames/agones/issues/1478#issuecomment-617350775, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABM5DZ6NB3GHBCNIOSTJ7OTRNXT6VANCNFSM4MJFCV7Q .

aLekSer commented 4 years ago

This time we also need to update HELM_VER in build/e2e-image/Dockerfile to version 2.12 as we did in build image.

heartrobotninja commented 4 years ago

Yeah, I will get started tomorrow. … On Tue, Apr 21, 2020 at 11:56 AM Robert Bailey @.***> wrote: I'll start working on the checklist this week. /assign — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1478 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABM5DZ6NB3GHBCNIOSTJ7OTRNXT6VANCNFSM4MJFCV7Q .

I've updated the client but am running into some problems which look a lot like https://github.com/kubernetes/kubernetes/issues/86666 (which seems to affect 1.16 and 1.17, but I am seeing this with 1.15.11)

markmandel commented 4 years ago

I've updated the client but am running into some problems which look a lot like kubernetes/kubernetes#86666 (which seems to affect 1.16 and 1.17, but I am seeing this with 1.15.11)

I've had namespaces get stuck because of finalisers on GameServer objects not yet being removed (often because of the Agones controller not existing). Is it akin to that, or are you getting some kind of error message @heartrobotninja ?

markmandel commented 4 years ago

Looks like the deployment scripts for the e2e tests have updated to 1.15 -- any reason I shouldn't upgrade the e2e cluster?

markmandel commented 4 years ago

I started doing some work on upgrading the client to 1.15 support, since I also had to update the script for generating the crd clients to make it work.

https://github.com/markmandel/agones/tree/feature/1.15-client

Unit tests all pass except for TestRoundTripTypes, which returns all kind of errors along the lines of:

=== RUN   TestRoundTripTypes/agones.dev.v1.Fleet
    --- FAIL: TestRoundTripTypes/agones.dev.v1.Fleet (0.03s)
        roundtrip.go:290: Fleet: object *v1.Fleet does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message ((*v1.Fleet){TypeMeta:(v1.TypeMeta){Kind:(string)Fleet APIVersion:(string)agones.dev/v1} ObjectMeta:(v1.ObjectMeta){Name:(string)ȢWK襐嬆T)9l GenerateName:(string)?zOiŐRþċĨA Namespace:(string)-趐杤/策ÌƑŦǛ砐燙拍Ku缨爻抩矙x SelfLink:(string)镃晎Ŷ素鹌¼ƆƗ齀 UID:(types.UID)Ó/ȗ唣bÇą枱m ResourceVersion:(string)2781691563462094258 Generation:(int64)-9043190328892808838 CreationTimestamp:(v1.Time){Time:(time.Time){wall:(uint64)0 ext:(int64)0 loc:(*time.Location)<nil>}} DeletionTimestamp:(*v1.Time)<nil> DeletionGracePeriodSeconds:(*int64)<nil> Labels:(map[string]string)<nil> Annotations:(map[string]string)map[遢b覰唆ɷ捆+ɏʆ:Ɓ弪祴ȕ廌抨zȞ蝱7痡E骬=!癙ƫ馏] OwnerReferences:([]v1.OwnerReference)<nil> Initializers:(*v1.Initializers)<nil> Finalizers:([]string)<nil> ClusterName:(string)ʪɅC ManagedFields:([]v1.ManagedFieldsEntry)<nil>} Spec:(v1.FleetSpec){Replicas:(int32)753588366 Strategy:(v1.DeploymentStrategy){Type:(v1.DeploymentStrategyType)伭ƙǢɠ軼廡s"B=Ĉc坍ɒz釙0ąɉ RollingUpdate:(*v1.RollingUpdateDeployment)<nil>} Scheduling:(apis.SchedulingStrategy)Ȣ匄鈴iưǗ胿岵ȣ Template:(v1.GameServerTemplateSpec){ObjectMeta:(v1.ObjectMeta){Name:(string)ǁ髱 GenerateName:(string)ox>,N sʙ窽BJĈ Namespace:(string)ȑ钥挻ŢơƆƜFǍ& SelfLink:(string)]跁í^住柦 UID:(types.UID)5R愑扞ąK4丛畄%Ȓ ResourceVersion:(string)12715234379899790373 Generation:(int64)5939464765547000371 CreationTimestamp:(v1.Time){Time:(time.Time){wall:(uint64)0 ext:(int64)0 loc:(*time.Location)<nil>}} DeletionTimestamp:(*v1.Time)<nil> DeletionGracePeriodSeconds:(*int64)5751301764914840917 Labels:(map[string]string)<nil> Annotations:(map[string]string)<nil> OwnerReferences:([]v1.OwnerReference)[{APIVersion:(string)!ǫ塘ɴ惦xhaʆc9ÊĪďŨ Kind:(string)礡ƃōŋ嬻顉ȧģ ǹƽf鿸颚ʔ3 Name:(string) UID:(types.UID)綫/WƩ6DŽśŌ蓍 Controller:(*bool)<nil> BlockOwnerDeletion:(*bool)<nil>}] Initializers:(*v1.Initializers)<nil> Finalizers:([]string)<nil> ClusterName:(string)õ婕ɾ絲濼亪łȯƑÍ経0眒嚾濐>襝湘 ManagedFields:([]v1.ManagedFieldsEntry)<nil>} Spec:(v1.GameServerSpec){Container:(string)ǐʧ翨幀鶈釽 Ports:([]v1.GameServerPort)[{Name:(string)$Â5 PortPolicy:(v1.PortPolicy)ɀW親>}藄Ɣ葟,災Ŏ霁銝ć烡ɠ<f朒 Container:(*string)Â皰-(4?Tǒɘʃʋƅ%Ƴ峸ĝ逛Ǭ9 ContainerPort:(int32)90988789 HostPort:(int32)849540502 Protocol:(v1.Protocol)Ƭ力菖层ǥþÔ\ş斫Q}] Health:(v1.Health){Disabled:(bool)false PeriodSeconds:(int32)445992989 FailureThreshold:(int32)-1491292646 InitialDelaySeconds:(int32)265658544} Scheduling:(apis.SchedulingStrategy)ǟ嬚Ɣ躮ɘȧB淣搗[¨Q匞BHhe` SdkServer:(v1.SdkServer){LogLevel:(v1.SdkServerLogLevel)h呉賰此 GRPCPort:(int32)-657222859 HTTPPort:(int32)-1889753336} Template:(v1.PodTemplateSpec){ObjectMeta:(v1.ObjectMeta){Name:(string)辍q兺埡$U&}xɟF GenerateName:(string)彀ɟƔ Namespace:(string) SelfLink:(string)LƓñ廻?VuǻÚĨ郲S婬$ UID:(types.UID)ºʕ錒 ResourceVersion:(string)13725770052953848551 Generation:(int64)7926794231339589236 CreationTimestamp:(v1.Time){Time:(time.Time){wall:(uint64)0 ext:(int64)0 loc:(*time.Location)<nil>}} DeletionTimestamp:(*v1.Time)<nil> DeletionGracePeriodSeconds:(*int64)<nil> Labels:(map[string]string)map[鐝ÏeȢ:e趬Ï發虀ƣüǼ] Annotations:(map[string]string)<nil> OwnerReferences:([]v1.OwnerReference)[{APIVersion:(string)J應·ŷ爘<ɻ8磂:,B Kind:(string)珌¦o稊ưɏʃŒ垍ʏ­ƛõĉ攆Ƙ劶 Name:(string) UID:(types.UID)蝒dʊÇ85 Controller:(*bool)true BlockOwnerDeletion:(*bool)true}] Initializers:(*v1.Initializers)<nil> Finalizers:([]string)[{żȢ] ClusterName:(string)D¥袧þ-啑*>伏§知Ɔŗ脡rȯfņ ManagedFields:([]v1.ManagedFieldsEntry)<nil>} Spec:(v1.PodSpec){Volumes:([]v1.Volume)[{Name:(string)奯Ş邛ɦĚS窕+Ī廒哳鸲傧ƻ惿ʅ VolumeSource:(v1.VolumeSource){HostPath:(*v1.HostPathVolumeSource){Path:(string) Type:(*v1.HostPathType)<nil>} EmptyDir:(*v1.EmptyDirVolumeSource)<nil> GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)<nil> AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource){VolumeID:(string)銽湣繨YJȨƮ FSType:(string)脑榡Ł嶘湆ư4Hþ瓹蛰 Partition:(int32)-1010304434 ReadOnly:(bool)true} GitRepo:(*v1.GitRepoVolumeSource)<nil> Secret:(*v1.SecretVolumeSource)<nil> NFS:(*v1.NFSVolumeSource){Server:(string)ʧt:}ɇ鰳[|Ɏ Path:(string)ƟHǼŶ寝`6XŤǴ課歫ʪɭ浬猸Jŵ驡 ReadOnly:(bool)true} ISCSI:(*v1.ISCSIVolumeSource){TargetPortal:(string)řê权y櫧埡砷±謊.5舒Ȝf濏 IQN:(string)ĄŰ KZ袡3餙烜拮vŸ Lun:(int32)144375628 ISCSIInterface:(string)0 FSType:(string)裈篃ƮR[âư薡譞Jz_ǩa濴龙ǟœi ReadOnly:(bool)true Portals:([]string)<nil> DiscoveryCHAPAuth:(bool)false SessionCHAPAuth:(bool)false SecretRef:(*v1.LocalObjectReference){Name:(string)柘穤囚9U擺ȋ羱ʅcl茐<} InitiatorName:(*string)<nil>} Glusterfs:(*v1.GlusterfsVolumeSource){EndpointsName:(string)GwĮ村ĺŝ. Path:(string)齎j ReadOnly:(bool)false} PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)<nil> RBD:(*v1.RBDVolumeSource){CephMonitors:([]string)<nil> RBDImage:(string)ȵ\VũM輩ɲY FSType:(string)ȼ愪 RBDPool:(string)ň劈揪皖鼶赻äúƆȼ窖 RadosUser:(string)砧y*蚖ȓ鳶玆=接 Keyring:(string)聏z SecretRef:(*v1.LocalObjectReference){Name:(string)4Ň¡¼1炒qá} ReadOnly:(bool)true} FlexVolume:(*v1.FlexVolumeSource){Driver:(string)fc緘ų撐論Q2%Ǖ颵ƱB FSType:(string)ȉ#őB³裀ɁĈ遛骐緬ZLʧ,_316 SecretRef:(*v1.LocalObjectReference){Name:(string)毚亰x僕$壌薉Š4,詖聺明赣lj趁ǐ} ReadOnly:(bool)true Options:(map[string]string)map[]} Cinder:(*v1.CinderVolumeSource){VolumeID:(string)/y}埩墍崿 FSType:(string)¸ q趄 ReadOnly:(bool)true SecretRef:(*v1.LocalObjectReference){Name:(string)Z{ʫk辂Ē}} CephFS:(*v1.CephFSVolumeSource)<nil> Flocker:(*v1.FlockerVolumeSource)<nil> DownwardAPI:(*v1.DownwardAPIVolumeSource){Items:([]v1.DownwardAPIVolumeFile)[{Path:(string)x觘Į蹕朦?&ɦėȃ辂T鞚 FieldRef:(*v1.ObjectFieldSelector)<nil> ResourceFieldRef:(*v1.ResourceFieldSelector)<nil> Mode:(*int32)1487896854}] DefaultMode:(*int32)<nil>} FC:(*v1.FCVolumeSource){TargetWWNs:([]string)<nil> Lun:(*int32)-843635935 FSType:(string)坻ĈŠ忥HR赗Ȓʢ蟫宾軵ʚźǞo ReadOnly:(bool)true WWIDs:([]string)<nil>} AzureFile:(*v1.AzureFileVolumeSource)<nil> ConfigMap:(*v1.ConfigMapVolumeSource)<nil> VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)<nil> Quobyte:(*v1.QuobyteVolumeSource){Registry:(string)Bþ鰗Û稒軃賑Ŀȼ鏈淿怔渏R[蟺 Volume:(string)Ƭɡíij_ 象 ReadOnly:(bool)true User:(string)Ơt冦Ƚ Group:(string)侜ȱ揨%O6Eǖ鷁纎駺2¬ę Tenant:(string)e嚘ȫɂ<ȊƁ} AzureDisk:(*v1.AzureDiskVolumeSource)<nil> PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)<nil> Projected:(*v1.ProjectedVolumeSource)<nil> PortworxVolume:(*v1.PortworxVolumeSource){VolumeID:(string)ʭcȨÝ FSType:(string)暤iW犏购ģɰ铕貛藿ǐ懧网 ReadOnly:(bool)false} ScaleIO:(*v1.ScaleIOVolumeSource)<nil> StorageOS:(*v1.StorageOSVolumeSource)<nil> CSI:(*v1.CSIVolumeSource){Driver:(string)舠聵蟏Ťʣ旱冐:ʦNd嶈2 ReadOnly:(*bool)false FSType:(*string) VolumeAttributes:(map[string]string)<nil> NodePublishSecretRef:(*v1.LocalObjectReference)<nil>}}}] InitContainers:([]v1.Container)<nil> Containers:([]v1.Container)[] RestartPolicy:(v1.RestartPolicy)Ɩ咸雠Ǫo<鹝濘5ßj弧鸿ŧ TerminationGracePeriodSeconds:(*int64)<nil> ActiveDeadlineSeconds:(*int64)-7738042456879931615 DNSPolicy:(v1.DNSPolicy)#鷨ơ汑Mɓ靛醿ƗƏ刣鏹/Ǹ疪Ǯ筸 NodeSelector:(map[string]string)<nil> ServiceAccountName:(string)İN DeprecatedServiceAccount:(string)挎x蔻,ŗ9绐ǯ孓ƭƏ AutomountServiceAccountToken:(*bool)<nil> NodeName:(string)<u脂鐒瀟l8ʓ HostNetwork:(bool)false HostPID:(bool)true HostIPC:(bool)false ShareProcessNamespace:(*bool)true SecurityContext:(*v1.PodSecurityContext)<nil> ImagePullSecrets:([]v1.LocalObjectReference)[] Hostname:(string)a§k&ōw臛嬔g藋片聦I埂!ƛ郏òœ Subdomain:(string) Affinity:(*v1.Affinity)<nil> SchedulerName:(string)浲zȷaʠ$楶ȩ憚¦齔Aʧǂ7ŀ Tolerations:([]v1.Toleration)<nil> HostAliases:([]v1.HostAlias)<nil> PriorityClassName:(string)Q+ Priority:(*int32)<nil> DNSConfig:(*v1.PodDNSConfig){Nameservers:([]string)<nil> Searches:([]string)[P] Options:([]v1.PodDNSConfigOption)<nil>} ReadinessGates:([]v1.PodReadinessGate)[{ConditionType:(v1.PodConditionType)樋T鳼îp加誖YNÇ埡戼Š肇ŁȀȱ}] RuntimeClassName:(*string)ǙdŬ!³ EnableServiceLinks:(*bool)<nil> PreemptionPolicy:(*v1.PreemptionPolicy)ʉß倧M&ȁō耗ƨ\ǘ}} Players:(*v1.PlayersSpec)<nil>}}} Status:(v1.FleetStatus){Replicas:(int32)2023974014 ReadyReplicas:(int32)-276248142 ReservedReplicas:(int32)128097702 AllocatedReplicas:(int32)455445990}})

First few e2e tests I've tried, all looks good, but this is the only blocking issue I've run into so far.

@aLekSer any ideas on the fuzzing test? @heartrobotninja is this what you ran into before?

aLekSer commented 4 years ago

In this PR https://github.com/kubernetes/kubernetes/pull/78309/files last skipProtobuf parameter becomes false

roundTripSpecificKind(t, gvk, scheme, codecFactory, fuzzer, nonRoundTrippableTypes, false)

https://github.com/kubernetes/apimachinery/commit/3610fc3a2147fe91751fa4bed0d896d088017fcf I have found a method RoundTripExternalTypesWithoutProtobuf() which is a workaround, but it is not in v0.15.1 https://github.com/kubernetes/apimachinery/commit/f059d0b52cb1813433acbad3889c2e90c155887f Original PR is here https://github.com/kubernetes/kubernetes/pull/86959

aLekSer commented 4 years ago

As a workaround, there is an option to use the code similar to what was used in Kubernetes 1.14:

    kinds := scheme.AllKnownTypes()
    for gvk := range kinds {
        if gvk.Version == runtime.APIVersionInternal || globalNonRoundTrippableTypes.Has(gvk.Kind) {
            continue
        }
        t.Run(gvk.Group+"."+gvk.Version+"."+gvk.Kind, func(t *testing.T) {
            roundtrip.RoundTripSpecificKindWithoutProtobuf(t, gvk, scheme, codecs, localFuzzer, nil)
        })
    }

instead of RoundTripExternalTypes(...). Also I will compare to this Roundtrip test and structures: https://github.com/kubernetes/api/blob/master/roundtrip_test.go

aLekSer commented 4 years ago

By the way with a new version there is a possibility to add Compatibility test, but need to add json files in testdata directory:

func TestCompatibility(t *testing.T) {
    scheme := runtime.NewScheme()
    localSchemeBuilder := runtime.SchemeBuilder{
        agonesv1.AddToScheme,
        allocationv1.AddToScheme,
        autoscalingv1.AddToScheme,
        multiclusterv1.AddToScheme,
    }
    assert.NoError(t, localSchemeBuilder.AddToScheme(scheme))
    roundtrip.NewCompatibilityTestOptions(scheme).Complete(t).Run(t)
}
markmandel commented 4 years ago

Just highlighting a risk -- if the plan was to get this in to the next release, RC is next Tuesday.

markmandel commented 4 years ago

Since we're moving to 1.16, should we close this ticket?

heartrobotninja commented 4 years ago

Yep. I think that makes sense.

On Tue, Jul 14, 2020, 10:15 AM Mark Mandel notifications@github.com wrote:

Since we're moving to 1.16, should we close this ticket?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/googleforgames/agones/issues/1478#issuecomment-658303905, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABM5DZ6YZINXKS6YHAA57R3R3SHCDANCNFSM4MJFCV7Q .

markmandel commented 4 years ago

Seems like we have consensus, I'll close this ticket then!