search

kubecost / cost-analyzer-helm-chart

Kubecost helm chart
http://kubecost.com/install
Apache License 2.0
466 stars 413 forks source link

Network costs not working correctly on Azure #122

Closed dwbrown2 closed 4 years ago

dwbrown2 commented 4 years ago

Azure builds with --set networkCosts.enabled=true are not returning data to the Kubecost frontend.

kubectl logs -n kubecost kubecost-cost-analyzer-5b88ddcfdf-bhq72 -c cost-model
I1009 17:35:10.781999       1 main.go:697] Starting cost-model (git commit "b87fd4847734ce5b3160e9b32814f8987565ebb0")
I1009 17:35:10.788581       1 main.go:724] Success: retrieved a prometheus config file from: http://kubecost-prometheus-server.kubecost
I1009 17:35:10.794467       1 main.go:730] Success: retrieved the 'up' query against prometheus at: http://kubecost-prometheus-server.kubecost
I1009 17:35:10.798041       1 provider.go:474] metadata reports we are in GCE
I1009 17:35:10.798529       1 reflector.go:122] Starting reflector *v1.StorageClass (0s) from pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
I1009 17:35:10.798544       1 reflector.go:122] Starting reflector *v1.Namespace (0s) from pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
I1009 17:35:10.798564       1 reflector.go:160] Listing and watching *v1.StorageClass from pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
I1009 17:35:10.798569       1 reflector.go:160] Listing and watching *v1.Namespace from pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
I1009 17:35:10.798616       1 reflector.go:122] Starting reflector *v1.Pod (0s) from pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
I1009 17:35:10.798628       1 reflector.go:160] Listing and watching *v1.Pod from pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
I1009 17:35:10.798786       1 reflector.go:122] Starting reflector *v1.Service (0s) from pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
I1009 17:35:10.798812       1 reflector.go:160] Listing and watching *v1.Service from pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
I1009 17:35:10.798954       1 reflector.go:122] Starting reflector *v1.Deployment (0s) from pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
I1009 17:35:10.798964       1 reflector.go:160] Listing and watching *v1.Deployment from pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
I1009 17:35:10.798964       1 reflector.go:122] Starting reflector *v1.PersistentVolume (0s) from pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
I1009 17:35:10.798973       1 reflector.go:160] Listing and watching *v1.PersistentVolume from pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
I1009 17:35:10.800342       1 reflector.go:122] Starting reflector *v1.Node (0s) from pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
I1009 17:35:10.800391       1 reflector.go:160] Listing and watching *v1.Node from pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
I1009 17:35:10.899360       1 watchcontroller.go:183] Starting *v1.Pod controller
I1009 17:35:10.899389       1 watchcontroller.go:183] Starting *v1.Node controller
I1009 17:35:10.899428       1 watchcontroller.go:183] Starting *v1.Service controller
I1009 17:35:10.899416       1 watchcontroller.go:183] Starting *v1.Deployment controller
I1009 17:35:10.899368       1 watchcontroller.go:183] Starting *v1.Namespace controller
I1009 17:35:10.899521       1 watchcontroller.go:183] Starting *v1.PersistentVolume controller
I1009 17:35:10.899661       1 watchcontroller.go:183] Starting *v1.StorageClass controller
I1009 17:35:10.921111       1 gcpprovider.go:600] Fetch GCP Billing Data from URL: https://cloudbilling.googleapis.com/v1/services/6F81-5844-456A/skus?key=AIzaSyDXQPG_MHUEy9neR7stolq6l0ujXmjJlvk
I1009 17:35:12.151146       1 gcpprovider.go:526] Adding RAM 0.004771 for us-east4,n1standard,ondemand,gpu
I1009 17:35:13.615670       1 gcpprovider.go:625] Page: map[us-east4,n1standard,ondemand:%!s(*cloud.GCPPricing=&{   <nil> [] []  0xc000b6a100 <nil>}) us-east4,n1standard,ondemand,gpu:%!s(*cloud.GCPPricing=&{   <nil> [] []  0xc000b6b300 <n
il>})] : %!v(MISSING)
I1009 17:35:13.615760       1 gcpprovider.go:627] Unmerged Page: us-east4,n1standard,ondemand : &{Name: SKUID: Description: Category:<nil> ServiceRegions:[] PricingInfo:[] ServiceProviderName: Node:0xc000b6a100 PV:<nil>}
I1009 17:35:13.615771       1 gcpprovider.go:627] Unmerged Page: us-east4,n1standard,ondemand,gpu : &{Name: SKUID: Description: Category:<nil> ServiceRegions:[] PricingInfo:[] ServiceProviderName: Node:0xc000b6b300 PV:<nil>}
I1009 17:35:13.615779       1 gcpprovider.go:625] Page: map[us-east4,pdstandard:%!s(*cloud.GCPPricing=&{services/6F81-5844-456A/skus/8AF1-1146-E7DA 8AF1-1146-E7DA Storage PD Capacity in Virginia 0xc000e7fe00 [us-east4] [0xc00081a7b0] Goog
le <nil> 0xc000e91f20})] : %!v(MISSING)
I1009 17:35:13.615786       1 gcpprovider.go:627] Unmerged Page: us-east4,pdstandard : &{Name:services/6F81-5844-456A/skus/8AF1-1146-E7DA SKUID:8AF1-1146-E7DA Description:Storage PD Capacity in Virginia Category:0xc000e7fe00 ServiceRegion
s:[us-east4] PricingInfo:[0xc00081a7b0] ServiceProviderName:Google Node:<nil> PV:0xc000e91f20}
I1009 17:35:13.615795       1 gcpprovider.go:625] Page: map[] : %!v(MISSING)
I1009 17:35:13.615803       1 gcpprovider.go:654] ALL PAGES: map[us-east4,n1standard,ondemand:0xc0009aec60 us-east4,n1standard,ondemand,gpu:0xc0009aecf0 us-east4,pdstandard:0xc000ea7560]
I1009 17:35:13.615811       1 gcpprovider.go:656] Returned Page: us-east4,n1standard,ondemand : &{Cost: VCPU: VCPUCost:0.035605000000000005 RAM: RAMBytes: RAMCost:0.004771 Storage: StorageCost: UsesBaseCPUPrice:false BaseCPUPrice: BaseRAM
Price: BaseGPUPrice: UsageType:ondemand GPU: GPUName: GPUCost:}
I1009 17:35:13.615825       1 gcpprovider.go:656] Returned Page: us-east4,n1standard,ondemand,gpu : &{Cost: VCPU: VCPUCost:0.035605000000000005 RAM: RAMBytes: RAMCost:0.004771 Storage: StorageCost: UsesBaseCPUPrice:false BaseCPUPrice: Bas
eRAMPrice: BaseGPUPrice: UsageType:ondemand GPU: GPUName: GPUCost:}
I1009 17:35:13.615835       1 gcpprovider.go:656] Returned Page: us-east4,pdstandard : <nil>
I1009 17:36:23.049491       1 costmodel.go:680] Unable to find historical data for pod 'trainmodeldatabrickscluster6modelistested-f7ff89667a4e4f9d92181ce3baf21eac'
I1009 17:39:52.549681       1 costmodel.go:680] Unable to find historical data for pod 'trainmodeldatabricksclusterterminatedatabrickscluster-bb68c9449c1546b2bebaacf17995e7ee'
I1009 17:56:10.622986       1 costmodel.go:680] Unable to find historical data for pod 'trainmodeldatabricksclusterterminatedatabrickscluster-62beece84ab040e98bdc7126eb22d03a'
I1009 18:00:51.553887       1 costmodel.go:680] Unable to find historical data for pod 'trainmodeldatabricksclustertrainmodel-e2dacfd4305b4852870dafedf45716b4'
I1009 18:09:02.689538       1 costmodel.go:680] Unable to find historical data for pod 'trainmodeldatabricksclusterfilesareuploaded-f37f0b11302c43d4b2f3bd4c71b1b5ed'
I1009 18:23:02.840923       1 costmodel.go:680] Unable to find historical data for pod 'trainmodeldatabricksclustergetfiles-257a188ac72e4083b09df0e362690e93'
I1009 18:31:13.766612       1 costmodel.go:680] Unable to find historical data for pod 'trainmodeldatabricksclusterclusteriscreated-6dcf92af71a44f9289689d080e279bc6'
I1009 18:31:13.766647       1 costmodel.go:680] Unable to find historical data for pod 'trainmodeldatabricksclustergetfiles-24aa4ca007664e47973ff052f36fc431'
I1009 18:46:33.396844       1 costmodel.go:680] Unable to find historical data for pod 'trainmodeldatabrickscluster6notifyclusterterminated-6a912e395d304c59831bbaf476a808de'
I1009 18:49:17.613433       1 main.go:78] Error returned to client: Error fetching historical pod labels: read tcp 10.32.5.42:60956->10.98.7.15:80: read: connection reset by peer
I1009 18:51:52.329483       1 main.go:78] Error returned to client: Error fetching historical pod labels: read tcp 10.32.5.42:35456->10.98.7.15:80: read: connection reset by peer
I1009 18:51:53.429411       1 main.go:78] Error returned to client: Error fetching historical pod labels: read tcp 10.32.5.42:35446->10.98.7.15:80: read: connection reset by peer
I1009 18:51:53.429465       1 main.go:78] Error returned to client: Error fetching historical pod labels: read tcp 10.32.5.42:35446->10.98.7.15:80: read: connection reset by peer
I1009 18:57:13.555632       1 costmodel.go:680] Unable to find historical data for pod 'trainmodeldatabrickscluster6filesareuploaded-85d680199d89404aafe69bbad7cd5679'
I1009 18:58:25.483973       1 costmodel.go:680] Unable to find historical data for pod 'trainmodeldatabrickscluster6modelistrained-a504bfcd448845c28c46bd9eb1649820'
I1009 19:01:55.967744       1 costmodel.go:680] Unable to find historical data for pod 'exampledagprintdate3-bf0d3a4aaa4049cb966a24390f957f34'
I1009 19:05:38.415817       1 main.go:78] Error returned to client: Error fetching historical pod labels: read tcp 10.32.5.42:49376->10.98.7.15:80: read: connection reset by peer
dwbrown2 commented 4 years ago

Looking at osImage = Ubuntu 16.04.6 LTS and the conntrack file is not in the expected location.

dwbrown2 commented 4 years ago

Issue addressed in v5 of network costs!