search

uber / peloton

Unified Resource Scheduler to co-schedule mixed types of workloads such as batch, stateless and stateful jobs in a single cluster for better resource utilization.
Apache License 2.0
643 stars 65 forks source link

[Resource manager] Invalid memory address or nil pointer dereference #14

Open TommyLike opened 5 years ago

TommyLike commented 5 years ago

Operation:

  1. run up minicluster with command: 
    PELOTON=app make minicluster
  2. check container status 
    ➜  peloton git:(master) PELOTON=app make minicluster
PELOTON=app ./scripts/minicluster.sh setup
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
tools/minicluster/main.py:61: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(f)
started container peloton-cassandra
failed to create cassandra store, retrying...
failed to create cassandra store, retrying...
cassandra store is created
started container peloton-zk
sleep 20 secs for zk to come up
started container peloton-mesos-master
started container peloton-mesos-agent0
started container peloton-mesos-agent1
started container peloton-mesos-agent2
docker image "uber/peloton" has to be built first locally by running IMAGE=uber/peloton make docker
app peloton-hostmgr0 is not up yet, retrying...
app peloton-hostmgr0 is not up yet, retrying...
app peloton-hostmgr0 is not up yet, retrying...
started peloton-hostmgr0
app peloton-resmgr0 is not up yet, retrying...
started peloton-resmgr0
app peloton-placement0 is not up yet, retrying...
started peloton-placement0
app peloton-placement1 is not up yet, retrying...
started peloton-placement1
app peloton-jobmgr0 is not up yet, retrying...
started peloton-jobmgr0
app peloton-aurorabridge0 is not up yet, retrying...
started peloton-aurorabridge0
➜  peloton git:(master) docker ps
CONTAINER ID        IMAGE                               COMMAND                  CREATED              STATUS              PORTS                                                                     NAMES
9409deaf5e49        uber/peloton                        "/bin/entrypoint.sh"     25 seconds ago       Up 24 seconds       0.0.0.0:5396->5396/tcp, 5290-5293/tcp, 0.0.0.0:8282->8282/tcp             peloton-aurorabridge0
dfa10c61cfeb        uber/peloton                        "/bin/entrypoint.sh"     30 seconds ago       Up 29 seconds       5290-5291/tcp, 0.0.0.0:5292->5292/tcp, 5293/tcp, 0.0.0.0:5392->5392/tcp   peloton-jobmgr0
f07c49345fc4        uber/peloton                        "/bin/entrypoint.sh"     36 seconds ago       Up 35 seconds       0.0.0.0:5303->5303/tcp, 5290-5293/tcp, 0.0.0.0:5403->5403/tcp             peloton-placement1
b0962774e39c        uber/peloton                        "/bin/entrypoint.sh"     42 seconds ago       Up 41 seconds       0.0.0.0:5293->5293/tcp, 5290-5292/tcp, 0.0.0.0:5393->5393/tcp             peloton-placement0
7c4ab56fe895        uber/peloton                        "/bin/entrypoint.sh"     48 seconds ago       Up 46 seconds       0.0.0.0:5290->5290/tcp, 0.0.0.0:5394->5394/tcp, 5291-5293/tcp             peloton-resmgr0
ddffb662e05b        uber/peloton                        "/bin/entrypoint.sh"     About a minute ago   Up About a minute   5290/tcp, 0.0.0.0:5291->5291/tcp, 5292-5293/tcp, 0.0.0.0:5391->5391/tcp   peloton-hostmgr0
b312e9107c1f        mesosphere/mesos-slave:1.6.0-rc1    "bash /files/run_mes…"   About a minute ago   Up About a minute   0.0.0.0:5053->5051/tcp                                                    peloton-mesos-agent2
c84a01d652f7        mesosphere/mesos-slave:1.6.0-rc1    "bash /files/run_mes…"   About a minute ago   Up About a minute   0.0.0.0:5052->5051/tcp                                                    peloton-mesos-agent1
6f321ccfb488        mesosphere/mesos-slave:1.6.0-rc1    "bash /files/run_mes…"   About a minute ago   Up About a minute   0.0.0.0:5051->5051/tcp                                                    peloton-mesos-agent0
354ae443d9e5        mesosphere/mesos-master:1.6.0-rc1   "bash /files/run_mes…"   About a minute ago   Up About a minute   0.0.0.0:5050->5050/tcp                                                    peloton-mesos-master
177c941daeb4        zookeeper:3.4                       "/docker-entrypoint.…"   About a minute ago   Up About a minute   2888/tcp, 3888/tcp, 0.0.0.0:8192->2181/tcp                                peloton-zk
1ca9583ad784        cassandra:3.0.14                    "bash /files/run_cas…"   About a minute ago   Up About a minute   7000-7001/tcp, 0.0.0.0:9042->9042/tcp, 7199/tcp, 0.0.0.0:9160->9160/tcp   peloton-cassandra
  3. create resource pool according to [tutorial] with yaml (https://github.com/uber/peloton/blob/master/docs/tutorial.md) 
    name: HelloWorldPool
owningteam: MyTeam
ldapgroups:
- MyGroup
description: "My first resource pool on Peloton"
resources:
- kind: cpu
reservation: 3
limit: 6
share: 1
- kind: memory
reservation: 2048
limit: 4096
share: 1
  4. resource manager container exited with logs Related logs: 
    
{"level":"info","msg":"LookupResourcePoolID called","request":{"path":{"value":"/"}},"time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"CreateResourcePool called","request":{"config":{"name":"HelloWorldPool","owningTeam":"MyTeam","ldapGroups":["MyGroup"],"description":"My first resource pool on Peloton","resources":[{"kind":"cpu","reservation":3,"limit":6,"share":1},{"kind":"memory","reservation":2048,"limit":4096,"share":1}],"parent":{"value":"root"}}},"time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Scheduling policy is not set, use default PriorityFIFO","time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"siblings to check","name":"HelloWorldPool","siblingNames":{},"time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"a8006272-9181-4529-b4b0-feccf339f654","slack_limit":{"CPU":0,"MEMORY":409.6,"DISK":0,"GPU":0},"time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":3,"MEMORY":2048,"DISK":0,"GPU":0},"respool_id":"a8006272-9181-4529-b4b0-feccf339f654","time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-06-18T08:39:35Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-06-18T08:39:35Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-06-18T08:39:35Z"}
{"demand_cap_by_limit":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","demand_cap_by_reservation_assignment":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","demand_not_satisfied":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"First pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":3,"limit":6,"share":1},"memory":{"kind":"memory","reservation":2048,"limit":4096,"share":1}},"slack_entitlement":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-06-18T08:39:35Z","total_share":{}}
{"final_assignment":{"CPU":6,"MEMORY":0,"DISK":0,"GPU":0},"level":"info","msg":"Third pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":3,"limit":6,"share":1},"memory":{"kind":"memory","reservation":2048,"limit":4096,"share":1}},"time":"2019-06-18T08:39:35Z"}
{"final_assignment":{"CPU":6,"MEMORY":4096,"DISK":0,"GPU":0},"level":"info","msg":"Third pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":3,"limit":6,"share":1},"memory":{"kind":"memory","reservation":2048,"limit":4096,"share":1}},"time":"2019-06-18T08:39:35Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xbac011]

goroutine 611 [running]: github.com/uber/peloton/resmgr/entitlement.(Calculator).getChildShare(0xc000683b60, 0x1011aa0, 0xc0005f2360, 0xe90cf3, 0x4, 0x3ff0000000000000) /go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:182 +0xd1 github.com/uber/peloton/resmgr/entitlement.(Calculator).distributeUnclaimedResources(0xc000683b60, 0x1011aa0, 0xc0005f2360, 0xc001cc38e8, 0xc001cc3a30) /go/src/github.com/uber/peloton/resmgr/entitlement/non_revocable.go:320 +0x189 github.com/uber/peloton/resmgr/entitlement.(Calculator).setEntitlementForChildren(0xc000683b60, 0x1011aa0, 0xc0005f2360) /go/src/github.com/uber/peloton/resmgr/entitlement/non_revocable.go:75 +0x572 github.com/uber/peloton/resmgr/entitlement.(Calculator).calculateEntitlement(0xc000683b60, 0x1002b00, 0xc0000b8070, 0x0, 0x0) /go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:162 +0x20e github.com/uber/peloton/resmgr/entitlement.(Calculator).Start.func1(0xc000683b60, 0xc0004821c0) /go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:106 +0x138 created by github.com/uber/peloton/resmgr/entitlement.(Calculator).Start /go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:96 +0x133

talaniz commented 5 years ago

Currently, I'm unable to reproduce this. What does your setup look like (OS, go version, python version)?

talaniz commented 5 years ago

@TommyLike Can you pull the latest version and see if you're still reproducing this issue? I checked with our team and it looks like there was a fix put in for this, but would like to get some verification.

talaniz commented 5 years ago

@TommyLike Just wanted to follow up to see if you were able to confirm that this issue has been resolved?

TommyLike commented 5 years ago

@talaniz sorry for late response, I will recheck this today:)

TommyLike commented 5 years ago

Hey @talaniz , I paste some of the configurations and logs here in case needed.

Environment:：Ubuntu 16.04 Docker version:

Client:
 Version:           18.09.7
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        2d0083d
 Built:             Thu Jun 27 17:56:17 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.7
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       2d0083d
  Built:            Thu Jun 27 17:23:02 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Python Version:

Python 2.7.12

Images information:

uber/peloton                  latest              sha256:36e4d089ba273cd6c3c88ec213e80a37b88aac16308c9bb12e3f017967886188   4c1d9f5761be        3 months ago        1.8GB
mesosphere/mesos-slave        1.6.0-rc1           sha256:b44439db7b5bb2e7742d8e83efaa82d6b276ab7287f0d99107ef6bdb6502599b   69562200cd36        14 months ago       1.02GB
mesosphere/mesos-master       1.6.0-rc1           sha256:e63cfdbd7265ad4a73c4e01bb8ff1393c3fcf49c68ce330b0c56fd3977f63d9f   b2bbd52d0b2b        14 months ago       1.02GB
cassandra                     3.0.14              sha256:616454e966db82cbea0fdf0b579fa827445de952214edbab5a23b63bc92a5aa0   3bdcdfbd60a0        21 months ago       380MB

Node info:

⚡ root@husheng-test  ~/go_projects/src/github.com/uber/peloton/bin   master  ./peloton hostmgr hosts
              Hostname|   CPU|   GPU|         MEM|         Disk|  State| Task Hold
  peloton-mesos-agent0|  4.00|  0.00|  2048.00 MB|  20000.00 MB|  ready|
  peloton-mesos-agent1|  4.00|  0.00|  2048.00 MB|  20000.00 MB|  ready|
  peloton-mesos-agent2|  4.00|  0.00|  2048.00 MB|  20000.00 MB|  ready|

Resource pool yaml:

name: HelloWorldPool
owningteam: MyTeam
ldapgroups:
- MyGroup
description: "My first resource pool on Peloton"
resources:
- kind: cpu
  reservation: 1
  limit: 2
  share: 1
- kind: memory
  reservation: 1024
  limit: 2048
  share: 1

Parts of the log out put

{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:00Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:00Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:00Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:00Z"}
{"level":"info","msg":"GetActiveTasks called","req":{"states":["LAUNCHING","PENDING","PLACED","PLACING","PREEMPTING","READY"]},"time":"2019-07-04T09:41:02Z"}
{"level":"info","msg":"GetActiveTasks returned","time":"2019-07-04T09:41:02Z"}
{"level":"info","msg":"GetPreemptibleTasks returned","preemptible_tasks":null,"time":"2019-07-04T09:41:02Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Eligible resource pools for preemption","pools":null,"time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:05Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:05Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:05Z"}
{"hosts":[],"level":"info","msg":"Draining hosts","time":"2019-07-04T09:41:06Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:10Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:10Z"}
{"level":"info","msg":"GetPreemptibleTasks returned","preemptible_tasks":null,"time":"2019-07-04T09:41:12Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Eligible resource pools for preemption","pools":null,"time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:15Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:15Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:15Z"}
{"hosts":[],"level":"info","msg":"Draining hosts","time":"2019-07-04T09:41:16Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:20Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:20Z"}
{"level":"info","msg":"GetPreemptibleTasks returned","preemptible_tasks":null,"time":"2019-07-04T09:41:22Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Eligible resource pools for preemption","pools":null,"time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:25Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:25Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:25Z"}
{"hosts":[],"level":"info","msg":"Draining hosts","time":"2019-07-04T09:41:26Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:30Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:30Z"}
{"level":"info","msg":"GetPreemptibleTasks returned","preemptible_tasks":null,"time":"2019-07-04T09:41:32Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Eligible resource pools for preemption","pools":null,"time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:35Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Completed Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Starting Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","slack_entitlement":"CPU:12.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:35Z"}
{"level":"info","msg":"Completed Slack Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:35Z"}
{"hosts":[],"level":"info","msg":"Draining hosts","time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"LookupResourcePoolID called","request":{"path":{"value":"/"}},"time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"CreateResourcePool called","request":{"config":{"name":"HelloWorldPool","owningTeam":"MyTeam","ldapGroups":["MyGroup"],"description":"My first resource pool on Peloton","resources":[{"kind":"cpu","reservation":1,"limit":2,"share":1},{"kind":"memory","reservation":1024,"limit":2048,"share":1}],"parent":{"value":"root"}}},"time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Scheduling policy is not set, use default PriorityFIFO","time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"siblings to check","name":"HelloWorldPool","siblingNames":{},"time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"41b5f815-89ec-4a59-906b-c0623b2728c7","slack_limit":{"CPU":0,"MEMORY":204.8,"DISK":0,"GPU":0},"time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":1,"MEMORY":1024,"DISK":0,"GPU":0},"respool_id":"41b5f815-89ec-4a59-906b-c0623b2728c7","time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"calculating entitlement","time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Setting slack limit","respool_id":"root","slack_limit":{"CPU":0,"MEMORY":1228.8000000000002,"DISK":12000,"GPU":0},"time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Setting reservation","reservation":{"CPU":12,"MEMORY":6144,"DISK":60000,"GPU":0},"respool_id":"root","time":"2019-07-04T09:41:36Z"}
{"level":"info","msg":"Updating root resources","root resource ":[{"kind":"cpu","reservation":12,"limit":12},{"kind":"gpu"},{"kind":"disk","reservation":60000,"limit":60000},{"kind":"memory","reservation":6144,"limit":6144}],"time":"2019-07-04T09:41:36Z"}
{"entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"Starting Entitlement cycle for respool","respool_id":"root","respool_name":"root","time":"2019-07-04T09:41:36Z"}
{"demand_cap_by_limit":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","demand_cap_by_reservation_assignment":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","demand_not_satisfied":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","entitlement":"CPU:12.00 MEM:6144.00 DISK:60000.00 GPU:0.00","level":"info","msg":"First pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":1,"limit":2,"share":1},"memory":{"kind":"memory","reservation":1024,"limit":2048,"share":1}},"slack_entitlement":"CPU:0.00 MEM:0.00 DISK:0.00 GPU:0.00","time":"2019-07-04T09:41:36Z","total_share":{}}
{"final_assignment":{"CPU":2,"MEMORY":0,"DISK":0,"GPU":0},"level":"info","msg":"Third pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":1,"limit":2,"share":1},"memory":{"kind":"memory","reservation":1024,"limit":2048,"share":1}},"time":"2019-07-04T09:41:36Z"}
{"final_assignment":{"CPU":2,"MEMORY":2048,"DISK":0,"GPU":0},"level":"info","msg":"Third pass completed for respool","respool_name":"HelloWorldPool","respool_resources":{"cpu":{"kind":"cpu","reservation":1,"limit":2,"share":1},"memory":{"kind":"memory","reservation":1024,"limit":2048,"share":1}},"time":"2019-07-04T09:41:36Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xbac011]

goroutine 79 [running]:
github.com/uber/peloton/resmgr/entitlement.(*Calculator).getChildShare(0xc00067fa40, 0x1011aa0, 0xc001632480, 0xe90cf3, 0x4, 0x3ff0000000000000)
    /go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:182 +0xd1
github.com/uber/peloton/resmgr/entitlement.(*Calculator).distributeUnclaimedResources(0xc00067fa40, 0x1011aa0, 0xc001632480, 0xc001ddb8e8, 0xc001ddba30)
    /go/src/github.com/uber/peloton/resmgr/entitlement/non_revocable.go:320 +0x189
github.com/uber/peloton/resmgr/entitlement.(*Calculator).setEntitlementForChildren(0xc00067fa40, 0x1011aa0, 0xc001632480)
    /go/src/github.com/uber/peloton/resmgr/entitlement/non_revocable.go:75 +0x572
github.com/uber/peloton/resmgr/entitlement.(*Calculator).calculateEntitlement(0xc00067fa40, 0x1002b00, 0xc0000400f0, 0x0, 0x0)
    /go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:162 +0x20e
github.com/uber/peloton/resmgr/entitlement.(*Calculator).Start.func1(0xc00067fa40, 0xc0005584d0)
    /go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:106 +0x138
created by github.com/uber/peloton/resmgr/entitlement.(*Calculator).Start
    /go/src/github.com/uber/peloton/resmgr/entitlement/calculator.go:96 +0x133
talaniz commented 5 years ago

@TommyLike thanks for this and apologies for the late reply. I'll check this with the team.