"cf dev start" Failing on Ubuntu Linux 18.04

fhartojo commented 5 years ago

CF newbie here. "cf dev start" consistently fails on my laptop. Some details:

From log/deploy-cf.log:

...

Task 71

Task 71 | 18:34:24 | Preparing deployment: Preparing deployment (00:00:10)
Task 71 | 18:34:44 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 71 | 18:34:44 | Creating missing vms: nats/16389963-b246-4c9e-b9a7-0aa2f8b812a8 (0)
Task 71 | 18:34:44 | Creating missing vms: credhub/866a65da-882c-42b6-b5bd-768b7d4d6b12 (0)
Task 71 | 18:34:44 | Creating missing vms: cc-worker/42a984db-7fd0-4a81-87b5-b32e9122320a (0)
Task 71 | 18:34:44 | Creating missing vms: singleton-blobstore/45963efd-34ac-4f05-826b-8aff0820cac9 (0)
Task 71 | 18:34:44 | Creating missing vms: diego-api/87c13232-3fcc-4efa-8285-8d028ee6e3a9 (0)
Task 71 | 18:34:44 | Creating missing vms: adapter/37d03f1b-5a08-4952-a937-8b06b27d95bc (0)
Task 71 | 18:34:44 | Creating missing vms: router/fa3cb3b6-421a-4bb0-b4d1-11a8a4ff7d19 (0)
Task 71 | 18:34:44 | Creating missing vms: api/c85e8b73-c839-4b26-a5d9-76b57931a6ef (0)
Task 71 | 18:34:44 | Creating missing vms: scheduler/07e53ccd-260c-41ff-84e2-2fb173007e09 (0)
Task 71 | 18:34:44 | Creating missing vms: doppler/3bbe547c-2ef4-4293-b105-e657f4209dc5 (0)
Task 71 | 18:34:44 | Creating missing vms: diego-cell/263d4ef4-a150-4281-ad7f-a11447c96946 (0)
Task 71 | 18:34:44 | Creating missing vms: log-api/d44e4adc-bac8-4d9a-a9fe-eb2b04907ab1 (0)
Task 71 | 18:34:44 | Creating missing vms: uaa/9f182029-e8e1-4a09-bcde-86d473a12a04 (0)
Task 71 | 18:34:44 | Creating missing vms: database/05b9772d-f58a-4f55-8980-652b36b9d26e (0)
Task 71 | 18:35:06 | Creating missing vms: adapter/37d03f1b-5a08-4952-a937-8b06b27d95bc (0) (00:00:22)
Task 71 | 18:35:06 | Creating missing vms: nats/16389963-b246-4c9e-b9a7-0aa2f8b812a8 (0) (00:00:22)
Task 71 | 18:35:07 | Creating missing vms: singleton-blobstore/45963efd-34ac-4f05-826b-8aff0820cac9 (0) (00:00:23)
Task 71 | 18:35:08 | Creating missing vms: credhub/866a65da-882c-42b6-b5bd-768b7d4d6b12 (0) (00:00:24)
Task 71 | 18:35:08 | Creating missing vms: diego-api/87c13232-3fcc-4efa-8285-8d028ee6e3a9 (0) (00:00:24)
Task 71 | 18:35:08 | Creating missing vms: cc-worker/42a984db-7fd0-4a81-87b5-b32e9122320a (0) (00:00:24)
Task 71 | 18:35:08 | Creating missing vms: doppler/3bbe547c-2ef4-4293-b105-e657f4209dc5 (0) (00:00:24)
Task 71 | 18:35:09 | Creating missing vms: uaa/9f182029-e8e1-4a09-bcde-86d473a12a04 (0) (00:00:25)
Task 71 | 18:35:09 | Creating missing vms: diego-cell/263d4ef4-a150-4281-ad7f-a11447c96946 (0) (00:00:25)
Task 71 | 18:35:09 | Creating missing vms: router/fa3cb3b6-421a-4bb0-b4d1-11a8a4ff7d19 (0) (00:00:25)
Task 71 | 18:35:10 | Creating missing vms: log-api/d44e4adc-bac8-4d9a-a9fe-eb2b04907ab1 (0) (00:00:26)
Task 71 | 18:35:10 | Creating missing vms: database/05b9772d-f58a-4f55-8980-652b36b9d26e (0) (00:00:26)
Task 71 | 18:35:10 | Creating missing vms: scheduler/07e53ccd-260c-41ff-84e2-2fb173007e09 (0) (00:00:26)
Task 71 | 18:35:11 | Creating missing vms: api/c85e8b73-c839-4b26-a5d9-76b57931a6ef (0) (00:00:27)
Task 71 | 18:35:11 | Updating instance nats: nats/16389963-b246-4c9e-b9a7-0aa2f8b812a8 (0)
Task 71 | 18:35:11 | Updating instance adapter: adapter/37d03f1b-5a08-4952-a937-8b06b27d95bc (0) (00:00:34)
Task 71 | 18:35:45 | Updating instance nats: nats/16389963-b246-4c9e-b9a7-0aa2f8b812a8 (0) (00:00:34)
Task 71 | 18:35:45 | Updating instance database: database/05b9772d-f58a-4f55-8980-652b36b9d26e (0) (00:03:09)
Task 71 | 18:38:54 | Updating instance singleton-blobstore: singleton-blobstore/45963efd-34ac-4f05-826b-8aff0820cac9 (0)
Task 71 | 18:38:54 | Updating instance cc-worker: cc-worker/42a984db-7fd0-4a81-87b5-b32e9122320a (0)
Task 71 | 18:38:54 | Updating instance api: api/c85e8b73-c839-4b26-a5d9-76b57931a6ef (0)
Task 71 | 18:38:54 | Updating instance diego-cell: diego-cell/263d4ef4-a150-4281-ad7f-a11447c96946 (0)
Task 71 | 18:38:54 | Updating instance doppler: doppler/3bbe547c-2ef4-4293-b105-e657f4209dc5 (0)
Task 71 | 18:38:54 | Updating instance credhub: credhub/866a65da-882c-42b6-b5bd-768b7d4d6b12 (0)
Task 71 | 18:38:54 | Updating instance diego-api: diego-api/87c13232-3fcc-4efa-8285-8d028ee6e3a9 (0)
Task 71 | 18:38:54 | Updating instance scheduler: scheduler/07e53ccd-260c-41ff-84e2-2fb173007e09 (0)
Task 71 | 18:38:54 | Updating instance log-api: log-api/d44e4adc-bac8-4d9a-a9fe-eb2b04907ab1 (0)
Task 71 | 18:38:54 | Updating instance router: router/fa3cb3b6-421a-4bb0-b4d1-11a8a4ff7d19 (0)
Task 71 | 18:38:54 | Updating instance uaa: uaa/9f182029-e8e1-4a09-bcde-86d473a12a04 (0)
Task 71 | 18:40:06 | Updating instance log-api: log-api/d44e4adc-bac8-4d9a-a9fe-eb2b04907ab1 (0) (00:01:12)
Task 71 | 18:40:07 | Updating instance singleton-blobstore: singleton-blobstore/45963efd-34ac-4f05-826b-8aff0820cac9 (0) (00:01:13)
Task 71 | 18:40:33 | Updating instance diego-api: diego-api/87c13232-3fcc-4efa-8285-8d028ee6e3a9 (0) (00:01:39)
Task 71 | 18:41:03 | Updating instance cc-worker: cc-worker/42a984db-7fd0-4a81-87b5-b32e9122320a (0) (00:02:09)
Task 71 | 18:42:09 | Updating instance scheduler: scheduler/07e53ccd-260c-41ff-84e2-2fb173007e09 (0) (00:03:15)
Task 71 | 18:44:02 | Updating instance uaa: uaa/9f182029-e8e1-4a09-bcde-86d473a12a04 (0) (00:05:08)
Task 71 | 18:44:09 | Updating instance doppler: doppler/3bbe547c-2ef4-4293-b105-e657f4209dc5 (0) (00:05:15)
Task 71 | 18:44:11 | Updating instance diego-cell: diego-cell/263d4ef4-a150-4281-ad7f-a11447c96946 (0) (00:05:17)
Task 71 | 18:44:15 | Updating instance api: api/c85e8b73-c839-4b26-a5d9-76b57931a6ef (0) (00:05:21)
Task 71 | 18:44:45 | Updating instance router: router/fa3cb3b6-421a-4bb0-b4d1-11a8a4ff7d19 (0) (00:05:51)
Task 71 | 18:44:52 | Updating instance credhub: credhub/866a65da-882c-42b6-b5bd-768b7d4d6b12 (0) (00:05:58)

Task 71 Started  Thu Sep 26 18:34:24 UTC 2019
Task 71 Finished Thu Sep 26 18:44:52 UTC 2019
Task 71 Duration 00:10:28
Task 71 done

Succeeded
Setting api endpoint to api.dev.cfdev.sh...
Request error: Get https://api.dev.cfdev.sh/v2/info: dial tcp 127.0.0.53:443: connect: connection refused
TIP: If you are behind a firewall and require an HTTP proxy, verify the https_proxy environment variable is correctly set. Else, check your network connection.
FAILED

cf version:


$ cf dev version
CLI: 0.0.17
BUILD: 14 (b36c82e)

cf: v11.2.0 cf-mysql: 36.19.0


- Computer:

8-core CPU (i7-8565U), 32GB RAM, 512GB NVMe SSD, Ubuntu 18.04 w/ latest updates.



I've had to add 127.0.0.53 to /etc/hosts because it previously failed on looking up api.dev.cfdev.sh with a "no host found" error.

- Network:
At home, no proxy.

I checked and there's no listener on 443 which makes sense given the connection refused error, so does that mean there's a service that failed to start?

Anyhow, any help on this would be greatly appreciated.

Thank you.

cf-gitbot commented 5 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/168791798

The labels on this github issue will be updated when the story is started.

fhartojo commented 5 years ago

Seems like I've opened this issue on a wrong repo. Closing.

aemengo commented 5 years ago

@fhartojo This is the correct repo to submit this issue.

api.dev.cfdev.sh shouldn't resolve to 127.0.0.53. This is a real DNS address that you need the public internet to resolve. Furthermore, since there are multiple subdomains that also need to be resolved, injecting entries into your host file isn't enough.

Is it possible that your laptop wasn't connected to the public internet at the time? Next steps should be finding out why resolving api.dev.cfdev.sh yields nothing for you.

fhartojo commented 5 years ago

Thanks for responding.

So, I tried looking up (i.e., nslookup) api.dev.cfdev.sh on a couple of different hosts and they both point to 10.144.0.34, which I think is one of the interfaces brought up during "cf dev start", is that right? Since it's a 10.* address I don't think it's going to be available publicly, yes?

Anyway, I added these entries to /etc/hosts:

10.144.0.34 api.dev.cfdev.sh 10.144.0.34 login.dev.cfdev.sh 10.144.0.34 ssh.dev.cfdev.sh 10.144.0.34 doppler.dev.cfdev.sh 10.144.0.34 uaa.dev.cfdev.sh

and now "cf dev start" completed successfully. But, when I tried to deploy MySQL as a test it failed the first time because it couldn't resolve p-mysql.dev.cfdev.sh. Once I added that to /etc/hosts it completed successfully. So, what I think I need to do is to configure dnsmasq to resolve *.dev.cfdev.sh to 10.144.0.34. Does that sound alright?

Thanks.

cf-gitbot commented 5 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/168795255

The labels on this github issue will be updated when the story is started.

aemengo commented 5 years ago

I think is one of the interfaces brought up during "cf dev start", is that right?

Yes

Since it's a 10.* address I don't think it's going to be available publicly, yes?

Yes

So, what I think I need to do is to configure dnsmasq to resolve *.dev.cfdev.sh to 10.144.0.34. Does that sound alright?

This might be true in your case, but we don't expect this by default for our users. If it makes sense why cf dev start cannot resolve the domain using your networking setup (such as running in an offline environment), then you should pursue this option.

fhartojo commented 5 years ago

Sorry for the late response.

So, I found out several things:

Apparently all, at least the ones I've tried, public DNS servers would respond with 10.144.0.34 to inquiries on *.dev.cfdev.sh. I've tried Google, my ISP, Cloudflare amongst others. They all work.
What doesn't work, however, is my WiFi router (i.e., Google WiFi) which acts as my local DNS as per the usual practice. When I point nslookup to the router as the server it fails to resolve *.dev.cfdev.sh. I don't know why that is at this point and I'll continue to investigate.
For now how I'm fixing the issue on my Ubuntu 18.04 laptop is the following:

sudo service systemd-resolved stop sudo systemctl disable systemd-resolved sudo rm -f /etc/resolv.conf // Edit /etc/NetworkManager/NetworkManager.conf and add a line in the "[main]" section that says "dns=dnsmasq". // Add a file under /etc/NetworkManager/dnsmasq.d that ends with a ".conf"---I called mine dnsmasq-dev.cfdev.sh.conf---and add a line in it that says "address=/dev.cfdev.sh/10.144.0.34". sudo service NetworkManager restart

I think that's all. I have to disable systemd-resolved because I have a hard time making it play nice with NetworkManager and dnsmasq.

Anyhow, it's not ideal, but it works. If I manage to figure out why Google WiFi doesn't seem to respond to inquiries on *.dev.cfdev.sh, I'll update.

Thanks for all your help.

cloudfoundry-attic / cfdev

"cf dev start" Failing on Ubuntu Linux 18.04 #118