search

GunSik2 / engineering_qna

Software Engineering Problem and Solving
0 stars 0 forks source link

cf cli usage and enabling insecure private docker registry #2

Open GunSik2 opened 5 years ago

GunSik2 commented 5 years ago

Problem

insecure private registry 의 경우 CF cli를 이용한 private registry 이미지 배포 시 오류가 발생

Env

Using docker repository password from environment variable CF_DOCKER_PASSWORD. Pushing app apipetstore to org 42-88 / space 42-88 as keprintels01... Getting app info... Creating app with these attributes...

Creating app apipetstore... Mapping routes...

Staging app and tracing logs... Cell cf21e481-4c27-404b-a312-cbb9c0feeda9 creating container for instance ad0fc8dd-2d99-41c4-9079-7e36b7a0c36f Cell cf21e481-4c27-404b-a312-cbb9c0feeda9 successfully created container for instance ad0fc8dd-2d99-41c4-9079-7e36b7a0c36f Staging... Staging process started ... Failed getting docker image by tag: pinging docker registry returned: Get https://112.160.**.**:38811/v2/: http: server gave HTTP response to HTTPS client Going to retry attempt: 1 Failed getting docker image by tag: pinging docker registry returned: Get https://112.160.**.**:38811/v2/: http: server gave HTTP response to HTTPS client Going to retry attempt: 2 Failed getting docker image by tag: pinging docker registry returned: Get https://112.160.**.**:38811/v2/: http: server gave HTTP response to HTTPS client Going to retry attempt: 3 Failed getting docker image by tag: pinging docker registry returned: Get https:///112.160.**.**:38811/v2/: http: server gave HTTP response to HTTPS client Staging process failed: Exit trace for group: builder exited with error: failed to fetch metadata from [cloudfoundry/test-app] with tag [latest] and insecure registries [] due to pinging docker registry returned: Get https://112.160.**.**:38811/v2/: http: server gave HTTP response to HTTPS client Exit status 2 Staging Failed: STG: Exited with status 2 Cell cf21e481-4c27-404b-a312-cbb9c0feeda9 stopping instance ad0fc8dd-2d99-41c4-9079-7e36b7a0c36f Cell cf21e481-4c27-404b-a312-cbb9c0feeda9 destroying container for instance ad0fc8dd-2d99-41c4-9079-7e36b7a0c36f Cell cf21e481-4c27-404b-a312-cbb9c0feeda9 successfully destroyed container for instance ad0fc8dd-2d99-41c4-9079-7e36b7a0c36f Error staging application: Staging error: staging failed FAILED



## Reference
- https://docs.cloudfoundry.org/devguide/deploy-apps/push-docker.html
GunSik2 commented 5 years ago

Solution

설정

when you have self-signed certs for your openstack, custom-ca is necessary. trusted-certs are only needed, if VMs deployed by bosh also need the same certs (i.e. if you deploy bosh with bosh)

### bosh-deployment 적용
- bosh-vars.yml

openstack_ca_cert: | -----BEGIN CERTIFICATE----- MIIEkzCCAvugAwIBAgIQbF/qRy7cys2yCnYSal5s9DANBgkqhkiG9w0BAQsFADBT MQwwCgYDVQQGEwNVU0ExFjAUBgNVBAoTDUNsb3VkIEZvdW5kcnkxKzApBgNVBAMT ImRlZmF1bHQuYmxvYnN0b3JlLWNhLmJvc2gtaW50ZXJuYWwwHhcNMTgxMDI0MDcy OTE0WhcNMTkxMDI0MDcyOTE0WjBTMQwwCgYDVQQGEwNVU0ExFjAUBgNVBAoTDUNs b3VkIEZvdW5kcnkxKzApBgNVBAMTImRlZmF1bHQuYmxvYnN0b3JlLWNhLmJvc2gt aW50ZXJuYWwwggGiMA0GCSqGSIb3DQEBAQUAA4IBjwAwggGKAoIBgQDQZxqufd8e Gy0Br5hEn0hRVCPtIiKNd6BdP5C/WEucU6yh+digA6xnYPQh6kP93cXc/ZObR3mT FEEnLXzTD33e6gEhf1+5gTxdswXE6uWJpgVxKjAY7KyHr3nzCeWXZ+AzEUIwMoWJ 8pKi/U/0xE3Eemg0FDgPcY8OnAPDZsYl86VI4msNKoxpVM1e/qvsL8j6CGrD3PZT BDuRtFqlPvz484KHm75mjHfMaL4YbUP6dpL1VX044APSR+ZUtO8Y3DIQNb8axvdv gdocJh5CnmkYTcwvsb3QDRxwYWbd3qTj8Mzx2u11yH9aN51K04ibauXAQVhjtCdW Y6O9neZhUNtxGTQgYkU+XeaO22Mp9ngT77YislBrhrMOz4IuCe6B8hmVVj6bLrXg BDQ9bjvp5T/3jDm67eucVxmgWj/XOzQU3Q9lkHbynYVLZLjZZDv3NUdz+OlaDBEK d3ZQ2W8XqjfLriY9DnBmzTXIW+SnIhCO8yil9QA8KKTnJv7803idBbUCAwEAAaNj MGEwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFA2+ ajt0c4zJ4JT7P0lJfUvll/YhMB8GA1UdIwQYMBaAFA2+ajt0c4zJ4JT7P0lJfUvl l/YhMA0GCSqGSIb3DQEBCwUAA4IBgQBDtK88T0zfGp4rfzDQ1VSKM8JaGjXk8mCQ 4PYwPp+E0SwbT0zePPQ9WO/tWnYz4FfpIDVyoM1A25SpQjZ0HafSPHw4stEx/IPJ Y+DUn991gI9eZ11PibkpqKluocFm0suGSiU7rLZctPfmJNRonlc+S4qQKYJsUlvN WSIJplMTCnilyZ9YPKN5vxwdPVXUmKqbBSqBQTwYbIHMQUfR9+SMxk5Y/p7QWXCX gN1v3Hc7TQVjM+LXG/RGVCZY1oJunz2nAvjysGqLzZ8IqvAYVmUBz6roVJ+9CY88 z3XBBu0WkGtqwlFAj9Rv+6rTvNQyYZmARWPdahDFcBZe74e/U/o+1+XxLMrcXX7G VX7lzERzuLSTg66iI/RLo1znLR3zxPNZjor/a42cyNEwhUURCD2E4RyUVHXor5lA H9IAv83Geu+qdJ4TObsQyMrxIPgn5ZHGlJ4UUF+dFnqxFGjyliEqCCLHd5+A1c7g 11mngHvu789gvjjKVEv5Q0cenf1m8zU= -----END CERTIFICATE----- external_db_host: 10.10.0.11 external_db_port: 3306 external_db_user: boshdb external_db_password: password external_db_adapter: mysql2 external_db_name: boshdb director_name: mybosh internal_cidr: 10.10.0.0/24 internal_gw: 10.10.0.1 internal_ip: 10.10.0.10 auth_url: http://192.168.10.5:5000/v3/ az: nova default_key_name: mybosh default_security_groups: [cf] net_id: be05e639-a1c2-453b-945c-f87c6dc1de60 openstack_password: 'password' openstack_username: admin openstack_domain: Default openstack_project: 'dev' private_key: /home/ubuntu/.ssh/mybosh.pem region: RegionOne

- interpolate (사전 테스트)

bosh interpolate bosh-deployment/bosh.yml \ --vars-file=bosh-vars.yml \ --vars-store=creds-interpolate.yml \ -o bosh-deployment/openstack/cpi.yml \ -o bosh-deployment/openstack/trusted-certs.yml \ -o bosh-deployment/misc/external-db.yml > bosh-int.yml

- deploy

bosh create-env bosh-deployment/bosh.yml \ --vars-file=bosh-vars.yml \ --state=state.json \ --vars-store=creds.yml \ -o bosh-deployment/openstack/cpi.yml \ -o bosh-deployment/openstack/trusted-certs.yml \ -o bosh-deployment/misc/external-db.yml 



## Reference
- [Enabling insecure docker registries](https://github.com/cloudfoundry/cf-deployment/pull/553)
- [Add custom-ca and trusted-certs](https://github.com/cloudfoundry/bosh-deployment/commit/a9f382b8838758e36546eb39b89998341aadf430)
- cf-deployment 에서[enable-docker.yml](https://github.com/robbo10/cf-deployment/blob/da4d5567c8e90f8f05f56ccb18c582cbdd788038/operations/enable-docker.yml) 이용한 처리 제안 (Reject 됨)
- [2015년 insecure docker registries 처리](https://github.com/cloudfoundry-incubator/diego-docker-cache-release/blob/develop/stubs-for-diego-release/bosh-lite-property-overrides.yml) 
- [bosh-cli usage](https://bosh.io/docs/cli-int/)