search

SteveLTN / https-portal

A fully automated HTTPS server powered by Nginx, Let's Encrypt and Docker.
MIT License
4.45k stars 295 forks source link

Volume to store certificates #239

Open Tombenpotter opened 4 years ago

Tombenpotter commented 4 years ago

Hello!

This is more of a question, but I'm in a spot of trouble. I've used this for a while without problems, and everything works great.

However, it seems like the certificates are re-generated each time I start my docker-compose services. This is fine in itself, but I've reached the maximum amount of requests for this week, so it seems like this is a bit more of a problem than I had originally thought.

I've looked through the README but couldn't find what I was looking for : is there a way to create a volume to store the certificates so as to avoid generating a brand new one each time I start my services?

Thanks for the help!

SteveLTN commented 4 years ago

Hi,

By default an anonymous volume is created, and HTTPS-PORTAL will try to avoid re-creating the certificates if possible. Is your Docker setup special that it ignores the volumes created?

Do you have FORCE_RENEW on?

If you have a normal docker-compose setup, paste your config here. I can have a look.

Tombenpotter commented 4 years ago

Thanks for the answer !

I don't have anything special in my Docker setup, and FORCE_RENEW isn't on. The only thing I could think of is that the file isn't docker-compse.yml but has a custom name that makes me use the -f option when I up and down my containers.

Here's my docker-compose file:

version: '3.7'

services:
  web:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: web
    env_file:
      - preprod.env
    restart: always
    volumes:
      - ./:/app/

  https-portal:
    image: steveltn/https-portal
    ports:
      - "80:80"
      - "443:443"
    environment:
      DOMAINS: 'mydomain.fr -> http://web:80'
      STAGE: 'production' # Don't use production until staging works
SteveLTN commented 4 years ago

One thing you can try is to remove the whole docker images together with their storage volumnes. You can also mount /var/lib/https-portal to an external volume, see this part of the document.

Tombenpotter commented 4 years ago

Thank you for the help! I'll try to mount /var/lib/https-portal to an external volume and get back to you

mkantor commented 3 years ago

I got bit by this too (and am currently rate-limited because of it 😭).

I learned that docker-compose does not reuse anonymous volumes if you do docker-compose down && docker-compose up (or anything else that causes the https-portal container to be removed). See this issue for details.

If your workflow entails removing the container then you need a non-anonymous volume. If you don't want to mount a specific host directory then you can use a named volume like so:

version: "3.8"

volumes:
  ssl-certificates:

services:
  https-portal:
    image: steveltn/https-portal:1
    # ...
    volumes:
      - ssl-certificates:/var/lib/https-portal
SteveLTN commented 3 years ago

Yeah... if you just need to stop the containers you just do ‘docker-compose stop.’

On Sat, 12 Dec 2020 at 2:48 AM Matt Kantor notifications@github.com wrote:

I got bit by this too (and am currently rate-limited because of it 😭).

docker-compose does not reuse anonymous volumes if you do docker-compose down && docker-compose up (or anything else that causes the https-portal container to be removed). See this issue https://github.com/docker/compose/issues/7444 for details.

If your workflow entails removing the container then you need a non-anonymous volume. If you don't want to mount a specific host directory then you can use a named volume https://docs.docker.com/compose/compose-file/#volume-configuration-reference like so:

version: "3.8"

volumes:

ssl-certificates:

services:

https-portal:

image: steveltn/https-portal:1

# ...

volumes:

  - ssl-certificates:/var/lib/https-portal

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/SteveLTN/https-portal/issues/239#issuecomment-743621843, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD4AZZQUSGB5YATQUH2A5DSULDXPANCNFSM4QPBYAWA .

djordjedjukic commented 3 years ago

Hey, I have a question. I hit rate limits for some reason. I wanted to check with you am I doing something wrong (same thing worked in the past). I have GH action which do this on master push:

docker-compose pull
docker-compose up -d

and here is my docker-compose.yml

version: '3'

services:
  billtracker:
    image: djordjedjukic/billtracker
  https-portal:
    image: steveltn/https-portal:1
    ports:
      - '80:80'
      - '443:443'
    links:
      - billtracker
    restart: always
    environment:
        DOMAINS: 'racunoteka.djole.net -> http://billtracker:80'
        STAGE: 'production'

I tried just with docker-compose up manually, and I am getting this in a loop:

https-portal_1  | [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
https-portal_1  | [s6-init] ensuring user provided files have correct perms...exited 0.
https-portal_1  | [fix-attrs.d] applying ownership & permissions fixes...
https-portal_1  | [fix-attrs.d] done.
https-portal_1  | [cont-init.d] executing container initialization scripts...
https-portal_1  | [cont-init.d] 00-welcome: executing...
https-portal_1  |
https-portal_1  | ========================================
https-portal_1  | HTTPS-PORTAL v1.18.0
https-portal_1  | ========================================
https-portal_1  |
https-portal_1  | [cont-init.d] 00-welcome: exited 0.
https-portal_1  | [cont-init.d] 20-setup: executing...
https-portal_1  | DH parameters appear to be ok.
https-portal_1  | -----BEGIN DH PARAMETERS-----
https-portal_1  | MIIBCAKCAQEAq3YaAx3NP+If9VNFa5hRQt51lvuvcKr+WhYys16RR8PifJyYEnrG
https-portal_1  | Yob7CapJUXtfxfo5t5EudGmMVaacY7q0unIwIMoi1jljkJFcT3/IZcSbotTHR4fY
https-portal_1  | zOMl0tblnMdpQvzRHW4P2gVQefexQx0jClfu/QSxoB6pNo2gJ0Rs7Rz7YgZv9H1M
https-portal_1  | OSJgP/BaVC5U8BqjpfT/tE9TbrQQSqdoiJlgi0M6kbkYERE/6BRidjZz+klmf4gD
https-portal_1  | 57/0EkrdRWfYSmUf6RiWq8kiugYuD91xhXsxyZwqA6gu91r66OLtyHJ9o/JBlXJz
https-portal_1  | 6YujUot9GiR6SsnEQq+/1b7jnErJ4wP8swIBAg==
https-portal_1  | -----END DH PARAMETERS-----
https-portal_1  | RSA key ok
https-portal_1  | Generating RSA private key, 2048 bit long modulus (2 primes)
https-portal_1  | ........................+++++
https-portal_1  | ........................................................+++++
https-portal_1  | e is 65537 (0x010001)
https-portal_1  | Signing certificates from https://acme-v02.api.letsencrypt.org/directory ...
https-portal_1  | Parsing account key...
https-portal_1  | Parsing CSR...
https-portal_1  | Found domains: racunoteka.djole.net
https-portal_1  | Getting directory...
https-portal_1  | Directory found!
https-portal_1  | Registering account...
https-portal_1  | Already registered!
https-portal_1  | Creating new order...
https-portal_1  | Traceback (most recent call last):
https-portal_1  |   File "/bin/acme_tiny", line 198, in <module>
https-portal_1  |     main(sys.argv[1:])
https-portal_1  |   File "/bin/acme_tiny", line 194, in main
https-portal_1  |     signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca, disable_check=args.disable_check, directory_url=args.directory_url, contact=args.contact)
https-portal_1  |   File "/bin/acme_tiny", line 121, in get_crt
https-portal_1  |     order, _, order_headers = _send_signed_request(directory['newOrder'], order_payload, "Error creating new order")
https-portal_1  |   File "/bin/acme_tiny", line 60, in _send_signed_request
https-portal_1  |     return _do_request(url, data=data.encode('utf8'), err_msg=err_msg, depth=depth)
https-portal_1  |   File "/bin/acme_tiny", line 46, in _do_request
https-portal_1  |     raise ValueError("{0}:\nUrl: {1}\nData: {2}\nResponse Code: {3}\nResponse: {4}".format(err_msg, url, data, code, resp_data))
https-portal_1  | ValueError: Error creating new order:
https-portal_1  | Url: https://acme-v02.api.letsencrypt.org/acme/new-order
https-portal_1  | Data: {"protected": "eyJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvbmV3LW9yZGVyIiwgImFsZyI6ICJSUzI1NiIsICJub25jZSI6ICIwMTA0OTU4QVFwcU5aZU9mS2tueE9RR0FZQWg5RllnU0ZyVjlWRGdGRGxIdTBORSIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xMTQ0NDA3MDgifQ", "payload": "eyJpZGVudGlmaWVycyI6IFt7InR5cGUiOiAiZG5zIiwgInZhbHVlIjogInJhY3Vub3Rla2EuZGpvbGUubmV0In1dfQ", "signature": "j7wS9Vjr3T8J1W41USoDyF0M3K1kDnRBIVJpmUL13qJP3hdHr357ncr0DHEx4zcl56mL0_gxynMcf7bhz55-iznG0g2Fq0fgu64yCguv6WML0W2Xo8XUDTfJ9ZB3zZl6ZgwnJHFJ5OMcdELiF-GCsxZQ7vDE_LXdZX1Rsph_kO6KwexKXa57fZS2l8ZpdDeiuoPqerF9_i6M-usypm5ZuiYvep90TakHOf9bx2apVGrwfycZB5BWjLXcS8NZiG-rt7rxTL-8-A_ApAMEEXugx4SwsAs85A9BDyNM8aSrmuwBKj4YVzbnF9A0wEkGzSMUmYZwKaD4M-cRPUfkEY2cNuLbGRtNOGMT3X3bKuo3DmXGmRs9l1UK9oIlXVgysPsIh8qPI5F810-uYPk_frV0t9DzRJxztth7bBX3m5EKj6bVTwjEO9DY3unCzKsWfXWaFhID2G3hEsr-SgrGNyT1qWf-qBzyIcT49_5NPU5g0gjssRuQallMemf2U_jcGAAXT2twvCD4OlFLAOEiSMonhYx1xq0r9jJ-F6N2tuzAlSszpqhKrclV1Wa34YTn5weHOByyvWfnUX3HoWlKTChQ1jJKOhHTMkQdz7lnbTc7zbNsC5_3yGCS66RGaTh8U8TWDKAZhrAJXtM23Ywcx_CqzKgN761y7ZDgdoPPCisOc7s"}
https-portal_1  | Response Code: 429
https-portal_1  | Response: {u'status': 429, u'type': u'urn:ietf:params:acme:error:rateLimited', u'detail': u'Error creating new order :: too many certificates already issued for exact set of domains: racunoteka.djole.net: see https://letsencrypt.org/docs/rate-limits/'}
https-portal_1  | ================================================================================
https-portal_1  | Failed to sign racunoteka.djole.net.
https-portal_1  | Make sure you DNS is configured correctly and is propagated to this host
https-portal_1  | machine. Sometimes that takes a while.
https-portal_1  | ================================================================================
https-portal_1  | Failed to obtain certs for racunoteka.djole.net
https-portal_1  | [cont-init.d] 20-setup: exited 1.
https-portal_1  | [cont-finish.d] executing container finish scripts...
https-portal_1  | [cont-finish.d] done.
https-portal_1  | [s6-finish] waiting for services.
https-portal_1  | [s6-finish] sending all processes the TERM signal.
https-portal_1  | [s6-finish] sending all processes the KILL signal and exiting.

Thank you

SteveLTN commented 3 years ago

@djordjedjukic It seems you are rate limited. Have you been trying to sign a lot of certificates, or pulled new versions very often recently? I just tried to upgrade from 1.17 to 1.18 myself. It seems to be working fine. However, if you don't have a named volumes set, you might end up obtaining a new certificate every time you upgrade the version.

Try this with staging, and move to production when the rate-limit is lifted:

version: '3'

services:
  billtracker:
    image: djordjedjukic/billtracker
  https-portal:
    image: steveltn/https-portal:1
    ports:
      - '80:80'
      - '443:443'
    links:
      - billtracker
    restart: always
    environment:
        DOMAINS: 'racunoteka.djole.net -> http://billtracker:80'
        STAGE: 'staging'
    volumes:
      - https-portal-data:/var/lib/https-portal

volumes:
  https-portal-data:

This should avoid getting new certs when upgrading the version.

I'll update the document as well.

djordjedjukic commented 3 years ago

@SteveLTN it's working with changes you suggested.

Have you been trying to sign a lot of certificates, or pulled new versions very often recently? GH action triggered more then 10 times in last two days, and every time it did:

docker-compose pull
docker-compose up -d

Can you just please explain me, was that a problem, or the problem was that I didn't have volumes mounted?

Thank you one more time.

SteveLTN commented 3 years ago

@djordjedjukic

When you don't configure the volumes, Docker will remove the volume and create a new one when HTTPS-PORTAL is upgraded. That means the previously signed certificate would be lost, therefore we need to sign again.

Normally this isn't a big problem. But if you upgrade HTTPS-PORTAL many times in a short period, you can potentially be rate-limited by Let's Encrypt server.

I'm not sure why Let's Encrypt decides to rate-limit you. That's why I asked you, did you upgrade the version many times recently?

djordjedjukic commented 3 years ago

Well I didn't do anything manually. I did docker-compose pull a lot of times in last two days, but I guess that could not upgrade HTTPS-PORTAL many times.

mrcloudbook commented 2 months ago

I'm encountering an issue with HTTPS-PORTAL where it regenerates SSL certificates every time the container restarts, despite using a named volume for persistence (https-portal-data:). This behavior is causing us to hit Let's Encrypt rate limits.