Closed timsun28 closed 3 years ago
+1
@timsun28 I'm having the same issue with multiple projects, I was thinking that the issue was related to my CI but now I'm guessing that is related to this update on a MUP dependency: https://github.com/nginx-proxy/docker-letsencrypt-nginx-proxy-companion/issues/718
https://github.com/nginx-proxy/docker-letsencrypt-nginx-proxy-companion
I'm trying to figure it out how to use a previous version of this dependency. @zodern can you give me a tip to do that?
@oinofactordevs Thank you for mentioning this, I will also have a look into it. I am happy it just happened with my beta domain and I could move it to a different domain, but I'm waiting before pushing a new version to production, because the beta one is still down after a week and I can't seem to get it up again ...
Hope this can get fixed soon or reverted to an old version for the moment.
Update: it just happened with my second beta domain now as well. And for users who use a domain name that only accept https connections (.app for example) they can't even access the site when adding an exception for the site.
@timsun28 I'm having the issue on multiple platforms, and seems to be related to the latest version of the docker image I shared. The update date of the docker image match with the beginning of this issue (a week ago). The platforms that already had the issue will not work for a week until letsencrypt allows you to get a new certificate. My plan is to change the MUP code on this file (from jrcs/letsencrypt-nginx-proxy-companion:latest --> to jrcs/letsencrypt-nginx-proxy-companion:1.13.1) and try to use the older version of the docker image.
@timsun28 I think I found the way to get the broken sites back:
1 - Go to your remote server and get the docker containerID assigned to the mup-nginx-proxy-letsencrypt 2 - stop the container: docker stop mup-nginx-proxy-letsencrypt 3 - remove the container: docker rm containerID 4 - On your the machine that you are using for the deployment, locate the mup package installation and change this on the source code: a) - src/plugins/proxy/assets/templates/start.sh
#79 -> jrcs/letsencrypt-nginx-proxy-companion -> to -> jrcs/letsencrypt-nginx-proxy-companion:v1.13.1
b) - src/plugins/meteor/assets/templates/start.sh
5 - Run mup init , mup setup and mup deploy for your project. I'm testing this on all the projects while I'm also writing it so -> sorry for any typo , I hope this could help you , I'm also running crazy with this and I have a lot of production platforms that I can't update until this were fixed.
Let me know if it works for you.
@oinofactordevs Thank you for sharing! I will try it tomorrow morning! I really appreciate it that you shared this, and hopefully you get your apps back online as well.
I will keep you updated!
@zodern I don't have a GitHub client right now so I imported the repo on bitbucket: https://bitbucket.org/bitskingdom/meteor-up/commits/ff1d6862eaef485a792f9bac127e367f1d2033e8 Please check that commit and let me know if this make sense regarding the @timsun28 issue.
@timsun28 this fixed all my projects. I'm not sure which exactly was the issue but is related to the last version of this docker image dependency: https://github.com/nginx-proxy/docker-letsencrypt-nginx-proxy-companion
I hope this helps!
Just as a reminder: I'm suspecting that the issue is related to the location of the certificates on the server, seems like every time that I was deploying on the server, the certificate was requested even if there was a certificate already created. That explains: 1 - why I was able to reproduce the issue on my staging environments (where I have hourly deploys) 2 - why after I changed the docker image the old certificates were recognized and wasn't needed to request a new certificate (have in mind that the same server using the same IP, previous the fix, was reaching the rate limit on letsencryp)
cc @zodern and @timsun28 BTW: I'm also @oinofactordevs
@fabian-aramendi I was able to get everything back online thanks to your guide! For others who have mup installed as a global package, you need to make the changes in the following location: ~/.nvm/versions/node/Your node version/lib/node_modules/mup
You can find this location by typing npm root -g in your terminal.
For me the files were also in a lib folder instead of the src folder you mentioned, but it all worked out well.
Reading from the change logs from the nginx-proxy-companion it seems like they released a 2.0 version without backwards compatibility for some parts which caused the issues.
Thanks for finding the cause. This is fixed in Mup 1.5.3.
Additional informations if your project uses jrcs/letsencrypt-nginx-proxy-companion:latest
:
Required read if you use the
latest
version : the recentv2.0.0
release of this project mark the switch of the ACME client used by the Docker image from simp.le to acme.sh. This switch result in some backward incompatible changes, so please read this issue and the updated docs for more details before updating your image. The single most important change is that the container now requires a volume mounted to/etc/acme.sh
in order to persist ACME account keys and SSL certificates. The last tagged version that uses simp_le isv1.13.1
.
I have updated to Mup 1.5.3 removed all docker containers from server. ran the deployment again.
But still have error:
ACME server returned an error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many certificates already issued for exact set of domains: #####: see
https://letsencrypt.org/docs/rate-limits/
I have updated to Mup 1.5.3 removed all docker containers from server. ran the deployment again.
But still have error:
ACME server returned an error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many certificates already issued for exact set of domains: #####: see
https://letsencrypt.org/docs/rate-limits/
But app is live next morning; Can there be an explanation ?
Mup version 1.5.2 (but I got this error around 4 days ago on 1.5.0.
Mup config
Output of command (mup proxy logs-le)
What I found on the letsencrypt site is that the limit is set to 5 times a week. This is for renewing the ssl certificate and from what I read this is only done once a month by Meteor up.
I haven't had this issue with previous projects, but I have my suspection of why this could be happening. In the past 2 weeks I changed two things for deploying with Meteor up.
Because of the bug I now have to switch back to an older version (12.18.3) using nvm to get it to deploy. This might be a reason why it's trying to refresh the domain everytime. I also implemented a simple bash script that would do these steps for me, because I kept forgetting about switching the nvm version back.
This is the bash script I used to deploy the project:
I am hoping that this can get fixed soon, because it's currently breaking my application for showing the following error:
subdomain.subdomain.host.com uses an invalid security certificate. The certificate is not trusted because it is self-signed. Error code: MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT
The app is running on the new .app from google, so it is also not accessible without https. I have currently switched to a new subdomain and have been able to deploy there without any issues, but I'm worried the same issue will reappear again after deploying too often.