dmolokanov
commented
3 years ago @huguesBouvier Do you know if customer can change 443 port for their service in nested edge?
Closed mramin2000 closed 2 years ago
dmolokanov
commented
3 years ago @huguesBouvier Do you know if customer can change 443 port for their service in nested edge?
huguesBouvier
commented
3 years ago @dmolokanov Yes, the process is described here: https://docs.microsoft.com/en-us/azure/iot-edge/how-to-configure-api-proxy-module?view=iotedge-2020-11#minimize-open-ports. The changes need to be made on the parent.
mramin2000
commented
3 years ago Now it works. Thanks.
mramin2000
commented
3 years ago Now I'm working with docker registry module on the parent too. I've changed the API Proxy Default port to 8000 (in its port bindings as well). On the child edge device I've changed mcr.microsoft.com to $upstream:8000 and also in its config (instead of $upstream I've used parent hostname and port 8000). The connection works, but a new module deployment gives the error: "Unable to process module SimulatedTemperatureSensor add or update as the module identity could not be obtained" in the edgeAgent logs. When I remove modules in IoT Hub, they disappear correctly on the child device, but deploying them again doesn't work. I still like to use port 443 for my own services. Are there any other settings I have to do for this scenario?
huguesBouvier
commented
3 years ago No, there might be something else. Could you give me the logs of: API proxy on parent edgeHub on parent edgeHub on children edgeAgent on children edged on children Thanks!
huguesBouvier
commented
3 years ago Just realized: Did you put back the API proxy from 443 to 8000? Then you need to put edgeHub back on 443. That is why you are seeing this issue. You can't use 443 without the API proxy, it is used by edgeHub.
However, API proxy is configurable, if your applications are using HTTP, it should be possible to configure the API proxy to route your messages. Please see: Readme: https://github.com/Azure/iotedge/tree/master/edge-modules/api-proxy-module API proxy config file: https://github.com/Azure/iotedge/blob/master/edge-modules/api-proxy-module/templates/nginx_default_config.conf
mramin2000
commented
3 years ago With the following settings I can successfully use port 8000 of the API Proxy module and deploy modules to the child device, but I can't use 443 for my own services: On the parent: -edgeHub Portbinding for 443 with hostport 443 (and also 5671 and 8883) -API Proxy Default Nginx Port 8000 and port binding for 8000 to hostport 8000 on container create options On the child: -$upstream:8000 on each module -hostname:8000 in [agent] section in config.toml -edgeHub Portbinding for 443 with hostport 443 (and alos 5671 and 8883)
I tried to change the portbinding for the edgeHub for both devices to 443 with hostport 8443. This has worked before on the devices without nested edge to make my own service on port 443 run. But when I do this in the nested scenario the parent works and can deploy modules while port 443 is used for my service while I get the following logs on the child:
Here the logs of the parent device:
Do you have any other suggestions to solve this problem without editing the API Proxy config? Thanks a lot!
huguesBouvier
commented
3 years ago Check those options: https://github.com/Azure/iotedge/blob/master/doc/EnvironmentVariables.md Change the environment variable: HttpSettings__Port of the child, it should do what you want.
mramin2000
commented
3 years ago I tried this with Port 8000 and 8443 as well, but both settings didn't work. Moreover the child reported the new config in IoT Hub, but the edgeHub module didn't restart itself, only manually. The errors in the logs are the same as before, the child still tries to use 443 for connecting.
huguesBouvier
commented
3 years ago Hum, I never used that option. It may use only the server part then. if it doesn't work, I see only this:
mramin2000
commented
3 years ago When I set the edgeHub port bindings on the parent on 443 to 443, everything works, while the settings on the child seem to be unnecessary. There I've set the env variable "HttpSettings_Port" on 8000 or 8443 and the edgeHub port bindings on 443/8000, 443/8443, 8000/8000 and 8000/8443 but without any effect. But when I change "HttpSettings_Port" on the parent to 8000 with the same port bindings for 443/443, the child can not pull new deployments. I think port 443 for the child is hard coded somewhere, maybe this is a bug. But thanks a lot for your help!
huguesBouvier
commented
3 years ago Looking at the code it seems it changes only the server port, not where you connect to. I think its doing what's intended, though it won't help you. There doesn't seem to be any simple config option. You should try option 1 above. If you use http it should not be so difficult.
mramin2000
commented
3 years ago I've tried a lot with iptables rules in the last days, but none of these settings have worked for me. Here are some variants:
Variant A:
Variant B:
Variant C:
Here are the settings of iptables for Variant C:
huguesBouvier
commented
3 years ago Looking hard at it, it is difficult to pinpoint what the issue could be. I can't really help for that level of details. If IPtable doesn't work for you, what protocol is your application using? If it uses http, then loading your own config in api should be easier.
huguesBouvier
commented
2 years ago Can I close this issue? I can provide help for configuring IP proxy but it will not be possible for IP tables. Thanks!
mramin2000
commented
2 years ago Our application uses https, so we need incoming traffic on port 443 for our service. Looking at the nginx config from here I found that the edgeHub endpoint is mapped to https as default. Could it help to change this part (line 17,18) to another port like 8443? If not, do you have any suggestions for what I have to change in the API Proxy config? We need to reroute edgeHub traffic to another port like 8443. And thanks a lot for your help so far!
huguesBouvier
commented
2 years ago You can change the configuration very easily to 8443 by adding the env var: NGINX_DEFAULT_PORT=8443 but it wont help you because you will also need to change the downstream edgeHub.
However, the API proxy can handle 2 connection on the same port and route them to different target.
For example look at the piece of code in the config:
**if ($uri ~ ^/v2*.)**
{
**rewrite ^(.*)$ /registry$1 last;**
}
**location ~^/registry/(.*) {**
proxy_http_version 1.1;
resolver 127.0.0.11;
set $endpoint http://${DOCKER_REQUEST_ROUTE_ADDRESS}/$1$is_args$args;
proxy_pass $endpoint;
}
**if ($http_x_ms_version)**
{
**rewrite ^(.*)$ /storage$1 last;**
}
**location ~^/storage/(.*){**
resolver 127.0.0.11;
proxy_http_version 1.1;
set $endpoint http://${BLOB_UPLOAD_ROUTE_ADDRESS}/$1$is_args$args;
proxy_pass $endpoint;
}if the request contains the header "http_x_ms_version" it is routed to the blob storage module if the request uri start with "/v2" then it is routed to the "registry module" (for docker)
You can make your own rule: if the request come from your custom module below, route it to your custom module upstream.
For debugging I suggest that you mount the file /app/nginx_default_config.conf on your drive so you can make changes fast. do nginx -s stop in api proxy container to reload nginx with the new config.
mramin2000
commented
2 years ago I tried it again with iptables and made it finally working. It was much easier than I thought in the other tries.
On the parent I have deleted the 443/443 port binding for edgeHub and set API Proxy on Nginx Port 443 with port binding for 443 to hostport 8443. On the child I set edgeHub Portbinding on 443 to Hostport 8443 and put in $upstream:443 for the modules.
Additionally I made one rule on the parent: iptables -t nat -A PREROUTING -p tcp -s
Thanks for your help!
huguesBouvier
commented
2 years ago You're welcome!
Hello,
I'm working with the nested edge scenario and I'd like to use port 443 on both the child and the parent device for different service, but it seems like the devices are communicating on port 443 among themselves. Is it possible to change this port? I've already changed the portbindings for edgeHub and the API Proxy module on another port, but this didn't help.