Automate configuring Dev Proxy certificate on Linux

plachance-q1 commented 6 months ago

Description

Hey everyone,

I've been trying to get devproxy to work on Linux for HTTPS proxying, but I'm hitting a wall. It works fine with HTTP, and on Windows, it's working as per the documentation. However, when I switch to Linux, HTTPS isn't cooperating.

I even tried exporting the Windows certificate and adding it to the trusted root certificates on Linux, but no luck there.

I've checked the docs, but I can't seem to find a fix. Any help or advice would be greatly appreciated!

I tried on Ubuntu app (Windows 11 pro wsl2) I tried on Ubuntu 18.04 I tried on a Docker container based on Ubuntu 22.04

Thanks!

Expected behaviour

The devproxy binary should register a root certificate when --install-cert is used

Actual behaviour

Any attempt to access a website through the proxy with HTTPS results with an error.

Steps to reproduce

Install dev proxy on a Linux (ubuntu) environment following documented instructions.

cd ~/devproxy
./devproxy --install-cert --log-level trace

Output is:

Loading plugin RetryAfterPlugin from: /home/patrick/devproxy/plugins/dev-proxy-plugins.dll
Loading plugin GenericRandomErrorPlugin from: /home/patrick/devproxy/plugins/dev-proxy-plugins.dll
8 error responses loaded from /home/patrick/devproxy/devproxy-errors.json
Listening on 127.0.0.1:8000...
  WARNING: Configure your operating system to use this proxy's port and address [127.0.0.1:8000](http://127.0.0.1:8000/)
Press CTRL+C to stop Dev Proxy

Open another terminal and try this:

wget -e use_proxy=yes -e https_proxy=https://127.0.0.1:8000/ https://jsonplaceholder.typicode.com/posts

The output:

Connecting to 127.0.0.1:8000... connected.
ERROR: cannot verify [jsonplaceholder.typicode.com](http://jsonplaceholder.typicode.com/)'s certificate, issued by ‘CN=Dev Proxy CA’:
  Unable to locally verify the issuer's authority.
To connect to [jsonplaceholder.typicode.com](http://jsonplaceholder.typicode.com/) insecurely, use `--no-check-certificate'.

If we follow recommendation and add --no-check-certificate, it works

 request     GET https://jsonplaceholder.typicode.com/posts
     api   ╭ Passed through
           ╰ GET https://jsonplaceholder.typicode.com/posts

Dev Proxy Version

0.15.0

Operating system (environment)

Linux

Shell

bash

Configuration file

{ "$schema": "https://raw.githubusercontent.com/microsoft/dev-proxy/main/schemas/v0.15.0/rc.schema.json", "plugins": [ { "name": "RetryAfterPlugin", "enabled": true, "pluginPath": "~appFolder/plugins/dev-proxy-plugins.dll" }, { "name": "GenericRandomErrorPlugin", "enabled": true, "pluginPath": "~appFolder/plugins/dev-proxy-plugins.dll", "configSection": "genericRandomErrorPlugin" } ], "urlsToWatch": [ "https://jsonplaceholder.typicode.com/*" ], "genericRandomErrorPlugin": { "errorsFile": "devproxy-errors.json" }, "rate": 50, "labelMode": "text", "logLevel": "information", "newVersionNotification": "stable" }

Additional Info

No response

plachance-q1 commented 6 months ago

My attempt to use windows' rootCert.pfx:

Open git bash, and navigate to my windows' devproxy install directory. openssl pkcs12 -in .\rootCert.pfx -out rootCert.crt -nodes

Then extract the certificate and copy it in a new file named dev-proxy-ca.crt

Switch to Ubuntu.

sudo cp dev-proxy-ca.crt /usr/local/share/ca-certificates/
sudo chmod 644 /usr/local/share/ca-certificates/dev-proxy-ca.crt
sudo update-ca-certificates

Output:

Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Adding debian:dev-proxy-ca.pem
done.
done.

Retrying the same test above (see steps to reproduce) and it still fails with the same error.

plachance-q1 commented 6 months ago

I found a workaround.

Using Ubuntu 18.04, VSCode and the latest .net sdk, I built the repository, set a breakpoint in the Proxy engine and started the app with --install-cert. Stepping through the code, I was able to locate the certificate written during the initial start of the proxy. The certificate installation still doesn't work but we can find the pfx file under this directory: ~/.config/dev-proxy/rootCert.pfx

Using the same procedure I described in my previous comment for the Windows certificate, it is now working as expected.

Important notes:

The certificate can't be reused. If it doesn't exist, a new certificate is created. If you test locally on your work desktop, you can't copy the certificate to another linux environment.
If the proxy runs in a docker container, the certificate is created in the [working dir]/dev-proxy/rootCert.pfx
If you want to use the docker container as proxy for your host machine you need to docker cp the file on the host and register the public certificate in the host trust store.

waldekmastykarz commented 6 months ago

Thank you for reaching out to us @plachance-qohash and sorry for the lack of clarity on our part. Like you noticed, we don't automatically register the root cert on Linux at the moment. I appreciate you digging into it and sharing your findings with us. It'll help us extend proxy with automatic setup on Linux. Speaking of which, would you be willing to contribute an equivalent of the macOS certificate setup script for Linux?

plachance-q1 commented 6 months ago

Hello @waldekmastykarz Yes, I will be happy to contribute a script for Linux configuration. Additionally, I will post a comment on how to utilize the dev-proxy within a Docker container and Docker Compose, enabling integration tests. I am currently working on the setup, and I'll share my findings in this thread as soon as possible.

waldekmastykarz commented 6 months ago

Awesome! Looking forward to it!

waldekmastykarz commented 3 months ago

Hey @plachance-qohash, are you still working on this? Anything that we can help with?

waldekmastykarz commented 3 months ago

Opening up due to lack of response

OnurGumus commented 1 month ago

I would be very happy if this issue is fixed.

plachance-q1 commented 1 month ago

I wish I had more time to invest time and offer you a Pull Request with what I did.

You can still use/try my fork: https://github.com/plachance-q1/dev-proxy

So we successfully managed to package the dev-proxy into a Docker image. The recording and playback can be controlled through script executed on the container. I had to make some modifications to the code, communication with dev-proxy is done through named pipe.

Once your container is up you can start or stop recording by executing this script on the container:

/app/recordings.sh start
/app/recordings.sh stop

Map a host directory with the container directory /app/dev-proxy to sync your mocks.

I don't have the time to give any support on this.

garrytrinder commented 1 month ago

Since this issue was first raised we have automated the install and trust of the certificate during first run on macOS.

https://github.com/microsoft/dev-proxy/blob/1df8846efaa25ce2fdc9e0f98c6b276d8336df80/dev-proxy/ProxyEngine.cs#L186-L208

We have manual install and trust instructions for Ubuntu in our docs, but we should be able to script this using a bash script like we have done with macOS.

OnurGumus commented 1 month ago

We have manual install and trust instructions for Ubuntu in our docs, but we should be able to script this using a bash script like we have done with macOS.

Manual instructions worked well, thanks.

microsoft / dev-proxy