Can mock APIs for HTTP but not HTTPS

[walkerab]

Describe the bug

We are able to mock HTTP API requests but when we attempt to mock HTTPS API requests, they instead pass through to the original, un-mocked endpoints.

To Reproduce

Here I've set up three mocks.

• Manually configured http://google.com
• Manually configured https://google.com
• Moctokit configured https://api.github.com/rate_limit

I added the moctokit one so I could be sure I wasn't doing the mock setup incorrectly.

Workflow

```yaml name: Mock API Test on: pull_request: jobs: metrics: name: Metrics runs-on: ubuntu-latest steps: - name: HTTP api call run: | result=$(curl -s http://google.com) echo "$result" - name: HTTPS api call run: | result=$(curl -s https://google.com) echo "$result" - run: | curl -s -L \ -H "Accept: application/vnd.github+json" \ -H "Authorization: Bearer ${{ github.token }}" \ -H "X-GitHub-Api-Version: 2022-11-28" \ -o response.json \ https://api.github.com/rate_limit cat response.json ```

Jest Test Code

```js let { MockGithub, Moctokit } = require("@kie/mock-github") let { Act, Mockapi } = require("@kie/act-js") let path = require("path") jest.setTimeout(60000) let mockGithub beforeEach(async () => { mockGithub = new MockGithub({ repo: { testCompositeAction: { files: [ { src: path.join(__dirname, "mock-api.test.yml"), dest: ".github/workflows/mock-api.test.yml", }, ], }, }, }) await mockGithub.setup() }) afterEach(async () => { await mockGithub.teardown() }) test("it mocks api calls", async () => { const moctokit = new Moctokit() const mockapi = new Mockapi({ google_https: { baseUrl: "https://google.com", endpoints: { root: { get: { path: "/", method: "get", parameters: { query: [], path: [], body: [], }, }, }, }, }, google_http: { baseUrl: "http://google.com", endpoints: { root: { get: { path: "/", method: "get", parameters: { query: [], path: [], body: [], }, }, }, }, } }) const act = new Act(mockGithub.repo.getPath("testCompositeAction")) .setGithubToken("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") // fake token const result = await act.runEvent("pull_request", { logFile: path.join(__dirname, "../logs/metrics.log"), mockApi: [ mockapi.mock.google_http.root .get() .setResponse({ status: 200, data: "mock response" }), mockapi.mock.google_https.root .get() .setResponse({ status: 200, data: "mock response" }), moctokit.rest.rateLimit .get() .setResponse({ status: 200, data: "mock response" }), ] }) console.log(result) }) ```

Test Output

``` console.log [ { name: 'Main HTTP api call', status: 0, output: 'mock response' }, { name: 'Main HTTPS api call', status: 0, output: '\n' + '\n' + '

301 Moved

\n' + 'The document has moved\n' + 'here.\n' + '' }, { name: 'Main curl -s -L \\', status: 0, output: '{\n' + '"message": "Bad credentials",\n' + '"documentation_url": "https://docs.github.com/rest"\n' + '}' } ] ```

Expected behavior

All three mocks should intercept the api calls and respond with 'mock response'.

Instead only the HTTP API request is being mocked. The other two are hitting the actual APIs.

Logs

Logs

``` [Mock API Test/Metrics] 🚀 Start image=ghcr.io/catthehacker/ubuntu:act-latest [Mock API Test/Metrics] 🐳 docker pull image=ghcr.io/catthehacker/ubuntu:act-latest platform=linux/amd64 username= forcePull=true [Mock API Test/Metrics] 🐳 docker create image=ghcr.io/catthehacker/ubuntu:act-latest platform=linux/amd64 entrypoint=["tail" "-f" "/dev/null"] cmd=[] [Mock API Test/Metrics] 🐳 docker run image=ghcr.io/catthehacker/ubuntu:act-latest platform=linux/amd64 entrypoint=["tail" "-f" "/dev/null"] cmd=[] [Mock API Test/Metrics] ⭐ Run Main HTTP api call [Mock API Test/Metrics] 🐳 docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/0] user= workdir= [Mock API Test/Metrics] | mock response [Mock API Test/Metrics] ✅ Success - Main HTTP api call [Mock API Test/Metrics] ⭐ Run Main HTTPS api call [Mock API Test/Metrics] 🐳 docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/1] user= workdir= [Mock API Test/Metrics] | [Mock API Test/Metrics] | [Mock API Test/Metrics] |

301 Moved

[Mock API Test/Metrics] | The document has moved [Mock API Test/Metrics] | here. [Mock API Test/Metrics] | [Mock API Test/Metrics] ✅ Success - Main HTTPS api call [Mock API Test/Metrics] ⭐ Run Main curl -s -L \ -H "Accept: application/vnd.github+json" \ -H "Authorization: Bearer ***" \ -H "X-GitHub-Api-Version: 2022-11-28" \ -o response.json \ https://api.github.com/rate_limit cat response.json [Mock API Test/Metrics] 🐳 docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/2] user= workdir= [Mock API Test/Metrics] | { [Mock API Test/Metrics] | "message": "Bad credentials", [Mock API Test/Metrics] | "documentation_url": "https://docs.github.com/rest" [Mock API Test/Metrics] | } [Mock API Test/Metrics] ✅ Success - Main curl -s -L \ -H "Accept: application/vnd.github+json" \ -H "Authorization: Bearer ***" \ -H "X-GitHub-Api-Version: 2022-11-28" \ -o response.json \ https://api.github.com/rate_limit cat response.json [Mock API Test/Metrics] 🏁 Job succeeded ```

Additional context

I've tried this on a macbook (Ventura 13.5) as well as an AWS EC2 instance (Amazon Linux 2023) with the same results.

[walkerab]

This seems pretty similar to https://github.com/kiegroup/mock-github/issues/72

[walkerab]

Is this an issue of documentation? Is it a known issue and we need to be more clear what's said here?:

You can use Mockapi and Moctokit to mock any kind of HTTP and HTTPS requests during your workflow run provided that the client being used honours HTTP_PROXY and HTTP_PROXY env variables. Depending on the client, for HTTPS they might issue a CONNECT request to open a secure TCP tunnel. In this case Act won't be able to mock the HTTPS request. (Note - For Octokit, you can mock HTTPS requests because it does not issues a CONNECT request)

• Is it common or uncommon for a client to honour these env vars?
• How do you know if your client honours these?
• Should a client that honours HTTP_PROXY not also honour HTTPS_PROXY?

[shubhbapna]

Hi @walkerab, yes this is a known issue. I am not able to mock HTTPS requests yet because under the hood I use a proxy which decides whether or not to mock a request. Now even if I set the HTTPS_PROXY env var to a "http://" location some clients will issue a CONNECT request first. This request is used to tell a proxy to set up TCP tunnel to the destination which is then secured by TLS. Since the tunnel is encrypted the proxy is not able to read the actual requests and is not able to mock it. So for example:

• Client wants to make a request to https://google.com via the proxy running at http://localhost:3000
• Client issues a CONNECT request to proxy. This request only contains the host ("google") and port ("443") and not nothing else from the request
• Proxy sets up a tunnel between client and google
• Client initiates TLS handshake after which any data flowing through the tunnel in encrypted

So, as for your questions:

Is it common or uncommon for a client to honour these env vars?

It is common for clients to honor these variables. I know that curl does but sometimes clients such as Octokit don't and have to be configured manually

How do you know if your client honours these?

It is usually in their documentation

Should a client that honours HTTP_PROXY not also honour HTTPS_PROXY?

They do but that is not the issue here as I explained above. I do try to "fool" these clients by setting HTTPS_PROXY to a http location but it doesn't work for all clients, for example it doesn't work for curl but it works for axios

Now as for the workarounds for this, I would suggest if possible store the base url of your APIs as env variables in your workflow. Then while testing you should be able to change the env variables and set your base url to be http (for example all workflows have $GITHUB_API_URL available, you can easily override that while testing). I do have a plan of trying to handle HTTPS directly by implementing some sort of a MITM proxy but haven't been able to get to it yet. I am open to more ideas, if you have any, on how to handle this :)

Sorry for the unclear documentation, I will fix it.

[walkerab]

Thanks for the response, @shubhbapna. What you are saying makes enough sense to me.

For now I have worked around this by using the mockttp library instead like so:

Workflow

```yml name: Mock API Test on: pull_request: jobs: metrics: name: Metrics runs-on: ubuntu-latest steps: - name: CA Cert File run: | echo "$CA_CERT" > /tmp/ca-cert.pem - name: HTTP api call run: | result=$(curl -s http://google.com) echo "$result" - name: HTTPS api call run: | result=$(curl --cacert /tmp/ca-cert.pem -s https://google.com) echo "$result" - run: | result=$(\ curl --cacert /tmp/ca-cert.pem -s -L \ -H "Accept: application/vnd.github+json" \ -H "Authorization: Bearer ${{ github.token }}" \ -H "X-GitHub-Api-Version: 2022-11-28" \ http://api.github.com/rate_limit \ ) echo "$result" ```

Jest Test Code

```js const { MockGithub } = require("@kie/mock-github") const { Act } = require("@kie/act-js") const mockttp = require('mockttp'); const path = require("path") jest.setTimeout(60000) let mockGithub beforeEach(async () => { mockGithub = new MockGithub({ repo: { testCompositeAction: { files: [ { src: path.join(__dirname, "../.actrc"), dest: "/.actrc", }, { src: path.join(__dirname, "mock-api.test.yml"), dest: ".github/workflows/mock-api.test.yml", }, ], }, }, }) await mockGithub.setup() }) afterEach(async () => { await mockGithub.teardown() }) test("it mocks a github api call", async () => { const https = await mockttp.generateCACertificate() const server = mockttp.getLocal({ https }) await server.start() await server.forGet("http://google.com").thenReply(200, "mock response") await server.forGet("https://google.com").thenReply(200, "mock response") await server.forGet("http://api.github.com/rate_limit").thenReply(200, "mock response") const server_url = `http://host.docker.internal:${ server.port }` const act = new Act(mockGithub.repo.getPath("testCompositeAction")) .setEnv("HTTP_PROXY", server_url) .setEnv("http_proxy", server_url) .setEnv("HTTPS_PROXY", server_url) .setEnv("https_proxy", server_url) .setEnv("CA_CERT", https.cert) .setGithubToken("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") // fake token const result = await act.runEvent("pull_request", { logFile: path.join(__dirname, "../logs/metrics.log"), }) console.log(result) await server.stop() }) ```

[shubhbapna]

Oh nice this library looks promising. Thank you!

Although it looks like it has the same issue I was worried about in a MITM proxy - manually having to pass the cacerts but I think it is a good starting point!

[shubhbapna]

updated the docs feel free to open it again if it is unclear: https://github.com/kiegroup/act-js#mocking-apis-during-the-run

opened an issue to hopefully implement complete mocking of HTTPS request: #53