Building behind corporate proxy

[carnak]

Okay, I give up... anybody been able to get this to build behind a corporate proxy?

I have been able to hack up the build enough to get here:

 > [misp-modules python-build 10/25] RUN poetry lock:
0.832 Creating virtualenv misp-modules-DIiUZofu-py3.12 in /root/.cache/pypoetry/virtualenvs
1.308 Updating dependencies
1.317 Resolving dependencies...
1.909 
1.909 HTTPSConnectionPool(host='github.com', port=443): Max retries exceeded with url: /cartertemm/ODTReader.git/info/refs?service=git-upload-pack (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)')))
------
failed to solve: process "/bin/sh -c poetry lock" did not complete successfully: exit code: 1

And looking here, https://github.com/python-poetry/poetry/issues/3748 , this looks grim.

Nothing i have tried so far to get poetry lock to ignore / honor my proxy cert hasn't worked.

[ostefano]

Does misp-core work after using the commented out lines 8-9 in the Dockerfile?

[carnak]

Does misp-core work after using the commented out lines 8-9 in the Dockerfile?

Just for the record, here are the diffs for the files i have hacked up to get this far:

(base) remnux@remnux:~/Downloads/misp-docker-ref$ diff ./template.env ../misp-docker/.env 
161,164c161,164
< # PROXY_ENABLE=true
< # PROXY_HOST=
< # PROXY_PORT=
< # PROXY_METHOD=
---
> PROXY_ENABLE=true
> PROXY_HOST=172.17.0.1
> PROXY_PORT=3128
> PROXY_METHOD=http
(base) remnux@remnux:~/Downloads/misp-docker-ref$ diff ./docker-compose.yml ../misp-docker/docker-compose.yml 
72a73,74
>           - http_proxy=http://172.17.0.1:3128
>           - https_proxy=http://172.17.0.1:3128
97c99
<       # - "./rootca.pem:/usr/local/share/ca-certificates/rootca.crt"
---
>       - "./redacted.crt:/usr/local/share/ca-certificates/redacted.crt"
(base) remnux@remnux:~/Downloads/misp-docker-ref$ diff ./core/Dockerfile ../misp-docker/core/Dockerfile 
3d2
< 
8,9c7,8
<     # COPY ./rootca.crt /usr/local/share/ca-certificates/rootca.pem
<     # COPY ./rootca.crt /usr/lib/ssl/cert.pem
---
>     #COPY ./redacted.crt /usr/local/share/ca-certificates/rootca.pem
>     #COPY ./redacted.crt /usr/lib/ssl/cert.pem
15c14,15
<     RUN curl -sSLo /tmp/debsuryorg-archive-keyring.deb https://packages.sury.org/debsuryorg-archive-keyring.deb
---
>     
>     RUN curl -k -sSLo /tmp/debsuryorg-archive-keyring.deb https://packages.sury.org/debsuryorg-archive-keyring.deb        
(base) remnux@remnux:~/Downloads/misp-docker-ref$ diff ./modules/Dockerfile ../misp-docker/modules/Dockerfile 
8c8
< 
---
>     
26c26
<             git clone https://github.com/MISP/misp-modules.git /srv/misp-modules && cd /srv/misp-modules && git checkout ${MODULES_COMMIT}
---
>             git -c http.sslVerify=false clone https://github.com/MISP/misp-modules.git /srv/misp-modules && cd /srv/misp-modules && git checkout ${MODULES_COMMIT}
28c28
<             git clone --branch ${MODULES_TAG} --depth 1 https://github.com/MISP/misp-modules.git /srv/misp-modules
---
>             git -c http.sslVerify=false clone --branch ${MODULES_TAG} --depth 1 https://github.com/MISP/misp-modules.git /srv/misp-modules
33c33
<     RUN pip install poetry
---
>     RUN pip install --proxy=http://172.17.0.1:3128 poetry
34a35,41
> 
>     RUN pip config set global.proxy http://172.17.0.1:3128
>     RUN pip config set global.cert false
>     ENV PYTHONHTTPSVERIFY 0
>     ENV SSL_CERT_FILE /usr/local/share/ca-certificates/redacted.crt
>     #RUN poetry source add gith https://github.com
>     #RUN poetry config certificates.gith.cert false

To answer your question directly, I had missed uncommenting these, but when i do, the build breaks earlier with php74, i.e.,:

#19 [misp-core php-base 8/8] RUN apt-get update
#19 0.390 Hit:1 http://deb.debian.org/debian bookworm InRelease
#19 0.416 Ign:2 https://packages.sury.org/php bookworm InRelease
#19 0.426 Hit:3 http://deb.debian.org/debian bookworm-updates InRelease
#19 0.464 Hit:4 http://deb.debian.org/debian-security bookworm-security InRelease
#19 1.569 Ign:2 https://packages.sury.org/php bookworm InRelease
#19 3.712 Ign:2 https://packages.sury.org/php bookworm InRelease
#19 7.869 Err:2 https://packages.sury.org/php bookworm InRelease
#19 7.869   Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 172.17.0.1 3128]
#19 7.869 Reading package lists...
#19 8.549 W: Failed to fetch https://packages.sury.org/php/dists/bookworm/InRelease  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 172.17.0.1 3128]
#19 8.550 W: Some index files failed to download. They have been ignored, or old ones used instead.
#19 DONE 8.6s

Once the php.list file for apt has been created, apt update breaks.

[ostefano]

I can do some tests and check that the whole build works with custom SSL certificates (of which you have the root stored somewhere) when MISP 2.5 is released (should be a matter of days).

I won't be able to test building using a proxy.

[carnak]

Sounds good, many thanks. And just fyi, I was able to reproduce this error at home, by pointing the proxy environment at this https://hub.docker.com/r/mitmproxy/mitmproxy . I was able to reproduce the same break on php.list with this approach, but ymmv.

[ostefano]

Try using env variables to configure the proxy. If you succeed doing so we might be able to merge things.

[carnak]

I want to follow up and report back that yes, it is possible to build misp-docker behind a corporate proxy, sort of. I don't think how we solved it is necessarily scalable to others situations, so i'm not going to share all the gory details, though I will say that misp-core builds pretty much out of the box, with minimal intervention, but misp-modules usage of "poetry" is a royal pita behind a proxy.

The Poetry component itself isn't easily amenable to usage behind a proxy, and we ended up pointing it at our local pypi repo clone. Even then, it had to be aware of how to get out of the proxy, and exhibits different behavior, based upon whether you interact via the commandline, vs docker RUN invocations...

I will close this one out by saying many thanks to the contributors to misp-docker, and misp-proper, as you all rock.

[carnak]

closing