Error sending request to remote registry

[mvhulten]

I've installed Sarus on a compute node on one of our HPC clusters in Norway. The compute nodes do not have direct access to the Internet. We use a proxy for this. When we use wget we set the shell variable http_proxy.

Is it possible to set a proxy for Sarus?

Of course, I may have misinterpreted the whole issue—here is what happens:

[mhu027@c13-6 ~]$ sarus pull alpine
# image            : index.docker.io/library/alpine:latest
# cache directory  : "/home/mhu027/.sarus/cache"
# temp directory   : "/tmp"
# images directory : "/home/mhu027/.sarus/images"
[2346780.841753263] [c13-6-79584] [main] [ERROR] Error trace (most nested error last):
#0   retrieveImageManifest at "Puller.cpp":333 Error while sending request for manifest to remote registry
#1   "unknown function" at "unknown file":-1 Request canceled by user.

I tested the same thing successfully from a virtual machine (that has full Internet access).

[Madeeks]

Hello @mvhulten, thanks for your message. I confirm that at the moment Sarus does not implement proxy support for pulling images. I can propose you the following options:

1. Obtain the image in the form of a Docker tar archive, which you can then load with Sarus without requiring an internet connection from Sarus itself. Such archive could be generated by Docker (with docker save) or with Skopeo (e.g. with the command skopeo copy docker://alpine docker-archive:<file-path>.tar), and then transferred to the HPC cluster. This last transfer step could be skipped if Skopeo is available directly on your system.

2. Since your compute nodes all go through a single point of access to the internet, it might be a possibility for you to have a Docker registry inside your site's local network, to act as a mirror or pull-through-cache to Docker Hub. Sarus could pull through such registry mirror, with the possibility to whitelist the local mirror for insecure connections, should you not wish to set up TLS/SSL verification.

3. Request to implement support for proxying in the Sarus Image Manager. I have looked into this, and we could include the feature in a patch release, however given the upcoming holiday period, an estimate for the availability of such release would be no earlier than mid-January.

[mvhulten]

Hello Alberto, thank you for these useful ideas.

We don't have Docker or Skopeo on the cluster (only Singularity as of yet), but I wil explore this in January (on holidays as of now). Also the pull-through cache could be a good option. Including a way to configure a proxy would of course be the most easiest on our side—mid-January would be fine on our side!

[mvhulten]

I've set up a Docker pull-through cache on a system with Internet access. The compute nodes have access to that machine.

Where do I configure Sarus (on the compute node) to use our Docker registry pull-through cache (instead of pulling directly from upstream)?

[Madeeks]

Hi Marco, you should be able to use the pull-through registry just by pre-pending its hostname to the image argument of a sarus pull command, like when you are pulling from a registry different from Docker Hub, e.g. sarus pull <registry>/library/ubuntu.

If you didn't set up TLS/SSL security to your intermediate registry, you can use the insecureRegistries parameter in sarus.json to tell Sarus it can connect unsafely to the pull-through registry.

I would also like to let you know that earlier today we merged support for HTTP/HTTPS proxying in the main development branch (https://github.com/eth-cscs/sarus/commit/c39ccdc87e8f8c1bbf777eb5b5fa9da00db68e7f), and we are working towards finalizing a release soon. You can get a preview of the feature documentation by looking at the latest ReadTheDocs build: https://sarus.readthedocs.io/en/latest/user/user_guide.html#pulling-images-using-a-proxy

[mvhulten]

Thank you for this update! We'll probably go with the Docker pull-through registry solution, but the proxy feature will come in handy as well (for testing, as well as a fall-back in case the pull-through cache doesn't work).

I'll update (and close) in due time.

[mvhulten]

I tried to build the updated source with the proxy feature.

yum install rapidjson-devel cmake
cmake -DCMAKE_TOOLCHAIN_FILE=${build_dir}/../cmake/toolchain_files/${toolchain_file} \
        -DCMAKE_PREFIX_PATH="/usr/include/rapidjson" \
        -DCMAKE_BUILD_TYPE=${build_type} \
        -DCMAKE_INSTALL_PREFIX=${install_dir}
...
CMake Error at CMakeLists.txt:36 (find_package):
  Could not find a package configuration file provided by "cpprestsdk"
...

The available repos don't show any package whose name includes cpprestsdk. But it is fine for me to wait for the next release of Sarus.

[Madeeks]

Hi Marco, we have just tagged version 1.4.1 and we'll be publishing a release later today. :slightly_smiling_face:

Regarding your attempts to build from source, are you working on a CentOS system? If so, I can confirm that there are no repository packages available for cpprestsdk. On CentOS and OpenSUSE 15.x I suggest to build from source (OpenSUSE packages lack correct CMake files unfortunately):

pwd_bak=$PWD

# Install cpprestsdk
cd /tmp && \
    ([ -e /usr/include/xlocale.h ] || ln -s /usr/include/locale.h /usr/include/xlocale.h) && \
    mkdir -p cpprestsdk/v2.10.18 && cd cpprestsdk/v2.10.18 && \
    wget https://github.com/Microsoft/cpprestsdk/archive/v2.10.18.tar.gz && \
    tar xf v2.10.18.tar.gz && \
    mv cpprestsdk-2.10.18 src && cd src/Release && \
    mkdir -p build && cd build && \
    cmake -DCMAKE_BUILD_TYPE=Release -DWERROR=FALSE -DCPPREST_EXCLUDE_WEBSOCKETS=ON .. && \
    sed -i /tmp/cpprestsdk/v2.10.18/src/Release/include/cpprest/asyncrt_utils.h -e '1s/^/#include <sys\/time.h>\n/' || true && \
    make -j$(nproc) && \
    sudo make install && \
    cd ${pwd_bak} && \
    rm -rf /tmp/cpprestsdk

[mvhulten]

Yes, it is on CentOS 7.9. Unfortunately this has an outdated cmake:

[root@c3-7 build]# cmake -DCMAKE_BUILD_TYPE=Release -DWERROR=FALSE -DCPPREST_EXCLUDE_WEBSOCKETS=ON ..
CMake Error at CMakeLists.txt:2 (cmake_minimum_required):
  CMake 3.9 or higher is required.  You are running version 2.8.12.2

[Madeeks]

Hi Marco, we just published Sarus 1.4.1 here on GitHub.

About CMake 3.x packages for CentOS 7, they can be installed from the EPEL repositories:

yum install epel-release

In general, to build Sarus from source, I recommend to follow our documentation also when installing/building dependencies: https://sarus.readthedocs.io/en/stable/install/requirements.html#software The snippets in the docs are rendered directly from our CI scripts and are regularly tested.

[mvhulten]

I tried the proxy option, but it does not seem to work. c27-1 is one of the compute nodes that can only reach the Internet through proxy.fram:3128.

[mhu027@c27-1 ~]$ sarus --version
1.4.1
[mhu027@c27-1 ~]$ https_proxy="https://proxy.fram:3128" sarus pull alpine
# image            : index.docker.io/library/alpine:latest
# cache directory  : "/home/mhu027/.sarus/cache"
# temp directory   : "/tmp"
# images directory : "/home/mhu027/.sarus/images"
[2702.486931410] [c27-1-12179] [main] [ERROR] Error trace (most nested error last):
#0   retrieveImageManifest at "Puller.cpp":325 Error while sending request for manifest to remote registry
#1   "unknown function" at "unknown file":-1 Error in SSL handshake

On another machine—which has direct Internet access—Sarus succesfully pulls the image, with or without setting https_proxy.

[mvhulten]

The proxy (option 3) works.