add a retry for getting a Cluster

[randytpierce]

We are seeing failures caused by a failure to connect (builder_common.ingest_manager.py line 122)

 self.cluster = Cluster(
     "couchbase://" + self.cb_credentials["host"], options
)

So add a retry loop to give this three tries separated by 5 second sleep.

[randytpierce]

implemented the retry loop.

[randytpierce]

For some reason the CI is failing to build the ingest container now....

9 |     
  10 | >>> RUN apt-get update && \
  11 | >>>     apt-get install -y curl wget && \
  12 | >>>     apt-get install -y build-essential libssl-dev libnetcdff-dev libopenjp2-7-dev gfortran make unzip git cmake && \
  13 | >>>     mkdir /build && cd /build && \
  14 | >>>     wget [https://confluence.ecmwf.int/download/attachments/45757960/eccodes-${ECCODES_VER}-Source.tar.gz](https://confluence.ecmwf.int/download/attachments/45757960/eccodes-$%7BECCODES_VER%7D-Source.tar.gz) && \
  15 | >>>     tar -xzf  eccodes-${ECCODES_VER}-Source.tar.gz && \
  16 | >>>     mkdir eccodes-build && cd eccodes-build && \
  17 | >>>     mkdir /eccodes && \
  18 | >>>     # Default install location is /usr/local/{bin,include,lib}, use the CMAKE prefix to change to /eccodes
  19 | >>>     # Note you'll need to set some bashrc variables (ECCODES_DEFINITION_PATH & ECCODES_DIR) in subsequent images
  20 | >>>     # cmake -DCMAKE_INSTALL_PREFIX=/eccodes -DBUILD_SHARED_LIBS=ON -DENABLE_JPG=ON ../eccodes-${ECCODES_VER}-Source && \
  21 | >>>     cmake -DCMAKE_INSTALL_PREFIX=/eccodes ../eccodes-${ECCODES_VER}-Source && \
  22 | >>>     # Use nproc to get the number of available cores - use all but 1 core for compilation
  23 | >>>     make -j$(( $(nproc)-1 )) && \
  24 | >>>     make install
  25 |

need to find out why before I can merge the branch and close this. This does not seem to be related to this small code change.

[randytpierce]

discovered a missing break!

[randytpierce]

Actually I see there are more connections that need this. At least the one in ingest_manager and also one is needed in vx_ingest.