search

Netflix / lemur-docker

Docker files for the Lemur certificate orchestration tool
170 stars 88 forks source link

bower ESOCKETTIMEDOUT during make or docker-compose #59

Closed jibbajabber closed 3 years ago

jibbajabber commented 3 years ago

Hi, I'm seeing issues when trying to evaluate lemur for the first when following the projects README https://pastebin.com/9wBZv8Zp

os: CentOS 7.6.1810 docker-ce: 20.10.5 docker-compose: 1.29.0 command ran: make result: During the make process bower is raising ESOCKETTIMEDOUT consistently so the containers are never created expected: All containers required created and lemur hosted on localhost:87 and localhost:447

I noticed when running make that npm is expected but not listed as a requirement, my local npm installed is 3.10.10

I also noticed running docker-compose up as is, also fails if make isn't run first as the expected directory structure for lemur is off as per the README, I think it needs to be git clone --depth=1 git@github.com:Netflix/lemur.git lemur-build-docker/lemur instead however perhaps it makes sense to use https urls throughout instead to satisfy all users so git clone --depth=1 https://github.com/Netflix/lemur lemur-build-docker/lemur maybe?

Also the docker-compose version is set to 3.7 (currently) but for me I have to change this to 3.3 else:

ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1.

I'm hoping this is all user error and I'm doing something wrong, but can't currently see why bower is failing to resolve what can be resolved on the host that runs docker.

In addition to the bower url failures (as per the pastebin URL) I see the following deprectaed warnings:

npm WARN deprecated gulp-util@3.0.8: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5
npm WARN deprecated gulp-foreach@0.1.0: Either use gulp-tap or gulp-flatmap, depending on your needs
npm WARN deprecated gulp-minify-css@1.2.4: Please use gulp-clean-css
npm WARN deprecated gulp-util@2.2.20: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5
npm WARN deprecated chokidar@2.1.8: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated node-uuid@1.4.8: Use uuid module instead
npm WARN deprecated fsevents@1.2.13: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated debug@4.1.1: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated browserslist@1.7.7: Browserslist 2 could fail on reading Browserslist >3.0 config used in other tools.

> gifsicle@5.2.0 postinstall /opt/lemur/node_modules/gifsicle
> node lib/install.js

  āœ” gifsicle pre-build test passed successfully

I did notice that from inside the container during the build process if I try to curl a bower failed url I get a redirection, could this be the cause?:

root@8675b56a6734:/opt/lemur# curl https://github.com/jquery/jquery-dist/archive/2.2.4.tar.gz
<html><body>You are being <a href="https://codeload.github.com/jquery/jquery- 
dist/tar.gz/2.2.4">redirected</a>

One other item to note, failed builds don't clean up what look like ephemeral containers, it's no big deal but would be nice if those were removed if a make/docker-compose fails

jtschladen commented 3 years ago

I tested this morning on MacOS and Centos and I'm unable to reproduce your issue - my containers come up fine. The only steps required for Centos (fresh VM, 8.2.2004 - I didn't match your version here) were installing docker-engine (20.10.5), docker-compose (1.29.0), git clone https://github.com/Netflix/lemur-docker.git, and make.

Is there any chance you could try this on a different host machine to see whether you get a different result? I don't have an explanation at the moment but searching for ESOCKETTIMEDOUT makes me wonder if the host machine might have a slow network or some other network configuration that could be causing problems.

Regarding some of your other observations:

I noticed when running make that npm is expected but not listed as a requirement, my local npm installed is 3.10.10

npm is installed here in the Dockerfile

I also noticed running docker-compose up as is...

You're correct, the directory got changed at some point and the README is wrong, I'll fix that - thanks for pointing it out.

Also the docker-compose version...

Not sure why this is showing up for you; I'm not seeing that when using the exact same docker version as you noted. Docker's versioning table seems to suggest 3.7 should work: https://docs.docker.com/compose/compose-file/compose-versioning/

In addition to the bower url failures (as per the pastebin URL) I see the following deprecated warnings:

I see these as well, but they shouldn't be a blocking issue. This is something that would need to be taken up in the main lemur repo, and I'm not sure if anything is blocking us on that.

I did notice that from inside the container during the build process if I try to curl a bower failed url I get a redirection, could this be the cause?:

I don't think so - I see the same in my (successful) container.

One other item to note, failed builds don't clean up what look like ephemeral containers

Thanks, good feedback; I can try to look into that.

jibbajabber commented 3 years ago

I'm not hugely familiar with npm but I'm attempting to build lemur again with the following additions (I'm not behind a proxy):

npm config delete proxy && \
npm config delete http-proxy && \
npm config delete https-proxy && \
npm config set proxy null && \
npm set maxsockets 3 && \ :

So this is now:

RUN apt-get update && \
    apt-get -y --no-install-recommends upgrade && \
    apt-get install -y --no-install-recommends libpq-dev curl build-essential locales libffi-dev libsasl2-dev libldap2-dev \
        dh-autoreconf git python3-dev python3-pip python3-venv python3-wheel nodejs npm && \
    locale-gen en_US.UTF-8 && export LC_ALL=en_US.UTF-8 && \
    npm config set registry http://registry.npmjs.org/ && \
    npm install npm -g && \
    echo "Running with nodejs:" && node -v && \
    python3 -m venv /opt/venv && \
    echo "Running with python:" && /opt/venv/bin/python3 -c 'import platform; print(platform.python_version())' && \
    /opt/venv/bin/python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel && \
    /opt/venv/bin/python3 -m pip install --no-cache-dir -e . && \
    npm config delete proxy && \
    npm config delete http-proxy && \
    npm config delete https-proxy && \
    npm config set proxy null && \
    npm set maxsockets 3 && \
    npm install --unsafe-perm && \
    node_modules/.bin/gulp --cwd /opt/lemur build && \
    node_modules/.bin/gulp --cwd /opt/lemur package && \
    npm cache clean --force && \
    rm -rf node_modules && \
    python3 -c 'print(" \033[32m BUILDER DONE \033[0m ")'

If my suggestions around https, clone path and npm are valid and helpful I can raise a PR

jtschladen commented 3 years ago

Interesting, I wonder why that would be necessary. If you've got time @jibbajabber, would you be able to try getting all those config values before deleting them? I'm curious to know if your container has a value set for some reason, and if there's anything we can change earlier to avoid needing this.

npm config get proxy && \
npm config get http-proxy && \
npm config get https-proxy && \
npm get maxsockets && \ :

At any rate, if it fixes your problem, we should probably add it. I'll find some time in an hour or two to verify it doesn't break anything for me, but if not I think a PR would be helpful (I already fixed the clone path though, thanks for pointing it out!).

jibbajabber commented 3 years ago

Thanks for the fast response šŸ˜„

No change so far sadly, still get time outs. I tried adding to package.json pretest and prelint and rebuilding to see if it resolved this issue (I'm quite clueless when it comes to npm so fumbling my way around) but no change, for clarity I added:

"scripts": {
    "postinstall": "node_modules/.bin/bower install --allow-root --config.interactive=false",
    "pretest": "npm config rm proxy && npm config rm https-proxy && npm config delete http-proxy && npm config delete https-proxy && npm install && npm run build_static",
    "build_static": "gulp build",
    "prelint": "npm config rm proxy && npm config rm https-proxy && npm config delete http-proxy && npm config delete https-proxy && npm install",
    "lint": "jshint lemur/static/app/",
    "test": "gulp test"
  },

Happy to try whatever helps the project could you elaborate on "getting all those config values", sorry not too sure what you'd like me to collect but happy to do so

The values added so far (but hasn't fixed my issue):

npm config rm proxy
npm config rm http-proxy
npm config rm https-proxy
npm config delete proxy
npm config delete http-proxy
npm config delete https-proxy
npm config set proxy null
npm set maxsockets 3

Will check in later on tonight to see if there is anything else I can do, if useful

jibbajabber commented 3 years ago

RE npm local requirement, you're right; I just removed npm from my system and ran make again without an issue relating to dependencies, hmm sorry about that, not sure what happened there and I don't have the log, must have been user error on my part.

I'll spin up another CentOS 7 and 8 VM where I am currently and on a different site to confirm and report back either later tonight or some time tomorrow šŸ‘šŸ¼

jibbajabber commented 3 years ago

Working without issue on a new centos 7 environment (no npm changes required), so I won't test on centos 8, could well be something really off in my other environment.

Apologies for distraction and thank you for taking a look at this šŸ˜ƒ